How to calibrate your camera

Abstract

A method for calibrating at least one of six degrees of freedom of all or some of the cameras in a formation arranged for scene capture, the method including an initial calibration step prior to scene capture, which step includes creating a reference video frame including a reference image of a fixed reference object. During scene capture, the method further includes a further calibration step in which the position of the reference image of the fixed reference object in the captured scene video frame is compared with the position of the reference image of the fixed reference object in the reference video frame, and a step of adapting at least one of the six degrees of freedom of the multiple cameras in the formation as needed to obtain an improved scene capture after the further calibration.

Description

[Technical Field]

[0001] The present invention relates to a method for calibrating at least one of six degrees of freedom of all or part of a camera in a formation positioned for scene capture, the method comprising an initial calibration step prior to scene capture, the step comprising creating a reference video frame having a reference image of a fixed reference object, During scene capture, the method further comprises: a further calibration step in which the positions of the reference images of the fixed reference objects in the captured scene video frames are compared with the positions of the reference images of the fixed reference objects in the reference video frames; - adapting at least one of the six degrees of freedom of the multiple cameras of said formation to obtain an improved scene capture after further calibration. [Background technology]

[0002] In this application, the term six degrees of freedom refers to three possible translational and three possible rotational movements of the camera position. In this application, the camera degrees of freedom are intended to cover not only physical translational or rotational movements of the camera, but also virtual translational or rotational movements of the camera. It is emphasized, therefore, that the camera is not translated or rotated; instead, similar actions (with similar results) are performed on the images generated by the camera. Therefore, in the case of virtual (electronic, digital) control of a camera, the adaptation of the six degrees of freedom of the camera is actually the adaptation of one or more of three possible camera position parameters and / or one or more of three possible rotational parameters. For example, an image acquired by a given camera can be spatially transformed to correspond to a virtual camera that is slightly rotated compared to the physical camera. This concept is also important from the perspective of virtual view synthesis and maintaining camera calibration. A combination of both physical and virtual movements, such as physical translational and virtual rotational movements, is not excluded.

[0003] The invention also relates to a camera formation and a computer program related to the calibration method.

[0004] In the prior art, a rig formation with cameras, for example, eight cameras, is known. This can be used in various applications, for example, in virtual and / or augmented reality recording systems, commonly referred to as VR / AR applications. Such VR / AR devices can be stationary or portable (handheld). The cameras on such devices are typically pre-calibrated in a factory (or laboratory) based on certain parameters, such as the focal length of the lenses. These parameters are then typically fixed and no longer need to be calibrated by the user of the VR / AR device. However, before actual use, the user must typically perform an initial calibration related to the actual scene setting. This allows one or more of the camera's six degrees of freedom to be adapted as needed. The VR / AR device is then ready to begin scene recording. During recording, it is of utmost importance that any degrees of freedom are changed only when desired, and thus only when the camera is, for example, rotated to another part of the scene. Therefore, during recording, the camera's translational or rotational movement is not (significantly) impeded; otherwise, accurate VR / AR recording could be hindered by unpleasant artifacts. Typically, rigs are relatively small with a relatively small number of cameras, and recording is mostly performed inside a house or building, so external disturbances can be largely avoided. However, especially with portable devices, the chances of such disturbances are higher. Of course, this problem is magnified in situations where the device is located outside and subject to vibrations caused by passing trucks or wind. This problem becomes even more serious when large formations of cameras are required, such as in sports stadiums like soccer stadiums. This is because the distance between cameras is generally large and the cameras are often no longer physically connected by a rigid structure. For example, in a small camera rig (small 3D look-around effect), the cameras may be fixedly mounted on a 30 mm x 8 mm thick aluminum bar, spaced, for example, 5 cm apart.However, this is generally not possible outdoors, where cameras may be mounted at meter intervals on existing infrastructure with unknown mechanical properties, since very small deviations in camera position or orientation can already give noticeable artifacts. Summary of the Invention [Problem to be solved by the invention]

[0005] The object of the present invention is to overcome or at least reduce the effects of camera disturbances during scene recording. [Means for solving the problem]

[0006] According to a first embodiment of the present invention, the method is characterized as defined in the characterizing part of claim 1. An advantage is that if one or more of the cameras become (overly) uncalibrated, an "online" further calibration can be started immediately so that the recording of the scene is not interrupted, to repeat the initial calibration similar to that performed during setup before the start of the scene recording. Another advantage is that the further calibration can be performed in time, before disturbances to the cameras become large enough to cause significant and annoying artifacts. Preferably, all cameras create reference video frames containing reference images of fixed reference objects, but it is also possible for only some of the cameras to do so.

[0007] If the further calibration is performed by virtually controlling the camera, and thus virtually adapting the camera's degrees of freedom, i.e., by transforming the reference image of a fixed reference object in the captured scene video frame to the same position and orientation as the reference video frame, the further calibration has another advantage in that it can be performed more quickly than if the further calibration were performed by mechanically controlling the camera. (Electronic control is usually faster than mechanical control.) To avoid undesirable outer regions in the calibrated (adapted) video frame, rescaling and cropping of the video frame can be performed. Rescaling and cropping techniques have been known for many years and are described, for example, in US Patent No. 7,733,405 B2, issued June 8, 2010, to Van Dyke et al.

[0008] If the further calibration is performed by mechanical control of the camera, this has the advantage that the rescaling and cropping techniques described above are not required.

[0009] A further advantage of mechanical control is that the range of calibration can be larger than with electronic control. In principle, separate (additional) servos can be applied for mechanical control, but mechanical control can also be performed by servos that are usually already available and intended for normal camera control.

[0010] Of course, corresponding to the previous definition of what is meant by camera degrees of freedom, camera adaptation is also intended to cover physical and / or virtual adaptation. The result of step B may be adaptation of two or more of the six degrees of freedom. For example, there is a lateral camera adaptation and an adaptation in both yaw and roll. The values ​​in step C may also include a sign (positive or negative), so that it is clear whether lateral is leftward or horizontal in the above example. The timestamps in step C do not need to refer to a real time, such as a country's local time or Greenwich Mean Time. They are only required for an arbitrary (freely) determined time. The classification of different root causes of camera disturbances is an invaluable feature of the present invention. These root causes can be recorded and used (later) to learn how to improve the camera position within the formation or where to better position the complete formation.

[0011] According to a second embodiment of the invention, the method is characterized as defined in the characterizing part of claim 2. The advantage is that it is also possible to further distinguish one-time or low-frequency disturbances of specific root causes.

[0012] According to third and fourth embodiments of the present invention, the methods are characterized as defined in the third and fourth characterizing parts of claims 3 and 4, respectively. This is advantageous when not all degrees of freedom need to be adaptive to perform the correction. For example, in certain settings, disturbances may primarily cause deviations in the camera orientation. The advantages are faster and easier control of the camera adaptation and a reduction in the amount of data, which is particularly advantageous when the data is transferred, for example, to a client system.

[0013] According to a fifth embodiment of the invention, the method is characterized as defined in the characterizing part of claim 5. This has the advantage that no unnecessary further calibration is performed, e.g. when the camera orientation deviation is so small that it does not result in noticeable artifacts in the image.

[0014] According to a sixth embodiment of the present invention, a method is characterized as defined in the characterizing part of claim 6. This is particularly advantageous when the formation includes a large number of cameras (e.g., 50 or more). Then, for example, for a second portion of cameras that require correction by further calibration (translation and / or rotation adaptation), it can be decided not to perform the further calibration but instead to use information from nearby (but not necessarily directly) neighboring cameras to perform the necessary correction. This has the advantage of increasing the (total) calibration speed of the camera formation and reducing the amount of data, which is particularly advantageous when the data is transferred, for example, to a client system. For example, consider a camera array consisting of 100 cameras distributed across four sides of a rectangular sports field. After an initial calibration of all cameras has been performed, all 100 cameras can be monitored by selecting two end cameras per side (total of eight cameras). If one of the two monitoring cameras on one side is no longer calibrated, all intermediate cameras can be compared to this camera and recalibrated based on the differences.

[0015] Seventh and eighth embodiments of the present invention are characterized as defined in the characterizing parts of claims 7 and 8, respectively. The more uniform the distribution, the more accurate the recording. In most cases, it is sufficient to use interpolation techniques, but especially near the end of the formation, it may be advantageous to also apply extrapolation techniques.

[0016] However, it should be emphasized that the best recording quality can be expected if no alternative calibration is applied (and therefore the further calibration is applied instead).

[0017] According to a ninth embodiment of the present invention, the method is characterized in that the initial calibration is repeated autonomously or continuously upon certain events. While the initial calibration should, in principle, be avoided as much as possible (preferably performed only once before scene recording), it may sometimes occur that further calibration and / or alternative calibration alone is no longer sufficient. This occurs when the camera deviates too far from calibration. This event can be detected manually or automatically. For example, a normal interruption in a soccer game is also a good moment to perform the initial calibration. However, to try to avoid such occurrences (or events), the initial calibration can also be repeated continuously, preferably autonomously, at a low repetition frequency. For example, the reference video frame may need to be updated because a fixed reference object gradually changes over time, for example, due to a changing lighting environment that casts a shadow on the fixed object.

[0018] According to the invention, a camera formation is defined in claim 10, which corresponds to claim 1. It is emphasized that the other disclosed methods of the invention, in particular the methods defined in claims 2 to 9, can also be advantageously applied to the camera formation.

[0019] The present invention further comprises a computer program comprising computer program code means adapted to be executed on a computer for performing any of the methods of the invention, in particular the methods defined in claims 1 to 9.

[0020] For a better understanding of the present invention and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0021] [Figure 1] Diagram showing two linear arrays of eight cameras each positioned along the side of a soccer field. [Figure 2]1 illustrates exemplary images of the leftmost and rightmost cameras in an eight-camera array as viewed from the side of a soccer player field. [Figure 3] FIG. 1 illustrates a multi-camera system for generating 3D data. [Figure 4] Diagram showing the alignment of two pairs of cameras in an eight-camera linear array. [Figure 5] 1A and 1B are diagrams illustrating a set of reference feature points and a set of corresponding feature points in a 2D image plane; [Figure 6] Calibration schematic. [Figure 7] FIG. 1 illustrates method steps for classifying recognized patterns to indicate different root causes of an uncalibrated camera. DETAILED DESCRIPTION OF THE INVENTION

[0022] The present invention will now be described with reference to the drawings.

[0023] It should be understood that the detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the invention. These aspects and advantages will become better understood from the following description, the appended claims, and the accompanying drawings.

[0024] FIG. 1 shows a camera formation with two linear arrays of eight cameras arranged along each side of a soccer field. The capture cameras (and therefore the physically present cameras) are represented by hollow triangles. The virtual cameras are shown by filled triangles. The virtual cameras do not physically exist; they are created and exist only by electronics and software. After calibration and depth estimation, a virtual view can be synthesized for positions between the capture cameras or even at positions advanced into the player field. Often, especially in large sports fields such as soccer fields, more cameras, e.g., 200 cameras, may be utilized. The cameras do not need to be aligned in a (virtual) straight line. For example, it may be advantageous to arrange the camera formation along the entire side of the soccer field, so that cameras closer to the edges of the camera formation (closer to the left / right goals) are positioned closer to and more oriented toward the center of the player field. The formation can, of course, also completely surround the player field. Also, the formation may actually be divided into several formations, for example, Figure 1 actually shows two formations of eight cameras.

[0025] Figure 2 shows example images captured by the leftmost and rightmost cameras in an eight-camera long linear array positioned along the side of a soccer field. As can be seen, the perspective of the players varies considerably from camera 1 to camera 8. The boundary visible at the far end of the soccer field can be used to define a useful image feature for calibration. This feature can then serve as a stationary object for creating one or more reference video frames. For example, a checkerboard panel or a staircase between seats would be suitable for this purpose.

[0026] Figure 3 shows a system diagram in which the coupled steps of image distortion correction, rectification, and disparity estimation rely on initial calibration and calibration monitoring and control (including further calibration). For simplicity, there is only one "connecting arrow" drawn from the top "Distortion Removal / Rectification Block" to the "First Calibration Block," and there is also one "connecting arrow" drawn from the "First Calibration Block" to the top "Distortion Removal / Rectification Block." However, it should be clear that these connections also exist to all other "Distortion Removal / Rectification Blocks."

[0027] These camera images are individually undistorted to compensate for lens distortion and focal length for each camera. However, the same spatial remapping process aligns each camera in each camera pair. For example, cameras 1 and 2 form the first pair in the array. Camera pairs are typically positioned spatially adjacent to each other in the array. The alignment step ensures that, for each pair, the input camera images are transformed into a stereo pair whose optical axes are aligned and whose rows in the images have the same positions. This is essentially a rotational operation and is commonly known as stereo rectification. This process is illustrated in Figure 4.

[0028] The calibration process typically relies on a combination of the following known algorithms (see, for example, Wikipedia): 1. True range multilateration for attitude determination based on laser measurements; 2. Perspective N-point calibration, combining N scene points with known 3D positions; 3. Structure-from-motion with bundle adjustment.

[0029] After an initial calibration upon installation, the multi-camera system can be used for different purposes. For example, depth estimation can be performed, followed by synthesizing virtual views to enable AR / VR experiences for observers of the match. However, the system may also be used to recognize the position of an object (e.g., a player's foot or knee) at a given time and determine its 3D position based on multiple views. It should be noted that in the latter application, while two views are theoretically sufficient, having many viewpoints minimizes the possibility of occlusion of the object of interest.

[0030] Figure 4 shows the alignment of the camera pairs of an eight-camera linear array. The alignment transforms each image so that the optical axes, indicated by the arrows (R1, R1a; R2, R2a; R3, R3a; R4, R4a), are parallel in pairs, indicated by the arrows (G1, G1a; G2, G2a; G3, G3a; G4, G4a), and orthogonal to the line connecting both cameras (dotted line). This alignment allows for easier disparity estimation and depth calculation.

[0031] Figure 5 shows a schematic diagram of a set of reference feature points and a set of corresponding feature points in a 2D image plane. The correspondence is determined by motion estimation and is indicated by the dashed lines. As can be seen, the corresponding points in the new frame cannot always be found because occlusions may occur or motion estimation may fail due to image noise.

[0032] Figure 6 illustrates a schematic of calibration by showing an example of a changing camera (in this case, camera 1). Original features are detected in the reference frame, and their image locations are represented by open circles. Corresponding feature points are then determined via motion estimation in the first and second frames. These corresponding points are represented as solid circles. As shown, these points do not change position in frame 1 for both cameras 1 and 2, and in frame 2 for camera 2. However, in camera 1, points 1, 2, and 3 do change position. Therefore, we can conclude that camera 1 is no longer calibrated. Note that for reference point 4, which is only visible in camera 1, no corresponding feature point was found. This point is marked as "invalid" and therefore is not considered when calculating the error metric that concludes the camera is no longer calibrated (see the equation below).

[0033] Therefore, 3D scene points are first projected onto a reference frame (represented by open circles). Using image-based matching, correspondences are estimated for these points for further frames (represented by filled circles). When a given percentage of corresponding feature points do not change position, the calibration status for the further frames is determined to be OK. This is the case for both camera 1 and camera 2 in frame 1. However, in frame 2, camera 1 exhibits a rotation error, as three points are displaced compared to the reference point.

[0034] FIG. 7 shows the steps for classifying the recognized patterns (in the 6-DOF disturbances of the camera) to indicate the respective root cause of the camera going out of calibration.

[0035] In step A, it is analyzed which cameras of the camera formation caused by disturbances, such as wind, have been adapted.

[0036] In step B, for each adapted camera, it is analyzed which of the six degrees of freedom have been adapted. For example, in step A, it may be determined that for a first camera, only the orientation direction "yaw" has been adapted, and for a second camera, both the orientation directions "yaw" and "roll" and the translation direction "left / right" have been adapted.

[0037] In step C, the fit values ​​and timestamps are determined. For example, for the first camera, the "yaw" value is 0.01°, and for the second camera, the "yaw" and "roll" values ​​are 0.01° and 0.02°, respectively, and the "left / right" value is +12mm (e.g., +12mm is 12mm to the right, -12mm is 12mm to the left).

[0038] In step D, a plurality of patterns along the camera formation are recognized along the camera formation by analyzing the information resulting from steps A, B, and C. For example, for a first timestamp, all values ​​of the adaptation related to "yaw" are registered as a first pattern of camera deviation, all values ​​of the adaptation related to "roll" are registered as a second pattern of camera deviation, etc.

[0039] In step E, the recognized patterns are classified, and this classification indicates different root causes of camera disturbances in one or more of the six degrees of freedom of the camera in the formation. Examples of classifications are "wind," "mechanical stress," "temperature effects," etc. These classifications are determined, for example, by knowledge gained from previous measurements. For example, the first pattern is compared with all possible classifications. It is not necessary for the first pattern to be considered a "hit" only if the corresponding classifications match exactly; a high similarity is sufficient. Any kind of (known) pattern recognition method can be used. Of course, if no "hit" is found at all, the pattern can be classified as an "unknown cause."

[0040] Optionally in step F, step E takes into account multiple analysis sessions of steps A, B, C and D. This gives additional possibilities for classifying patterns. For example, if a calibration problem occurs periodically for, say, only two cameras in a large formation of cameras, the classifier can output a "recurrent group disturbance".

[0041] A suitable error metric for detecting calibration problems is the sum of the distances between corresponding positions for all valid feature points.

number

number

[0042] where T is a preset threshold value [pixels]. This threshold is typically set to a magnitude of 0.1 to 5 pixels. If the test fails, the status of the associated camera is set to "Uncalibrated".

[0043] If the calibration test fails for a single camera but not for others, it can be concluded that a local disturbance only affected that single camera. Under the (realistic) assumption that only the camera's orientation has changed, the camera can be recalibrated in real time.

[0044] First, since neither the scene points nor the camera position change, the distance r from the camera to each 3D scene point i Ensure that remains constant. The camera coordinates can be calculated using:

number

[0045] It is also useful to detect trends in slowly changing camera orientation: if such a trend exists, the root cause is likely to be mechanical stress or temperature, when the reprojection error exceeds a threshold. A linear array of five cameras may exhibit the following spatial pattern of errors:

number

[0046] Strong winds, cheering crowds (stadium vibration), or a passing truck can cause the cameras to change orientation. These root causes typically affect all cameras but only temporarily affect the orientation of all cameras. These types of disturbances temporarily increase the reprojection error for all cameras, but due to the elasticity of the system, it is possible for the reprojection error to decrease again (e.g., when people stop cheering). In this situation, where the reprojection error for all cameras is only slightly above the threshold, we can choose to take no action at all; i.e., we tolerate this error and pass this information downstream to the view synthesis and analysis module. Running the same analysis after the game on the log data again provides valuable insight into whether to make mechanical changes to the camera configuration (e.g., increasing the mass of each camera or changing mounting points on the stadium's existing infrastructure). Spatiotemporal analysis (e.g., Fourier transform, spectral analysis) can be used to classify which events occurred.

[0047] While spatiotemporal patterns in reprojection error are likely informative enough to perform status classification, it can be even more informative to observe the signal of camera orientation changes as a function of space and time. This can reveal the specific rotational motions experienced by the camera rig. For example, if mechanical flexibility is greater in the vertical direction, the motion is primarily around the horizontal axis. Therefore, such data can provide ideas for improved camera mounts for future captures.

[0048] Some applications require all cameras in a camera array to be oriented in real time toward the action taking place on the playing field. A control algorithm then continuously controls each camera's servo to best orient each camera toward the action. In this situation, a rotation matrix estimate must be updated in real time for each camera. In this case, performing status classification becomes more complex. To classify the status, the camera orientation is continuously predicted using the servo control signals. The orientation prediction is then used to update the view matrix, and the reprojection error is evaluated for the updated view matrix. External effects such as wind can be classified via the reprojection error. However, it is also possible to use a pan-tilt system to compensate for errors introduced by wind, for example. In that case, the control signal is generated to not only orient the camera toward the action, but also to have a component that compensates for wind in real time. Estimating the effect of wind and controlling for it can be performed using a Kalman filter (see Wikipedia).

[0049] It is emphasized that the present invention is not limited to sporting events but can be applied to many different applications, for example it can be applied to shopping centers, marketplaces, the automotive industry, etc. It is also not limited to outdoor activities and can be useful for example for real estate brokers, medical staff in hospitals, etc.

[0050] With regard to the automotive industry, the present invention can be applied to automobiles, for example. For example, a camera array can be installed on the rear bumper. These cameras can replace the interior and / or exterior mirrors. In the case of a car with a camera array built into the rear of the car looking backward, the road is constantly moving in the image. However, if the speed of the car is known, the road portion of the image can be made stationary. In that case, the situation is the same as a camera array looking at a stationary sports field. For example, vibrations of the car due to a gravel road cause small changes in the camera rotation with high temporal frequencies. These can be detected and corrected for using a propped approach. Recognition / classification of road type (flat highway vs. gravel road) aids the real-time calibration process. An initial calibration begins when the car turns onto another road. As the car drives onto a new road, the reference frame is continuously updated (tracked) while the new road is visible during driving.

[0051] Referring to the tenth embodiment (claim 10), step F: In the context of an automobile, the (mentioned) low-frequency event is an image of a road moving in a translating manner below the automobile, and the (mentioned) high-frequency event is vibrations caused by gravel roads or road bumps. Note that gravel roads and road bumps are expected to also cause changes in the relative orientation / position between the bumper-mounted cameras. When driving on a smooth highway, these relative changes are expected to be small. Because automobile bumpers have elastic properties, the calibration can be temporarily turned off (due to road bumps) and then recover by itself (due to the flexible structure of the automobile bumper).

[0052] It should be noted further that any kind of camera formation can be used, for example, it is also possible to use multiple drones, each equipped with one or more cameras. A great advantage of drones is that the relative positions of the cameras can be very easily adapted. The term "formation" can be considered similar to the terms "arrangement" or "group," etc.

[0053] Among many VR / AR applications (e.g., with a computer or VR goggles), the present invention can also be used in both 2D or 3D applications.

[0054] The present invention can generally be summarized by: Embodiments (1-12): 1. A method for calibrating at least one of six degrees of freedom of all or a portion of a camera in a formation positioned for scene capture, the method comprising an initial calibration step prior to scene capture, the step including creating a reference video frame including a reference image of a fixed reference object, and during scene capture the method further comprising: a further calibration step in which the positions of the reference images of the fixed reference objects in the captured scene video frames are compared with the positions of the reference images of the fixed reference objects in the reference video frames; - adapting, if necessary, at least one of the six degrees of freedom of the plurality of cameras in the formation in order to obtain an improved scene capture after further calibration. 2. The method of embodiment 1, wherein the "at least one" is "three." 3. The method of embodiment 2, wherein the three degrees of freedom are yaw, roll, and pitch. 4. The method of any preceding embodiment, wherein the adapting step is performed only when the outcome of the comparison exceeds a threshold. 5. Only a first portion of the camera is calibrated in the initial calibration and the further calibration, and a remaining second portion is - First calibration and alternate calibration only, or - Alternative calibration only, calibrated only with one of the 10. The method of any preceding embodiment, wherein the alternative calibration of a camera belonging to the second portion is performed by comparison with one or more adjacent cameras of the first portion. 6. The method of embodiment 5, wherein the second portion of cameras is distributed evenly along all cameras in the formation. 7. The method of embodiment 6, wherein the comparison with one or more adjacent cameras is performed using interpolation and / or extrapolation techniques. 8. The method according to any of the preceding embodiments, wherein the initial calibration is continuously repeated autonomously or upon a specific event. 9. A method according to any preceding embodiment, wherein the method further comprises: - Step A: Analyze which cameras in the formation are applied, Step B: Analyze which of the six degrees of freedom of the adapted camera have been adapted; Step C: determining the corresponding values ​​and timestamps of the adaptations; - Step D: By analyzing the information obtained from steps A, B and C, multiple patterns along the camera formation are recognized. Step E: Classifying the recognized patterns, said classification indicating different root causes of camera disturbances in one or more of the six degrees of freedom of the camera of said formation. 10. The method according to embodiment 9, further comprising: Step F: In step E, multiple analysis sessions of steps A, B, C and D are taken into account. 11. A formation of cameras arranged for scene capture, wherein at least one camera has at least one of six degrees of freedom, the at least one camera is initially calibrated before the scene capture, during the initial calibration a reference video frame is created including a reference image of a fixed reference object, during the scene capture the at least one camera is further calibrated, the position of the reference image of the fixed reference object in the captured scene video frame is compared with the position of the reference image of the fixed reference object in the reference video frame, and at least one of the six degrees of freedom of the at least one camera in the formation is adapted if necessary to obtain an improved captured scene. 12. A computer program comprising adapted computer program code means for performing the method according to any of embodiments 1 to 10 when said computer program is run on a computer. Further details are provided in the appended claims.

[0055] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0056] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprise" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

Claims

1. 1. A method for calibrating at least one degree of freedom out of six degrees of freedom of all or some of a plurality of cameras arranged for scene capturing, comprising: an initial calibration step prior to said scene capturing, said step comprising generating a reference video frame including a reference image of a fixed reference object; During scene capturing, the method comprises: a further calibration step in which the position of a reference image of the fixed reference object in the captured scene video frame is compared with the position of the reference image of the fixed reference object in the reference video frame; and adapting the at least one degree of freedom of the six degrees of freedom of at least one camera of the plurality of cameras to obtain improved scene capturing after the further calibration, the method further comprising: A step A of analyzing which degrees of freedom of the plurality of cameras have been adapted in the adapting step; a step B of analyzing which of the six degrees of freedom have been adapted in the adapting step; a step C of determining a corresponding value and timestamp of said adaptation; Step D of recognizing a plurality of patterns of said adaptation along said plurality of cameras by analyzing information obtained from steps A, B and C; E. classifying the recognized patterns, the classification indicating different root causes of camera disturbances in one or more of the six degrees of freedom of the multiple cameras.

2. The method described in claim 1, wherein in step E, the multiple patterns obtained as a result of repeating steps A, B, C and D multiple times are classified.

3. A method as described in claim 1 or 2, wherein three of the six degrees of freedom are calibrated.

4. The method of claim 3 , wherein the three degrees of freedom are yaw, roll, and pitch.

5. 5. The method of claim 1, wherein the step of adapting is performed only if at least one 2D image location in the set of 2D feature point image locations changes, whereby the magnitude of this change exceeds a threshold.

6. Only a first group of the plurality of cameras is calibrated by the initial calibration and the further calibration, and a second group of the remaining cameras is the first calibration and the alternate calibration only; or Alternate calibration only, calibrated only by either 6. The method of claim 1, wherein the alternative calibration of a camera belonging to the second group is performed by comparison with one or more nearby cameras of the first group.

7. The method of claim 6 , wherein the second group of cameras is evenly distributed within the arrangement of multiple cameras.

8. The method according to claim 1 , wherein the initial calibration is continuously repeated autonomously or upon specific events.

9. 1. A capture device including a plurality of cameras arranged for scene capturing, at least one camera having at least one degree of freedom out of six degrees of freedom, the at least one camera being initially calibrated prior to the scene capturing, during the initial calibration a reference video frame having a reference image of a fixed reference object being created, during the scene capturing the at least one camera being further calibrated to compare a position of the reference image of the fixed reference object in a captured scene video frame with a position of the reference image of the fixed reference object in the reference video frame, and the at least one degree of freedom out of the six degrees of freedom of the at least one camera of the plurality of cameras being adapted to obtain an improved captured scene.

10. A computer program which, when executed by a computer, causes the computer to carry out the method of any one of claims 1 to 8.

Images