Self-calibration for hybrid camera array

Abstract

A method for calibrating a camera pose in a hybrid camera array, the hybrid camera array comprising two or more color sensors and one or more depth sensors. The method includes: obtaining a depth map for each of the depth sensors; obtaining a set of images from the color sensor; and generating an indefinite scale camera pose and an indefinite scale point cloud for a color sensor using the set of images. The one or more depth maps are then used to estimate scaling of the indefinite scale camera pose and the indefinite scale point cloud.

Description

Technical Field

[0001] The present invention relates to the field of multi-view imaging. In particular, the present invention relates to calibrating camera poses in a hybrid camera array. Background Art

[0002] 3D scene reconstruction with multiple cameras can benefit from using one or more depth sensors as part of a color camera array. Among the different depth sensing principles, indirect time-of-flight (iToF) has become popular.

[0003] Hybrid depth sensing devices typically consist of a color camera, an infrared projection mechanism, and a depth sensor camera built into a single device. Microsoft's Azure Kinect (trademark) is an example of such a device. Intrinsic parameters such as focal length, primary focus, and lens distortion are typically provided by the manufacturer for both the color camera and the depth sensor. External parameters describing the relative position and rotation between the color camera and the depth camera of a single device are also provided. The device is typically designed to minimize the baseline between the color sensor and the depth sensor inside its device in order to minimize parallax errors. This is important because depth maps are conventionally represented in the coordinates of the image sensor and vice versa.

[0004] Figure 1 An exemplary hybrid camera array is shown. The hybrid camera array includes three hybrid devices 102. Each hybrid device 102 includes a color sensor 104 and a depth sensor 106. The distance between the color sensor 104 and the depth sensor 106 of the hybrid device 102 (i.e., the baseline between these sensors) is shown by arrow 108. The distance between the color sensors 104 of two different devices (i.e., the baseline between the color sensors) is shown by arrow 110.

[0005] When using a single mixing device 102, factory calibration will suffice for most applications. However, for applications that require capturing scenes from different orientations, it is necessary to know the extrinsic parameters (rotation and translation) between multiple different mixing devices 102. One approach could be to use a calibration pattern. However, this requires a calibration pattern which is not practical in most cases.

[0006] Another approach could be to use depth maps for calibration. However, due to the lack of spatial details and the presence of noise in the depth map, this geometry-based approach may not be very accurate.

[0007] Existing structure from motion algorithms using visual feature point detection can provide self-calibration without the need for a calibration pattern that uses only visual images from a color sensor. Years of research have made these structure from motion tools very accurate and robust. However, structure from motion algorithms alone cannot solve the calibration problem for hybrid camera arrays.

[0008] The first problem is that structure from motion relies on matching features between views, so the views need to be of the same modality (e.g., grayscale or color images). Inputting images and depth maps into structure from motion algorithms will not work because the features cannot be accurately matched across different modalities.

[0009] The second problem is that structure from motion provides calibration results that only up to ambiguous or unknown scaling factors (i.e., camera rotation and translation in a common coordinate system), with the same arbitrary dimensionless units for camera position and point cloud point positions. For many applications, this is not a problem, since the scale parameter is arbitrary and knowledge of the scale is not required to construct 3D geometric models (such as meshes) and to texture the models. However, in hybrid camera arrays including depth sensors that provide depth maps in metric units (e.g., mm), this is a problem, since we cannot use, for example, the depth at a given pixel in each depth camera and warp it to one of the cameras of another device (color or depth). For this, we need to know the camera / device translation in metric units.

[0010] Therefore, there is a need for an accurate self-calibration method when used with a hybrid camera array.

[0011] Affzal Hasan et al.: “RGB-D Multi-view System Calibration for Full3DScene Reconstruction”, 18 th International Conference on Pattern Recognition, DOI: 10.1109 / ICPR.2014.425, discloses combining bundle adjustment and iterative closest point algorithm to estimate the pose parameters of cameras in a multi-view system.

[0012] Yoon Byung-Hyun et al.: “Targetless Multiple Camera-LiDAR Extrinsic Calibration using Object Pose Estimation”, 2021 IEEE Conference on Robotics and Automation (ICRA), DOI: 10.1109 / ICRA48506.2021.9560936, discloses the use of object pose estimation to calibrate the external parameters between multiple cameras and LIDAR sensors. Summary of the invention

[0013] The invention is defined by the claims.

[0014] According to one aspect of the present invention, a method for calibrating camera poses in a hybrid camera array is provided, wherein the hybrid camera array includes two or more color sensors and one or more depth sensors, the method comprising:

[0015] obtaining a depth map for each of the one or more depth sensors and a set of images from the color sensor;

[0016] generating a scaled camera pose and a scaled point cloud for the color sensor using the set of images; and

[0017] The one or more depth maps are used to estimate a scale of the unscaled camera pose and the unscaled point cloud.

[0018] Scaling can be used to calibrate the unscaled camera pose. In other words, scaling can be used to obtain a camera pose that can provide the true distance between the color sensor and / or the depth sensor. The camera pose can then be used, for example, for multi-view image / video encoding and / or for novel view synthesis.

[0019] The unscaled camera pose provides a rotation between the color sensors; and also provides a translation between the color sensors, but only provides an unknown scale factor. The unscaled point cloud provides a measure of depth from the image set, but only provides an unknown scale factor. The depth map provides a metric measure of depth (i.e., depth values ​​calibrated to real-world units (e.g., mm)). Thus, it has been recognized that the unscaled point cloud can be compared to the depth map to estimate the scale of the unscaled point cloud (and therefore the unscaled camera pose).

[0020] Generating the unscaled camera pose and the unscaled point cloud may include inputting the set of images into a structure from motion algorithm, wherein the structure from motion algorithm is configured to output an unscaled camera pose for the color sensor and, optionally, an unscaled point cloud.

[0021] Structure from motion algorithms or SfM algorithms are often used to extract 3D structure from 2D images. For example, the movement of a color sensor can be inferred at scale based on images taken during movement. Similarly, images of a scene from various color sensors at different positions can also be used instead of using the movement of a single color sensor.

[0022] The structure from motion algorithm can be configured to output both an unscaled camera pose and an unscaled point cloud. However, if the structure from motion algorithm does not output an unscaled point cloud or is not desired, a new (or additional) unscaled point cloud can be obtained from the unscaled camera pose using, for example, feature detection and triangulation.

[0023] Estimating the scale of the unscaled camera pose and the unscaled point cloud may include obtaining a plurality of suggested scales, and for each suggested scale: computing a suggested camera pose and a suggested point cloud by applying the suggested scale to the unscaled camera pose and the unscaled point cloud, respectively, comparing the suggested point cloud to one or more depth maps; and determining one or more error metrics based on the comparison. The scale may be estimated based on the suggested scale having the lowest one or more error metrics.

[0024] This may be viewed as a "loop" implemented for various proposed ratios, wherein at least one error metric is determined for each proposed ratio.

[0025] It should be understood that there are various methods for obtaining multiple suggested ratios. In a first example, the suggested ratios may be randomly assigned. In a second preferred example, an educated guess may be made for one or more of the suggested ratios, and other values ​​may be derived from nearby educated guesses (e.g., ±50%). Similarly, the "next" suggested ratio in the cycle used may be based on an error metric of the previously suggested ratio.

[0026] In a first comparison method, comparing the proposed point cloud to one or more depth maps may include projecting the proposed point cloud to a view of one or more depth sensors using a proposed camera pose, evaluating multiple depth-based errors between depth values ​​in the one or more depth maps and corresponding depth values ​​resulting from projecting the proposed point cloud, wherein determining one of the error metrics is based on the multiple depth-based errors.

[0027] Therefore, the scale can be found by minimizing the error between the predicted depth associated with the projected point and the measured depth map. It has been found that this greatly increases the accuracy and robustness of the estimated scale.

[0028] In a second comparison method, comparing the proposed point cloud may include: projecting the proposed point cloud to views of two or more color sensors using a proposed camera pose to generate projected points; identifying depth map values ​​of one or more depth maps corresponding to feature points of the proposed point cloud; projecting the depth map values ​​to views of the two or more color sensors using the proposed camera pose; and, in an imaging plane of each of the two or more color sensors, evaluating multiple pixel-based errors between pixel coordinates generated by the projected depth map values ​​and pixel coordinates generated by the projected proposed point cloud, wherein determining one of the error metrics is based on the multiple pixel-based errors.

[0029] Using pixel-based errors rather than depth-based errors is often more intuitive to the user. Additionally, pixel-based errors somewhat mimic the process of synthesizing new views in multi-view imaging. This can provide improved results when synthesizing new views because the errors have already been minimized between existing views of the color sensor.

[0030] Estimating the scale of the indefinite-scale camera pose and the indefinite-scale point cloud may also include, for each proposed scale: determining whether one or more projected points in the projected point cloud are occluded in the view of any one of the depth sensor and / or the color sensor, and based on the specific projected point being occluded, weighting a lower error in one or more error metrics corresponding to the specific projected point.

[0031] The point cloud may include feature points of the back or side of an object that is not visible from the view of the depth sensor. Therefore, it is preferred to reduce the weight of the occluded points in the sum of the errors.

[0032] For the first comparison method, determining whether one or more projected points in the projected point cloud are occluded may include determining whether a depth-based error between a depth of the one or more projected points in the projected point cloud and a depth of the depth map exceeds a depth-based occlusion threshold.

[0033] For the second comparison method, determining whether one or more projected points in the projected point cloud are occluded may include determining whether a pixel-based error between pixel coordinates of the projected points and pixel coordinates of the depth map points exceeds a pixel-based occlusion threshold.

[0034] Determining whether one or more projected points in the projected point cloud are occluded may include identifying projected points within a local neighborhood of one or more central projected points, and for each local neighborhood, determining whether a depth-based error and / or a pixel-based error exceeds a neighborhood threshold, wherein the neighborhood threshold optionally depends on the depth of the identified projected points in the local neighborhood.

[0035] Determining whether one or more projected points in the projected point cloud are occluded may include, for each projected point: identifying a pixel color for each view of one or more depth sensors and / or two or more color sensors based on the image set; and comparing the pixel colors from each view.

[0036] Determining whether the projected point is occluded may include determining whether a difference between pixel colors exceeds a pixel color threshold.

[0037] The method may also include: generating a second point cloud using the one or more depth maps; and, for each suggested scale: projecting the second point cloud to the view of the one or more depth sensors using the suggested camera pose; and evaluating a second plurality of depth-based errors between depth values ​​in the one or more depth maps and corresponding depth values ​​resulting from the projection of the second point cloud, wherein determining one of the error metrics is based on the second plurality of depth-based errors.

[0038] The different error metrics may be weighted based on one or more of: the number of points in the proposed point cloud, the number of points in the second proposed point cloud, and the relative importance between the proposed point cloud and the second proposed point cloud.

[0039] The present invention also provides a computer program product comprising computer program code, which, when executed on a computing device having a processing system, causes the processing system to execute all the steps of the aforementioned method.

[0040] The present invention also provides a system for calibrating camera postures in a hybrid camera array, the system comprising:

[0041] a hybrid camera array comprising two or more color sensors and one or more depth sensors; and

[0042] Processor for:

[0043] obtaining a depth map from each of the one or more depth sensors and a set of images from the color sensor;

[0044] generating a scaled camera pose and a scaled point cloud for the color sensor using the image integration; and

[0045] The one or more depth maps are used to estimate a scale of the unscaled camera pose and the unscaled point cloud.

[0046] It should be understood that the processor of the system may also be configured to perform any of the aforementioned method steps.

[0047] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] For a better understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0049] Figure 1 An exemplary hybrid camera array is shown;

[0050] Figure 2 shows the self-calibration process of the hybrid camera array;

[0051] Figure 3 and Figure 4 illustrates a scale estimation process using a single depth sensor and a first scale parameter and a second scale parameter, respectively;

[0052] Figure 5 and Figure 6 illustrates the scale estimation process using two depth sensors;

[0053] Figure 7 Figure 1 illustrates the scale estimation process when dealing with occluded points.

[0054] Figure 8 Illustration of occlusion handling during scale estimation using local neighborhoods;

[0055] Fig. 9 illustrates occlusion handling during ratio estimation using pixel color; and

[0056] Fig.10 The scale estimation process using pixel-based errors is illustrated. DETAILED DESCRIPTION

[0057] The present invention will be described with reference to the accompanying drawings.

[0058] It should be understood that the detailed description and specific examples are intended for illustrative purposes only while indicating exemplary embodiments of the device, system and method, and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, system and method of the present invention will be better understood through the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar components.

[0059] The present invention provides a method for calibrating camera poses in a hybrid camera array, the hybrid camera array comprising two or more color sensors and one or more depth sensors. The method comprises: obtaining a depth map for each of the depth sensors; obtaining a set of images from the color sensors; and using the set of images to generate an unscaled camera pose and an unscaled point cloud for the color sensors. Then one or more depth maps are used to estimate the scale of the unscaled camera pose and the unscaled point cloud.

[0060] Existing structure from motion software can output a point cloud along with the expected camera pose (i.e., orientation and translation). Known packages with structure from motion software include AliceVision and Colmap. The point cloud is often used internally to solve the structure from motion problem in a process known as bundle adjustment. The points in the point cloud correspond well to detectable features in the image.

[0061] Although this point cloud is also known only at indefinite scale, it provides an unexpected alternative to depth warping and image difference computation methods. The points in this point cloud consist of well-matched and distinguishable features with sufficient local image texture variation. Furthermore, since the points in the point cloud coincide with matched features on multiple views, there will be few occluded features, since these exist due to the combination of textures at different depth levels.

[0062] Based on this implementation, it is proposed to solve the hybrid camera self-calibration problem by first running the structure from motion algorithm on the color image and then, as a second step, using the point cloud output by the structure from motion algorithm to determine the unknown scale parameter by comparing it with information from (one or more) depth maps.

[0063] For a given scale, the point cloud can first be converted to a metric point cloud. The metric point cloud is then warped to all views containing a depth map. The observed depth values ​​from the depth map are then compared to the depth values ​​of the projected points in view coordinates. Thus, the error metric is a geometric metric (e.g., rather than an image-based metric).

[0064] Figure 2 The self-calibration process of a hybrid camera array is shown. A sequence of color images 202 is obtained from the color sensor, and a depth map 204 is obtained from the depth sensor. The structure from the motion algorithm 206 is used to estimate the scaled camera pose 208 of the color sensor, including the rotation matrix and [R1, ..., R N ] unscaled translation vector [st1, ..., st N]. The camera pose of the sensor can be used to warp the image to other views. However, in order to warp the image to views corresponding to different sensors, a scale-resolved metric translation vector [t1, ..., t N ].

[0065] The structure from motion algorithm 206 also outputs an unscaled point cloud [sx1, ..., sx M ] 210. This data may be passed to a scale estimation algorithm 212 which uses the metric depth map 204 to determine the scale 214 of the unscaled camera pose and the unscaled point cloud via a scale parameter s.

[0066] The exemplary ratio estimation algorithm 212 has the following steps:

[0067] 1. Input - [R1, ..., R N ],[st1,...,st N ] and [sx1, ..., Sx M ].

[0068] 2. Input - A rough estimate of the average distance between color sensors.

[0069] 3. Calculate the scale parameter s based on the input average distance between the depth sensors init A rough initial estimate of .

[0070] 4. For the proposed ratio s, from s = s in steps of Δs start =f start s init to s=s stop =f stop s init , perform the following steps:

[0071] 4.1 Solve [x1, ..., x using the proposed ratio s M ]

[0072] 4.2 Solve [t1, ..., t using the proposed ratio s N ]

[0073] 4.3 Replace [x1, ..., x M ] is projected onto [t1, ..., t N ] of depth sensors [1, ..., N].

[0074] 4.4 Evaluate the following error metrics:

[0075]

[0076] in:

[0077] -

[0078] - is the depth of the projected point j from view i at a given proposed scale s.

[0079] - is the depth map value from view i in the direction of point j.

[0080] - and are the coordinates of the depth map at view i corresponding to projection point j.

[0081] 5. Determine the value s=s corresponding to the minimum value E(s) min .

[0082] 6. Return the value corresponding to s min The calibrated translation [t1,…,t N ].

[0083] The minimum value E(s) can be obtained by using the threshold minimum value E min In this case, if E(s) <E min , the threshold minimum value can be updated to a lower value.

[0084] Figure 3 and Figure 4 The scale estimation process using a single depth sensor 306 and a first scale parameter and a second scale parameter are illustrated, respectively. Projected points 302 correspond to cloud points projected to the depth sensor 306 using corresponding scale parameters. Metric depth map 304 is a depth map observed by depth sensor 306.

[0085] exist Figure 3 , the selected first scale parameter is too small, resulting in a large error 308 between the predicted depth of the projected point 302 and the observed depth at the corresponding position on the depth map 304. The corresponding position on the depth map is the point on the depth map that intersects the vector formed by the projected point 302 and the origin of the depth sensor 306.

[0086] exist Figure 4 , the selected second scale parameter corresponding to a larger zoom shows a smaller error 402 and therefore provides a more accurate estimate of the metric translation vector.

[0087] Figure 5 and Figure 6A scale estimation process using two depth sensors 506a and 506b is shown. In this case, a plurality of first errors 508a are calculated between the projected point 502 and the depth map 504 from the first depth sensor 506a, and a plurality of second errors 508b are calculated between the projected point 502 and the depth map 504 from the second depth sensor 506a. The plurality of first errors 508a and the plurality of second errors 508b are summed to determine a total error corresponding to each proposed scale s. Therefore, the second depth sensor 506b now also contributes to the total error via its own observed depth map.

[0088] exist Figure 5 In , the selected scaling parameter is too small, resulting in large errors 508a and 508b. Figure 6 , the selected scale parameters show smaller errors (only error 602 is visible), and therefore provide a more accurate estimate of the metric translation vector.

[0089] Figure 7 The scale estimation process is illustrated when dealing with occluded points 710. Since the point cloud is sparse, by definition, projected points 702 from the back side of the object will incorrectly contribute to the error calculation.

[0090] This can be Figure 7 , where projected point 710 corresponds to the back side of the object represented by depth map 704. This results in an erroneous error 708 between the point of the depth map visible to depth sensor 706 and projected point 710 contributing to the overall error metric.

[0091] A solution could be to assume that objects have a minimum size and exclude errors above a given occlusion threshold t (e.g., t=0.5 m). The corresponding error metric could therefore be:

[0092]

[0093] in:

[0094]

[0095] Other methods for handling occlusion may also be used to suppress errors, such as using local occlusion reasoning between points in the projected view.

[0096] Figure 8 Illustrated is occlusion handling in a scale estimation process using a local neighborhood 810. A local neighborhood 810 is provided around a given projection point 808 selected from a plurality of projection points 802. The local neighborhood 810 of the given projection point 808 may be defined as all other points that are closer (measured in image space) than a given distance expressed in pixels (e.g., 10 pixels).

[0097] If the projected point 808 has a depth greater than the neighborhood threshold, then compared to all neighboring points, it can be concluded that the projected point 808 coincides with a point on the object that is not visible from the depth sensor 806 (e.g., the back side of the object). Therefore, the error between the projected point 808 and the observed depth map 804 will not be accumulated in the error sum.

[0098] Alternatively or additionally, pixel colors may be compared between views to determine whether a projected point is occluded for a given view.

[0099] Fig. 9 Illustrated is the handling of occlusion in the process of ratio estimation using pixel color. If for a given projection point in a given view, the observed color is significantly different from all color values ​​observed in other views (for the same projection point), then the projection point is likely occluded, and thus its contribution to the error sum can be reduced or removed entirely. For example, a weighting can be added to the error sum based on pixel color comparison.

[0100] exist Fig. 9 , two projected points 902a and 902b are shown. Projected point 902a can be seen by all three depth sensors 906a, 906b, and 906c, and therefore the corresponding pixel color (e.g., obtained by warping the color image from the color sensor to the depth sensor) should be similar for all three views. However, projected point 902b can be seen by depth sensors 906a and 906b, but not by depth sensor 906c. Therefore, depth sensor 906c "sees" the pixel color of projected point 902b as the color at point 908. Therefore, if the colors of points 902b and 908 do not match, the error of point 902b from depth sensor 906c can be ignored in the total error calculation.

[0101] Visual feature points (i.e., points in the point cloud) output by the structure from motion algorithm typically correspond to image regions that show spatial texture variation (e.g., corners.) Therefore, image regions that do not have much texture or texture variation will not generate feature points and are therefore not used in the scale estimation process.

[0102] To address this issue, the observed depth map can be used as a source of additional point cloud data. The depth maps for all views can be transformed into a common world space coordinate system using the proposed scale and camera pose. This gives a depth map-based point cloud whose size is the number of views multiplied by the number of depth map pixels per view. For a given proposed scale, two types of point clouds are now available: a scaled version of the point cloud provided by the structure from motion algorithm and a point cloud derived from the depth map.

[0103] Since both the origin and number of points in each type of point cloud are significantly different, it can be beneficial to normalize each type by the number of points and weight their contributions separately:

[0104]

[0105] wherein α (between 0 and 1) weights the relative importance of the point cloud provided by the structure from motion. The particular value of α may be selected by a skilled artisan (eg, via experimentation and / or trial and error).

[0106] Back to Figure 1 , the baseline 108 between the color sensor 104 and the depth sensor 106 in the mixing device 102 may be known (eg, from the factory).

[0107] The baseline length 110 between the color sensors may be estimated. In fact, the average baseline length 110 between all color sensors may be estimated. The baseline length 110 may then be used to calculate a rough estimate of the scale parameter. The range of suggested scales (around the rough estimate of the scale parameter) may then be used in the scale estimation process as described above.

[0108] In an example, a rough estimate of the scale parameter can be calculated by dividing the estimated baseline length 110 by the average of the shortest distances between camera poses. If the color sensors are somewhat evenly spaced in 3D space (i.e., the distance between the color sensors does not vary significantly), the rough estimate of the scale parameter can be within 10% of the true scale parameter. Therefore, under the assumption of evenly spaced cameras, the range of recommended scales can be within 10% of the rough estimate. However, it should be understood that in practice, such an assumption may not always be correct. Therefore, it has been found that a reasonable range of recommended scales can be within 50% of the rough estimate. Of course, the exact range may depend on the positioning of the color sensors, the available processing resources, and the preferences of the technician.

[0109] The range of suggested ratios includes multiple suggested ratios.

[0110] In an example, a rough estimate may be a guess (eg, random or based on previous ratios). However, this may require more iterations / a larger range of suggested ratios, and in some cases, may diverge towards the wrong ratio.

[0111] As will be appreciated, the aforementioned error metric is based on a depth-based error contribution. In other words, the error metric is based on the difference in depth values. However, many users may be more familiar with pixel-based errors (e.g., distances in pixels). Therefore, a second solution is proposed in which the error metric is determined using pixel-based errors. The sum of the residual pixel-based errors (after the optimal ratio has been estimated) can be used as a quality metric, regardless of whether a depth-based or pixel-based error metric is used to guide the selection of the optimal ratio.

[0112] Fig.10 The scale estimation process using pixel-based error 1008 is illustrated. Feature point 1002 is a feature point of the proposed scaled point cloud. Point 1004 is a point inferred from the depth map by identifying the depth value 1003 corresponding to the line of sight as the feature point 1002. This will be referred to as a depth map point. It has been "unprojected" from the imaging plane 1010a of the first color sensor 1006a into 3D space using the known depth value 1003 from the depth map. (Here "unprojected" refers to the process of inferring a 3-D point from a depth value in a 2-D depth map.) As can be seen, the depth map point 1004 corresponds to the feature point 1002 of the view from the first color sensor 1006a.

[0113] Using the proposed camera poses of the color sensors 1006a, 1006b, and 1006c, the depth map points 1004 and feature points 1002 are reprojected to imaging planes 1010b and 1010c corresponding to the second color sensor 1006b and the third color sensor 1006c, respectively.

[0114] A pixel-based error 1008 may then be found between the reprojections of the feature point 1002 and the depth map point 1004 to both imaging planes 1010b and 1010c. Similar to the depth-based approach, the pixel-based error may be used to determine an error metric.

[0115] Therefore, instead of comparing the predicted depth value after projection to the observed depth map value, this alternative pixel-based error method projects the depth map point 1004 to various views, where it then compares its pixel position to the direct projection of the corresponding point cloud point. This method makes the total number of errors contributing to the error metric equal to the number of feature points in the point cloud multiplied by the number of cameras minus one.

[0116] Since this error metric involves reprojecting points between views, it is similar to the view synthesis process typically used during rendering of a new view in multi-view imaging. Thus, minimizing the pixel-based error metric may provide a further accurate rendering of the new view, since this should translate into minimization of visible errors in the new view.

[0117] Of course, it is also possible to use both methods to obtain different error metrics. These error metrics will include different dimensions (i.e., pixel distance and depth difference), and therefore combining them may not be desirable unless they are appropriately weighted.

[0118] Similar to depth-based scale estimation, a second point cloud can also be calculated from one or more depth maps, and the second point cloud is used to obtain more depth information of the scene. The use of the second point cloud can be used to determine a second depth-based error metric. It should be noted that pixel-based error metrics cannot be used with a second point cloud calculated only from depth map data. For points in such a point cloud, there is no information about the correspondence of points between views. Therefore, it is not possible to perform a point-by-point comparison. In contrast, correspondence information is inherent in the point cloud generated by the structure from motion algorithm because this is a single "real" 3-D point cloud in 3D coordinates, which is then projected into the 2-D imaging plane of each sensor.

[0119] It should be understood that more than one error metric may be determined for a hybrid sensor array setup.The number of error metrics used and the type of error metrics used may depend on available processing resources and / or user preferences.

[0120] It should be appreciated that any of the above occlusion handling methods may also be used in a pixel-based scale estimation process with minimal variation. For example, the occlusion threshold may be based on distance in pixels rather than distance in a metric measurement.

[0121] Another solution to the self-calibration problem that has been considered is to first run a structure from motion algorithm on the color image and then use the depth map(s) to find the correct scale parameters. In theory, this can be achieved by warping the color image from one device to another based on the depth map and then taking an image difference metric (e.g., mean squared error). Minimizing this difference over all possible warps from one camera to another can then produce the best fitting scale parameters.

[0122] However, this alternative approach may be more sensitive to noise in the depth map, especially for large baselines. Furthermore, in some cases, the use of an image-based difference metric may lead to biases in the scale estimate caused by varying surface reflectance, resulting in color variations in the image. Additionally, this solution may be relatively more computationally intensive, since all pixels in all images need to be warped to all other images at each iteration in the search for the best scale. The alternative solution may be most useful in cases where the baseline is relatively small and the depth map is expected to have a low level of noise.

[0123] As previously described, a point cloud can be obtained using a structure from motion algorithm. In some cases, the structure from motion algorithm will output a point cloud. However, in some cases, it may be preferable to determine a new unscaled point cloud for scale estimation (optionally using the unscaled camera pose output by the structure from motion algorithm). For example, feature detection and matching can be applied to a collection of images, and a new unscaled point cloud can be obtained using triangulation based on the unscaled camera pose. The new unscaled point cloud can be used alone to estimate the scale, or it can be used to enhance the point cloud according to the structure from motion algorithm.

[0124] Those skilled in the art will easily develop a processor for executing any method described herein. Therefore, each step of the flowchart may represent a different action performed by the processor and may be performed by a corresponding module of the processor.

[0125] As described above, the system utilizes a processor to perform data processing. The processor can be implemented in many ways, with software and / or hardware to perform the various functions required. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware that performs some functions and one or more programmed microprocessors and associated circuits that perform other functions.

[0126] Examples of circuits that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0127] In various embodiments, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform desired functions when executed on one or more processors and / or controllers. The various storage media may be fixed within the processor or controller, or may be portable so that one or more programs stored thereon can be loaded into the processor.

[0128] Variations of the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, based on a study of the drawings, the present disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0129] The functions implemented by a processor may be implemented by a single processor or by multiple separate processing units, which may be collectively considered to constitute a “processor.” In some cases, such processing units may be remote from each other and communicate with each other in a wired or wireless manner.

[0130] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0131] The computer program may be stored / distributed on suitable media such as optical storage media or solid-state media supplied together with or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems.

[0132] If the term "suitable for" is used in the claims or description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or description, it should be noted that the term "arranged" is intended to be equivalent to the term "system" and vice versa.

[0133] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method for calibrating camera poses in a hybrid camera array, the hybrid camera array comprising two or more color sensors and one or more depth sensors, the method comprising: obtaining a depth map (204) from each of the one or more depth sensors and a set of images (202) from the color sensor; generating a scaled camera pose (208) and a scaled point cloud (210) for the color sensor using the set of images; as well as The method further comprises estimating the unscaled camera pose and the scale of the unscaled point cloud using the one or more depth maps, wherein estimating the unscaled camera pose and the scale of the unscaled point cloud comprises: The proportion of people receiving multiple recommendations; For each suggested ratio: computing a proposed camera pose and a proposed point cloud by applying the proposed scale to the unscaled camera pose and the unscaled point cloud, respectively; comparing the proposed point cloud to the one or more depth maps; determining one or more error metrics based on the comparing; and The scaling is estimated based on the proposed ratio having the lowest one or more error metrics.

2. The method according to claim 1, wherein: Generating the unscaled camera pose and the unscaled point cloud comprises inputting the set of images into a structure from motion algorithm (206), wherein the structure from motion algorithm is configured to output the unscaled camera pose for the color sensor and, optionally, the unscaled point cloud.

3. The method according to claim 1 or 2, wherein: Comparing the proposed point cloud with the one or more depth maps comprises: projecting the proposed point cloud to a view of the one or more depth sensors using the proposed camera pose; and A plurality of depth-based errors are evaluated between depth values ​​in the one or more depth maps and corresponding depth values ​​resulting from projecting the proposed point cloud (308, 402), wherein determining an error metric in an error metric is based on the plurality of depth-based errors.

4. The method according to any one of claims 1 to 3, wherein: The proposed point clouds for comparison include: projecting the proposed point cloud to views of the two or more color sensors using the proposed camera pose to generate projected points; identifying depth map values ​​of the one or more depth maps corresponding to feature points of the proposed point cloud; projecting the depth map values ​​to the views of the two or more color sensors using the proposed camera pose; and In an imaging plane (1010) of each color sensor of two or more color sensors, a plurality of pixel-based errors (1008) are evaluated between pixel coordinates resulting from projecting the depth map values ​​and pixel coordinates resulting from projecting the proposed point cloud, wherein determining one of the error metrics is based on the plurality of pixel-based errors.

5. The method according to any one of claims 1 to 4, wherein: Estimating the scale of the unscaled camera pose and the unscaled point cloud further comprises, for each proposed scale: determining whether one or more projected points in the projected point cloud are occluded in a view of any of the depth sensor and / or the color sensor; as well as Based on the particular projection point being occluded, a lower error corresponding to the particular projection point among the one or more error metrics is weighted.

6. The method according to claim 3, in, Estimating the unscaled camera pose and the scaling of the upscaled point cloud further comprises, for each proposed scale: determining whether one or more projected points in the projected point cloud are occluded in a view of any of the depth sensor and / or the color sensor; as well as Based on the occluded specific projection point, weighting a lower error corresponding to the specific projection point in the one or more error metrics, and Wherein, determining whether one or more projected points in the projected point cloud are occluded includes determining whether the depth-based error between the depth of one or more projected points in the projected point cloud and the depth of the depth map exceeds a depth-based occlusion threshold.

7. The method according to claim 4, in, Estimating the scale of the unscaled camera pose and the unscaled point cloud further comprises, for each proposed scale: determining whether one or more projected points in the projected point cloud are occluded in a view of any one of the depth sensor and / or the color sensor; as well as Based on the occluded specific projection point, weighting a lower error corresponding to the specific projection point in the one or more error metrics, and Wherein, determining whether one or more projected points in the projected point cloud are occluded includes determining whether the pixel-based error between the pixel coordinates of the projected point and the pixel coordinates of the depth map point exceeds a pixel-based occlusion threshold.

8. The method according to claim 5, 6 or 7, wherein: Determining whether one or more projection points in the projection point cloud are blocked includes: identifying projection points within a local neighborhood of one or more central projection points; and For each local neighborhood, it is determined whether the depth-based error and / or the pixel-based error exceeds a neighborhood threshold, wherein the neighborhood threshold is optionally dependent on the depth of the identified projection point in the local neighborhood.

9. The method according to any one of claims 5 to 8, wherein: Determining whether one or more projection points in the projection point cloud are blocked includes: For each projection point, identifying a pixel color of each view of the one or more depth sensors and / or the two or more color sensors based on the set of images; and For each projection point, the pixel colors from each view are compared.

10. The method according to any one of claims 1 to 9, further comprising: generating a second point cloud using the one or more depth maps; For each suggested ratio: projecting the second point cloud to a view of the one or more depth sensors using the proposed camera pose; A second plurality of depth-based errors are evaluated between depth values ​​in the one or more depth maps and corresponding depth values ​​resulting from the projection of the second point cloud, wherein determining one of the error metrics is based on the second plurality of depth-based errors.

11. A computer program product comprising computer program code, which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of the method according to any one of claims 1 to 10.

12. A system for calibrating camera poses in a hybrid camera array, the system comprising: a hybrid camera array comprising two or more color sensors (104) and one or more depth sensors (106); as well as A processor configured to: Obtaining a depth map (204) from each of one or more depth sensors and a set of images (202) from the color sensor; generating a non-scaled camera pose (208) and a non-scaled point cloud (210) for the color sensor using the set of images; and The processor is configured to estimate the scale of the unscaled camera pose and the unscaled point cloud using the one or more depth maps by: The proportion of people receiving multiple recommendations; For each suggested ratio: computing a proposed camera pose and a proposed point cloud by applying the proposed scale to the unscaled camera pose and the unscaled point cloud, respectively; comparing the proposed point cloud to the one or more depth maps; determining one or more error metrics based on the comparing; and The scaling is estimated based on the proposed ratio having the lowest one or more error metrics.

13. The system according to claim 12, wherein: The processor is configured to generate the unscaled camera pose and the unscaled point cloud by inputting the set of images into a structure from motion algorithm (206), wherein the structure from motion algorithm is configured to output the unscaled camera pose for the color sensor and, optionally, the unscaled point cloud.

Images