System and methods for multi-camera automatic calibration and three-dimensional reconstruction of humans

Abstract

Method and apparatus receive image data captured simultaneously from a plurality of cameras. A human-like object is detected in the image data captured by two or more cameras of the plurality of cameras, and a 3D human body mesh (HBM) of the human-like object is reconstructed to obtain a set of 3D HBMs. 3D HBMs from the set of 3D HBMs that were reconstructed based on images in the image data that were captured at a same time by the two or more cameras are matched to obtain one or more matched pairs of the 3D HBMs, and a matched pair of the 3D HBMs is identified from the one or more matched pairs of the 3D HBMs for a camera pair from the two or more cameras. Pairwise camera calibration is performed for the camera pair using the matched pair of the 3D HBMs.

Description

System and Methods for Multi-Camera Automatic Calibration and Three-Dimensional Reconstruction of HumansCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This filing claims priority to U.S. Provisional Application Ser. No. 63 / 707,590 filed on October 15, 2024, the disclosure of which is incorporated herein by reference.TECHNICAL FIELD

[0002] Disclosed embodiments relate generally to multi-camera calibration, and more specifically to systems and methods for multi-camera automatic calibration and three-dimensional (3D) reconstruction of humans.BACKGROUND

[0003] Human Pose and Shape Estimation (HPSE) is a field of computer vision and machine learning focused on detecting, reconstructing, and modeling the position, orientation, pose, and 3D body shape of humans from images, videos, or sensor data. HPSE is used in many human-centric applications such as, but not limited to, virtual-reality (VR), augmented-reality (AR), mixed-reality (MR), extended-reality (XR), immersive telepresence, interactive conferencing, metaverse, healthcare monitoring, sports and gait analysis, human activity recognition, and humanoid robots.

[0004] However, the accuracy of deep learning (DL) based monocular 3D HPSE often suffers from self or mutual occlusion and depth ambiguity that can lead to inaccurate estimated body tilt and posture. It may fail for rare or unseen poses, and occluded bodies due to limited training data lacking diversity in human poses, appearances, and camera viewpoints. Additionally, while fisheye cameras are able to provide omnidirectional vision with up to three hundred sixty degree (360°) field of view (FoV) that can cover a given space with fewer cameras, the objects in images captured by these cameras are often distorted. Thus, the use of fisheye cameras or other wide- angle cameras are problematic for HPSE because most DL-based monocular HPSE models are trained using images in perspective view with a normal FoV and having few shape distortions. Multi-camera systems with wide baselines may overcome some of these challenges, but they require accurate and robust calibration that involves a tedious process using special calibration patterns.SUMMARY

[0005] A first aspect relates to a method, implemented by a computing device, for calibrating multiple cameras using 3D human body meshes reconstructed from image data captured simultaneously by the cameras. The method includes receiving image data captured simultaneously from a plurality of cameras, detecting a human in the image data captured by two or more cameras of the plurality of cameras, reconstructing a 3D human body mesh of the human based on detection of the human to obtain a set of recovered 3D human body meshes of the human, matching recovered 3D human body meshes from the set of recovered 3D human body meshes to obtain one or more matched pairs of the recovered 3D human body meshes, identifying a matched pair of the recovered 3D human body meshes from the one or more matched pairs for a camera pair from the two or more cameras, and performing pairwise camera calibration for the camera pair using the matched pair of the recovered 3D human body meshes.

[0006] Optionally, in a first implementation according to the first aspect, performing the pairwise camera calibration includes projecting vertices of the matched pair of recovered 3D human body meshes onto two-dimensional image planes of the camera pair to determine keypoint correspondences, estimating a fundamental matrix for the camera pair using the keypoint correspondences, rejecting outliers from the keypoint correspondences using a Random Sample Consensus (RANSAC) algorithm to obtain inlier correspondences, estimating an essential matrix based on the inlier correspondences, decomposing the essential matrix to obtain candidate rotations and translations, and selecting a physically valid rotation and translation from the candidate rotations and translations for which triangulated points have positive depth in both cameras.

[0007] Optionally, in a second implementation according to the first aspect or any implementation thereof, projecting the vertices includes performing adaptive vertex sampling to ensure uniform coverage of body parts and reduce redundant correspondences.

[0008] Optionally, in a third implementation according to the first aspect or any implementation thereof, estimating the fundamental matrix is performed on normalized image coordinates, and estimating the essential matrix includes performing singular value decomposition with a rank-2 constraint.

[0009] Optionally, in a fourth implementation according to the first aspect or any implementation thereof, the method further includes repeating the performing of the pairwise camera calibration for multiple camera pairs of the plurality of cameras to obtain a set of relativepose parameters between camera pairs, and performing pose accumulation and global optimization to compute globally consistent camera poses for the plurality of cameras and to minimize a reprojection error across the plurality of cameras.

[0010] Optionally, in a fifth implementation according to the first aspect or any implementation thereof, minimizing reprojection error includes performing bundle adjustment to jointly refine relative pose parameters for the plurality of cameras, and enforcing cycle-consistency constraints among the camera poses during the pose accumulation and global optimization.

[0011] Optionally, in a sixth implementation according to the first aspect or any implementation thereof, the plurality of cameras includes fisheye cameras, and the method further includes performing spherical human detection on fisheye images to identify the human within a distorted field of view, mapping a detected region of interest to a perspective image patch by projecting a fisheye image region onto a tangent plane at a viewing direction to define a virtual camera view, and reconstructing the 3D human body mesh using the perspective image patch corresponding to the virtual camera view.

[0012] Optionally, in a seventh implementation according to the first aspect or any implementation thereof, the method further includes deriving a fisheye camera pose from a virtual camera pose by aligning an orientation of the fisheye camera with mapping angles used in the projecting.

[0013] Optionally, in an eighth implementation according to the first aspect or any implementation thereof, the method further includes performing calibration when multiple persons are present by reconstructing respective 3D human body meshes for the multiple persons in images captured by the plurality of cameras, and determining correspondences between the respective 3D human body meshes that represent a same person across different cameras using multi-view matching to produce matched pairs usable for performing the pairwise camera calibration.

[0014] Optionally, in a ninth implementation according to the first aspect or any implementation thereof, determining the correspondences includes computing a geometric affinity matrix and an appearance affinity matrix, combining the matrices to produce a combined affinity matrix, and optimizing a permutation matrix using an alternating direction method of multipliers to enforce correspondence consistency across multiple views.

[0015] Optionally, in a tenth implementation according to the first aspect or any implementation thereof, the geometric affinity matrix is based at least in part on difference betweenpose and shape of human body meshes, and the appearance affinity matrix is based at least in part on learned re-identification features.

[0016] Optionally, in an eleventh implementation according to the first aspect or any implementation thereof, detecting the human comprises processing the image data using a neural network trained to identify human regions or keypoints, and reconstructing the 3D human body mesh includes generating a mesh representation based on estimated human pose and shape parameters using a human mesh recovery model and a parametric human body model.

[0017] A second aspect relates to an apparatus comprising a memory configured to store instructions and one or more processors coupled to the memory and configured to execute the instructions to perform the method according to the first aspect or any implementation thereof.

[0018] A third aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium and that, when executed by one or more processors of an apparatus, cause the apparatus to perform the method according to the first aspect or any implementation thereof.

[0019] For clarity, any one of the foregoing aspects may be combined with any one or more of the other foregoing aspects to create a new embodiment within the scope of the present disclosure.

[0020] These and other features, and the advantages thereof, will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF DRAWINGS

[0021] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0022] FIG. 1 is a diagram illustrating an example of a multi-camera system in accordance with the present disclosure.

[0023] FIG. 2 is a diagram illustrating a calibration pattern that is traditionally used in performing multi-camera calibration.

[0024] FIG. 3 is a diagram illustrating multi-camera calibration in accordance with the present disclosure.

[0025] FIG. 4 is a block diagram of a multi-camera automatic calibration system in accordance with an embodiment of the present disclosure.

[0026] FIG. 5 is a diagram illustrating recovered 3D body meshes of a person in accordance with an embodiment of the present disclosure.

[0027] FIG. 6 is a block diagram of a fisheye multi-camera automatic calibration system in accordance with an embodiment of the present disclosure.

[0028] FIG. 7 is a block diagram of a fisheye multi-person multi-camera automatic calibration system in accordance with an embodiment of the present disclosure.

[0029] FIG. 8 is a block diagram of a multi-view matcher in accordance with an embodiment of the present disclosure.

[0030] FIG. 9 is a flowchart illustrating a method for performing pairwise calibration of a multi-camera system using human body meshes in accordance with an embodiment of the present disclosure.

[0031] FIG. 10 is a flowchart illustrating a method for performing pairwise calibration of a multi-camera fisheye system using human body meshes in accordance with an embodiment of the present disclosure.

[0032] FIG. 11 is a diagram illustrating an apparatus according to an embodiment of the present disclosure.DESCRIPTION OF EMBODIMENTS

[0033] It should be understood at the outset that, although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0034] The present disclosure describes various embodiments of systems and methods configured to provide fast and accurate automatic calibration of multi-camera systems utilizing human semantic information resulting from HPSE functions without using predetermined calibration patterns or objects. For instance, the disclosed embodiments include an automaticcalibration method for multi-camera systems that supports reliable image-only two-dimensional (2D) camera to 3D world coordinate transformation of a human body with reduced effects from occlusion and depth ambiguity without the use of extra markers or sensors (i.e., generating and using a human body mesh instead of a calibration pattern for calibrating multi-camera systems). Additionally, some embodiments of the present disclosure enable automatic calibration for hybrid or mixed-type multi-camera systems that include omnidirectional cameras, perspective cameras, wide-angle cameras, hemispherical cameras, and / or spherical fisheye cameras. Further, embodiments of the present disclosure enable automatic calibration for multi-camera systems using a single human body or multiple human bodies located in a common field of view (FoV) of two or more cameras of a multi-camera system.

[0035] FIG. 1 is a diagram illustrating an example of a multi-camera system in accordance with the present disclosure. In the depicted embodiment, the multi-camera system includes 2 cameras (camera 001 and camera 002) that are part of an AR / VR gaming system. In an embodiment, the AR / VR system uses the multi-camera system to capture image data (e.g., video) of a user 010 during gameplay to track the movement / actions of the user 010. While the multicamera system depicted in FIG. 1 only includes 2 cameras, the disclosed embodiments are applicable to more advanced multi-camera systems comprising any number of cameras and may be applied to other environments or applications. For example, other non-limiting applications of multi-camera systems applicable to the disclosed embodiments may include 3D image capturing (e g., recording a sports athlete’s movement for videogame rendering, or recording an actor’s movement for computer-generated imagery (CGI) rendering), AR / VR / MR / XR, immersive telepresence, and security or surveillance systems.

[0036] In some applications, and in particular applications that utilize HPSE for detecting, reconstructing, and modeling the position, orientation, pose, and 3D body shape of humans from images, videos, or sensor data, the multi-camera system is calibrated to avoid depth ambiguity that can lead to inaccurate estimated body tilt and posture. Calibration determines the necessary mathematical parameters to relate points in the 3D world to their corresponding 2D locations in the image plane of each camera. Without accurate and robust calibration, multi-camera systems with overlapping FoV cannot effectively mitigate problems like depth ambiguity, self or mutual occlusion, and out-of-view truncation. Calibration generally involves estimating intrinsic parameters and extrinsic parameters. Intrinsic parameters describe internal properties of a singlecamera such as focal length and distortion parameters. Extrinsic parameters define a camera’s pose (i.e., position and orientation) in the 3D world relative to a fixed coordinate system. Various embodiments of the present disclosure estimate one or more extrinsic parameters (e.g., rotation and translation matrices) assuming one or more intrinsic parameters (e.g., focal length, principal point, distortion) are known or pre-calibrated offline.

[0037] FIG. 2 is a diagram illustrating a calibration pattern 020 that is traditionally used in performing multi-camera calibration. As shown in FIG. 2, when a person holds the calibration pattern 020, a number of the cameras of a multi-camera system may be positioned or oriented in a manner that does not provide a camera view of the calibration pattern 020 (as indicated by an X next to cameras 022, 024, 026, 028, and 030). For example, cameras 024, 026, and 028 may be above and / or behind the calibration pattern 020, and cameras 022 and 030 may be beside and / or behind the calibration pattern 020, instead of those cameras having a clear view of the checkered front face of the calibration pattern 020. Thus, because only a few cameras of the multi-camera system in FIG. 2 are able to provide a camera view of the calibration pattern 020 at the same time (as indicated by a check mark next to cameras 032, 034, and 036), the use of traditional calibration patterns for performing multi-camera calibration becomes problematic due to the difficulty in correspondence matching among inputs from different cameras.

[0038] FIG. 3 is a diagram illustrating multi-camera calibration in accordance with the present disclosure. As shown in FIG. 3, every camera in the multi-camera system used in FIG. 2 is able to capture a view or angle of a person 040. Thus, as stated above, the present disclosure describes various embodiments of systems and methods configured to provide fast and accurate automatic calibration of multi-camera systems utilizing human semantic information without predetermined calibration patterns or objects.

[0039] FIG. 4 is a block diagram of a multi-camera automatic calibration system 400 in accordance with an embodiment of the present disclosure. The multi-camera automatic calibration system 400 is designed to fully utilize human semantic information for calibration without using predetermined patterns or objects. The multi-camera automatic calibration system 400 includes a human estimator 110, a cross-view matching selector 116, a mesh matching unit 120, and a pose accumulation and global optimization module 126.

[0040] As shown in FIG. 4, the multi-camera automatic calibration system 400 receives image data from each camera out of a plurality of cameras (e.g., camera 102, camera 104, camera 106,and camera 108 in FIG. 4). In an embodiment, the cameras are traditional types of cameras that output image data including images that capture a perspective view of a scene based on the pose of the camera. In some embodiments, the image data includes raw or processed, live or saved, videos or still images that are captured by each of the cameras. For example, the cameras are positioned at different locations of a room to capture various angles of a person playing a VR game as described in FIG. 1. Although four cameras are depicted in FIG. 4, the embodiments of the present disclosure are applicable to any number of cameras in a multi-camera system.

[0041] In an embodiment, the image data comprises videos that capture images of a single person (e.g., a Person 8 in FIG. 4) at a particular location. As shown in FIG. 4, based on a camera pose (i.e., location and orientation), the images can capture simultaneous different views of the Person 8 at the location. In an embodiment, the image data is first processed by the human estimator 110. In an embodiment, the human estimator 110 comprises a Human Mesh Recovery (HMR) model 114 and a Skinned Multi-Person Linear (SMPL) model 112. As referenced in the present disclosure, a model is a system, program, algorithm, or mathematical framework that has been trained to make predictions or infer certain outputs based on a given input.

[0042] In an embodiment, the HMR model 114 combined with the SMPL model 112 provides an end-to-end framework for reconstructing a full 3D mesh of a human body from a single RGB image. An RGB image is an image where each pixel is defined by the amount of red, green, and blue colors. In some embodiments, the HMR model 114 is a deep learning model (e.g., a neural network) that has been trained to determine extrinsic camera parameters and SMPL parameters (e g., SMPL pose parameters ( / ) and SMPL shape parameters ( / )) directly from 2D images. The extrinsic camera parameters (TT) may include a translation vector and a rotation matrix for establishing the geometric relationship between the camera and the recovered 3D human body mesh with SMPL parameters (n, ). In an embodiment, the translation vector defines position / location of a camera in 3D space relative to the recovered 3D human body mesh, while the rotation matrix defines the camera's orientation with respect to the recovered 3D human body mesh. In some embodiments, when a camera pose (i.e., location and orientation) in world coordinate of one reference camera is known, the camera poses of all other cameras in the multicamera system can be calculated by using the multi-camera automatic calibration system 400.

[0043] The SMPL parameters are 3D human body parameters defining characteristics of a body such as, but not limited to, a pose and shape. For example, shape parameters define physicalattributes such as height, weight, and body proportions of a person identified in the image data by the HMR model 114. The pose parameters define the articulation, joint rotations, and configuration of the body, governing how the 3D surface of the body deforms with movement. For example, in an embodiment, the output of the HMR model 114 includes the SMPL model parameters for pose ( / / G R24x3x3) and shape (J3 E R10), and the extrinsic camera parameters consist of an orientation matrix R E R3 3and translation vector t E R3. In an embodiment, the SMPL parameters are provided as input to the SMPL model 112.

[0044] In an embodiment, the SMPL model 112 is a parametric human body model such as, but not limited to, extended SMPL-X and upgraded STAR models. Based on the SMPL parameters, the SMPL model 112 is configured to generate and output a 3D human body mesh. In an embodiment, 3D human meshes are estimated within a bounding box containing a detected person in 2D camera coordinates. The 3D human body mesh is composed of a collection of vertices (e.g., 6,890 vertices) representing the person. The number of vertices used to generate a 3D human body mesh is not limited in the present disclosure. In some embodiments, the human estimator 110 is able to reconstruct a full 3D human body mesh even if the person is partially occluded or truncated in the image.

[0045] In the depicted embodiment, the cross-view matching selector 116 receives the recovered 3D body meshes. The cross-view matching selector 116 is configured to select matched 3D body meshes that were generated from images taken at the same time by the different camera views to ensure proper calibration. In an embodiment, the cross- view matching selector 116 is configured to send the matched 3D body meshes in pairs to the mesh matching unit 120 for performing pairwise camera calibration. For example, the matched 3D body meshes of the Person 8 from cam 102 and cam 104 may be one pair, and the matched 3D body meshes of the Person 8 from cam 102 and cam 106 may be a second pair.

[0046] In an embodiment, the multi-camera automatic calibration system 400 is configured to perform pairwise camera calibration using all or some combination of the camera pairs of the multi-camera system. For example, in some embodiments, the multi-camera automatic calibration system 400 may perform pairwise camera calibration using camera pairs that include one reference camera having a known camera pose (i.e., location and orientation) in world coordinate.

[0047] In an embodiment, the mesh matching unit 120 includes a keypoint sampling and correspondence extraction module 122 and a structure-from- motion (SIM) essential matrixestimation module 124. Tn an embodiment, the keypoint sampling and correspondence extraction module 122 is configured to determine corresponding keypoints between the recovered 3D body meshes of the same person between the camera pairs. In an embodiment, the keypoint sampling and correspondence extraction module 122 projects the vertices of each recovered 3D human body mesh onto the 2D image plane because multi-view geometry (e.g., the epipolar constraint) can be applied at the 2D level to match points between cameras. The epipolar constraint states that the image of a 3D point in one camera must lie on a specific line, known as the epipolar line, in the second camera's image plane. In an embodiment, the keypoint sampling and correspondence extraction module 122 is configured to adaptively sample the vertices to select a subset of vertices for use as feature points (i.e., 2D keypoints) in the calibration process. The goal is to accurately establish which keypoints from one camera view match the keypoints from another camera view to ensure that keypoints correspond to the same 3D point in the world. By using mesh vertices, the method greatly reduces the complexity of correspondence matching between keypoints captured by different cameras. The abundance of vertices provides many more chances for correct correspondence matches and typically results in higher accuracy compared to using sparse joints.

[0048] In an embodiment, the SIM essential matrix estimation module 124 uses an SfM algorithm to determine relative camera poses (i.e., extrinsic parameters) of the two cameras in the camera pair based on their 2D keypoints. Examples of SfM algorithms that may be utilized in accordance with one or more embodiments include, but are not limited to, OpenSfM-project (version 0.4) and SfM algorithms that are available in the OpenCV library. In an embodiment, the SfM algorithm computes an essential matrix of the camera pair based on the corresponding 2D keypoints. The essential matrix encodes the geometric relationship between two cameras. In some embodiments, the essential matrix is calculated iteratively (with exit conditions defining when to stop iteration) to minimize a geometric error measure or a reprojection error. In an embodiment, the exit condition is set according to the zero-matrix product rule and / or the matrix singular property. The zero-matrix product rule states that convergence has occurred when the product of two matrices is a zero-matrix, which indicates that further updates will not produce any meaningful changes. The matrix singular property indicates that when a matrix being updated becomes singular (or nearly singular), further updates will not produce any meaningful changes. A singular matrix is a matrix whose determinant is zero, meaning the matrix cannot be inverted.

[0049] In some embodiments, because the 2D keypoint correspondences may contain detection errors, noise, or be inconsistent, the SfM essential matrix estimation module 124 performs the essential matrix calculation within an iteration of a RANSAC algorithm loop or other similar refinement algorithms to remove / exclude outliers in the corresponding 2D keypoints from the essential matrix calculation and choose the most consistent inliers in order to obtain the most accurate essential matrix. Thus, the disclosed embodiments are less susceptible to partially visible human bodies due to self or mutual occlusion and out-of-view truncation, because only reliable 2D keypoints are selected from recovered human meshes with plenty of vertices.

[0050] Based on the calculated essential matrix, the SfM essential matrix estimation module 124 then determines the relative camera poses (i.e., extrinsic parameters including a translation vector (indicating location) and a rotation matrix (indicating orientation) of the cameras relative to each other). However, due to the internal properties of the essential matrix, specifically having two equal singular values and one zero singular value, the essential matrix yields four sets of solutions for the relative camera extrinsic parameters when decomposed. Of the four set sets of solutions, only one solution is physically plausible. In an embodiment, to determine the only physically plausible solution, the SfM essential matrix estimation module 124 performs triangulation using the 2D keypoints to reconstruct a 3D structure. The SIM essential matrix estimation module 124 then checks the reconstructed 3D structure against the four sets of solutions. The correct set of extrinsic parameters (i.e., the correct rotation matrix and translation vector combination) in the four sets of solutions is the set for which all points of the 3D structure have a positive depth. If any point of the 3D structure is behind either camera (i.e., having a negative depth), then that solution is physically impossible and is rejected.

[0051] In an embodiment, the pose accumulation and global optimization module 126 accumulates all the pairwise calibration data to obtain a global optimized calibration results for the multi-camera system as a whole. For example, in some embodiments, the pose accumulation and global optimization module 126 uses a bundle adjustment (BA) algorithm that refines both the 3D structure (the positions of points in 3D space) and the camera calibration parameters so that all the camera projections agree as closely as possible with the observed 2D image points (i.e., the BA algorithm determines camera poses and 3D points that minimize total reprojection error). In the depicted embodiment, the pose accumulation and global optimization module 126 is configured to output the calibration result (i.e., camera pose) for each camera of the multi-camera system.

[0052] In some embodiments, the pose accumulation and global optimization module 126 also outputs one or more calibration performance metrics such as, but not limited to, a reprojection error p. In an embodiment, the reprojection error p is a 2D reprojection error that serves as a metric of how well the estimated / reconstructed 3D structure aligns with the observed image data. For example, in an embodiment, after reconstructing the 3D points in the world coordinate frame by triangulation, the estimated camera projection matrices are used to reproject these 3D points back into a 2D image space. The 2D reprojection error p can be determined as the Euclidean distance between the initially observed 2D points and the reprojected 2D points.

[0053] P || ^estimated Wreprojected || 2 (pixels) (1)

[0054] In other embodiments, the pose accumulation and global optimization module 126 outputs a rotation error i and translation error d. Rotation error ip and translation error d are key metrics used to quantify the discrepancy between estimated and ground truth camera poses. The rotation error ip represents the angular deviation between the estimated and the ground true orientations of the camera in the world frame, typically measured in degrees. The translation error d refers to the Euclidean distance between the estimated and the ground true position vectors of the cameras, expressed in meters within the world frame.

[0055] p angle ( ^estimated, ^groundtruth) (degrees). (2)

[0056] (meters). (3)

[0057] FIG. 5 is a diagram illustrating recovered 3D body meshes of a Person 500 in accordance with an embodiment of the present disclosure. In the depicted embodiment, the Person 500 is captured in a first image at a first camera view by Camera 0. The global orientation of Camera 0 in the 3D space is defined by rotation matrix 0 (2?o), and the position / location of the Camera 0 in the 3D space relative to the world coordinate system is defined by translation vector 0 (#o). The Person 500 is also captured in a second image at second camera view by Camera 1. The global orientation of Camera 1 in the 3D space is defined by rotation matrix 1 (R ), and the position / location of the Camera 1 in the 3D space relative to the world coordinate system is defined by translation vector 1 (0).

[0058] In an embodiment, the first image and the second image are part of the input image data received from a plurality of cameras of a multi-camera system as described in FIG. 4. The first image and the second image are processed by the human estimator 110. As described above, the human estimator 110 uses the HMR model 114 and the SMPL model 112 to detect the Person500 in the first image and the second image, determine respective body parameters of the Person 500 in the first image and the second image, and generate corresponding 3D human body mesh of the Person 500 based on the first image and the second image. For instance, as shown in FIG. 5, a recovered 3D body mesh Mo is generated based on the first image captured by Camera 0. In addition, a recovered 3D body mesh Mi is generated based on the second image captured by Camera 1. As shown in FIG. 5, based on the location and orientation of Camera 0 and Camera 1, the recovered 3D body mesh Mo represents more of a side view of the Person 500. In contrast, the recovered 3D body mesh Mi represents more of a front view of the Person 500. As described above, the recovered 3D body meshes Mo and Mi along with other recovered 3D body meshes (which may or may not represent the Person 500) are passed to the cross- view matching selector 116 for selecting matched 3D body meshes belonging to the same person for performing pairwise calibration as described in FIG. 4.

[0059] FIG. 6 is a block diagram of a fisheye multi-camera automatic calibration system 600 in accordance with an embodiment of the present disclosure. The fisheye multi-camera automatic calibration system 600 is configured to perform automatic calibration for a multi-camera system that includes one or more fisheye cameras. Fisheye cameras are cameras that provide omnidirectional vision with up to 360° FoV, which is much wider than typical perspective vision from traditional cameras. As such, the use of fisheye cameras is advantageous in various fields or applications such as, but not limited to, photography, telepresence, autonomous driving, cinema, surveillance, or robotics.

[0060] Accurate calibration of multiple fisheye cameras ensures a minimum number of fisheye cameras are able to cover a wide area and to mitigate occlusion and depth ambiguity problems. One challenge with performing multi-camera automatic calibration of fisheye cameras using human body meshes is that the image output format of fisheye cameras (e.g., equirectangular, spherical (as shown in FIG. 6), or cubemap formats) often have shape distortions. That is because fisheye camera lenses are designed to intentionally bend light to fit more of the scene into the image frame to create the wider FoV. This bending of the light results in radial distortion. For instance, straight lines or edges in the scene that does not pass through the image center becomes curved, causing objects such as people in the image to become distorted. Additionally, the farther from the image center, the more the shape becomes distorted such that objects near the edge of the frame appear stretched or warped.

[0061] Since most DL-based monocular human detection and HPSE models are trained using images in perspective view with normal FoV 60°) having few, if any, shape distortions, these models would likely not work well with the image output formats of fisheye cameras. Therefore, to address this challenge, the fisheye multi-camera automatic calibration system 600 modifies the multi-camera automatic calibration system 400 in FIG. 4 to include a Spherical Human Detector 610 and a Perspective Mapping module 612 in the Human Estimator 110.

[0062] As shown in FIG. 6, image data from fisheye cameras 602, 604, 606, and 608 are first fed into Spherical Human Detector 610. The image data includes wide-angle images such as, but not limited to, equirectangular, cubemap, or spherical images. The Spherical Human Detector 610 is configured to detect (i.e., identify) humans in the images in the input image data. In an embodiment, the Spherical Human Detector 610 is implemented using a model such as the spherical Faster Region-based Convolutional Neural Network (spherical Faster R-CNN). The spherical Faster R-CNN is an adaptation of the traditional Faster R-CNN object detection framework designed to handle spherical or 360-degree images, such as those captured by fisheye or omnidirectional cameras. In an embodiment, the spherical Faster R-CNN extends the standard 2D detection pipeline into the spherical domain by replacing the standard Convolutional Neural Network (CNN) backbone with Spherical Convolution (SphConv) to handle feature extraction from spherical images. SphConv is a CNN operation designed for data represented on a spherical surface such as, but not limited to, 360° images. The result is a detection framework capable of accurately localizing and classifying objects throughout 360-degree panoramic images without suffering from the distortions that degrade standard CNN performance. In an embodiment, the Spherical Human Detector 610 analyzes the input image and outputs a bounding FoV with an object class and a softmax score in [0, 1] for each detected object. In an embodiment, only humans detected with the object class Person and a score larger than a preset threshold (e.g., 0.6) will be processed for calibration. In an embodiment, the Spherical Human Detector 610 outputs an a- degree bounding FoV centered at (0, cp) associated with each detected human, where 9 is the polar angle and tp is the azimuthal angle in the input spherical image lsfrom a fisheye camera.

[0063] In some embodiments, the Spherical Human Detector 610 may also send detection information related to the bounding FoV containing a detected human in the input image to the Cross- View Matching Selector 116. In these embodiments, once the Cross- View Matching Selector 116 receives the reconstructed 3D human meshes from the SMPL model 112, the Cross-View Matching Selector 116 can utilize the detection information to select the reconstructed 3D human meshes corresponding to the same person.

[0064] In an embodiment, the bounding FoV information is then passed to the Perspective Mapping module 612. The Perspective Mapping module 612 is configured to undistort the detected human in the bounding FoV by performing (inverse) perspective mapping using a unified spherical model. For instance, in an embodiment, for each detected human in an image, the bounding FoV is cropped and converted to an image patch by perspective mapping at each corresponding view angle. In an embodiment, the Perspective Mapping module 612 performs a perspective mapping (IS, a, 0, tp) = Ipby projecting the associated a-degree FoV from Isto a IK* IK pixels image patch Ipon the tangent plane at view angle (9, (p). The view-angle dependent perspective mapping associated with each detected person can be considered to be a virtual camera, with its optical axis pointing at the center of the associated bounding FoV and its view covering the associated bounding FoV. For each virtual camera, the ideal pinhole camera model is assumed to provide an undistorted perspective view, thus eliminating the serious geometric distortion caused by the 360° fisheye camera.

[0065] By removing the shape distortion, the image containing the undistorted detected human can now be processed the same way as a traditional perspective image described in FIG. 4. For instance, the image containing the undistorted detected human is passed to the HMR model 114 and the SMPL model 112 to reconstruct foil 3D human meshes of a detected human from a pair of virtual cameras corresponding to a pair of fisheye cameras from the plurality of fisheye cameras, which are then used to estimate the virtual camera pose and derive the corresponding fisheye camera pose.

[0066] In an embodiment, for each virtual camera, the position of the virtual camera is assumed to coincide with the corresponding 360° fisheye camera, while the rotation matrix Rvand intrinsic matrix Kvof the virtual camera can be derived directly from the associated bounding FoV: Rv~ ROT(^)ROT(0) and K = f, 0 xo; 0 ,-yo; 0 0 1], where ROT(^) and ROT(f) are rotation matrices about the Y and X axes, respectively. f, = JF72tan(a / 2) is the focal length and xo =yo = H72 are the principal point offsets of the virtual camera. For each virtual camera, the extrinsic camera matrix from the 3D world coordinates to its 2D pixel coordinate is [7?v| 0] [1? / | tj, where Ri and f are the rotation matrix and translation vector of the corresponding fisheye camera. The 3D human body mesh can be projected onto the 2D image plane of each virtual camera using a perspectiveprojection with the extrinsic parameters [ / ? | t\ estimated from each virtual camera by the HMR model 114.

[0067] Similar to the process described in FIG. 4, in the mesh matching unit 120, the vertices of each recovered 3D human body mesh are projected onto the 2D image plane of the corresponding virtual camera by the keypoint sampling and correspondence extraction module 122. SfM algorithms in OpenCV library are used to reconstruct 3D shapes from a pair of virtual cameras, using iterative RANSAC algorithm to remove outliers when the essential matrix is being calculated in each iteration. In an embodiment, by assuming one main fisheye camera’s pose in the world coordinate is known, the poses (i.e., the camera extrinsic parameters) of all other fisheye cameras in the multi-camera system are readily calculated from the estimated essential matrix and the view angle parameters for the associated perspective mappings for the corresponding pair of virtual cameras. Specifically, the relative pose parameters between a virtual camera pair are calculated by decomposing the estimated essential matrix during pairwise calibration performed by the mesh matching unit 120, and for each virtual camera its relative pose (i.e., the view angle (9, (p) for perspective mapping) with respect to the corresponding fisheye camera are determined by the Spherical Human Detector 610, therefore, the relative pose parameters between a fisheye camera pair can be calculated accordingly. Similarly, the calibration results of the overall system are further optimized using a BA algorithm by the pose accumulation and global optimization module 126. Thus, the fisheye multi-camera automatic calibration system 600 provides fast and accurate automatic calibration of multiple fisheye cameras utilizing human semantic information, such as human body mesh without retraining monocular 3D HPSE.

[0068] In the preceding embodiments, multi-camera automatic calibration was performed using the recovered 3D body meshes of a just a single person (e.g., because the image data input contains only images of one person). This reduces the overall complexity since the cross-view matching selector 116 only had to be concerned with selecting matched 3D body meshes based on time to ensure that the recovered 3D body meshes correspond to camera views taken at the same time. However, often times, a multi-camera system is used to monitor heavy foot traffic areas and the image data captured by the multi-camera system contains images depicting multiple people. Thus, it would be beneficial to be able to perform multi-camera automatic calibration on such systems.

[0069] FIG. 7 is a block diagram of a fisheye multi-person multi-camera automatic calibration system 700 in accordance with an embodiment of the present disclosure. The fisheye multi-person multi-camera automatic calibration system 700 extends the fisheye multi-camera automatic calibration system 600 to include a re-identification (Re-ID) model 702 and a multi-view matcher 704 in the human estimator 110 to create a multi-human estimator to support fisheye multi-person multi-camera automatic calibration.

[0070] It should be noted that in another embodiment, the same extension can be applied to the multi-camera automatic calibration system 400 in FIG. 4 to support multi-person multi-camera automatic calibration for traditional multi-camera systems.

[0071] In the depicted embodiment, the fisheye multi-person multi-camera automatic calibration system 700 receives and processes the image data from fisheye cameras 602, 604, 606, and 608 through the Spherical Human Detector 610 and Perspective Mapping module 612 to produce an undistorted image of each of one or more detected persons as described above in FIG. 6. In the depicted embodiment, the undistorted image of each of the one or more detected persons is then passed to the HMR model 114 and the Re-ID model 702. The HMR model 114 is used to estimate 3D human body meshes of the one or more persons detected in each view from each camera. In particular, as stated above, the HMR model 114 is configured to determine extrinsic camera parameters and SMPL parameters defining characteristics of the one or more detected persons such as, but not limited to, pose (e g., articulation, joint rotations, and configuration of the body), shape (e g., height, weight, and body proportions), and / or position of the detected person relative to the camera.

[0072] In an embodiment, the Re-ID model 702 is a pre-trained person re-identification network designed to recognize and match the same individual or object across different images, video frames, or camera views — even when the viewpoint, lighting, or background changes. In an embodiment, the Re-ID model 702 is configured to extract discriminative appearance features (e.g., a person’s clothing color, texture, shape, etc.) as descriptor feature vectors from the cropped image of each bounding box associated with each of the one or more persons detected in each view from each camera.

[0073] In an embodiment, the multi-view matcher 704 is configured to receive the descriptor feature vectors from the Re-ID model 702, the perspective mapping vectors from the Spherical Human Detector 610, and the pose and shape vectors from the HMR model 114. Based on thereceived vectors, the multi-view matcher 704 is configured to establish the correspondences of the one or more detected bodies (represented by shape, pose, and feature) across the different camera views so that a same person can be identified in images by the different camera views. In an embodiment, the multi-view matcher 704 is configured to calculate and optimize a permutation matrix P that provides the cycle-consistent correspondences of human bodies across all views. An embodiment of the multi-view matcher 704 is further described below in FIG. 8.

[0074] As described above, the SMPL model 112 is then configured to generate a 3D human body mesh based on the SMPL parameters of each detected body matched and permuted across difficult views by the multi-view matcher 704. In an embodiment, the cross-view matching selector 116 selects matched 3D body meshes that were generated from images taken of the same person at the same time by the different camera views to ensure proper calibration. The crossview matching selector 116 sends the recovered body meshes of the same person generated from images taken at the same person from pairs of cameras to the mesh matching unit 120 to perform pairwise camera calibration to estimate a 3D pose for each camera pair using corresponding 3D human body meshes as described in FIG. 4 and FIG. 6.

[0075] It should be noted that one or more processing stages of the disclosed embodiments may be performed in a sequential, concurrent, or hybrid scheme depending on various factors such as, but not limited to, system hardware / software resources, speed, calibration time, and accuracy requirements. For example, in some embodiments, the image data for each camera is processed in parallel to identify and generate the 3D human body meshes to speed up calibration time. In some embodiments, different stages of the disclosed multi-camera automatic calibration systems are also performed concurrently to increase efficiency.

[0076] FIG. 8 is a block diagram of a multi-view matcher 800 in accordance with an embodiment of the present disclosure. In an embodiment, the multi-view matcher 704 shown in FIG. 7 is implemented using the multi-view matcher 800. In the depicted embodiment, the multiview matcher 800 includes a geometric affinity matrix calculation 802, an appearance affinity matrix calculation 804, a combined affinity matrix calculation 806, a permutation matrix calculation 808, a permutation matrix optimization 810, and a permutation for multi-view matching 812.

[0077] The geometric affinity matrix calculation 802 is configured to receive pose and shape vectors from a HMR model such as HMR model 114 described in FIG. 7. The geometric affinitymatrix calculation 802 computes geometric affinity scores Ag between detected human body meshes from different camera views. Each element of the geometric affinity matrix Agy is calculated by determining the Euclidean distance between corresponding SMPL parameter vectors (pose and shape) of a pair of detected human body meshes detected from view i and view j. The calculated distances are then mapped to a normalized range between 0 and 1 using a sigmoid function.

[0078] The appearance affinity matrix calculation 804 is configured to receive feature vectors from a re-ID model such as re-ID model 702 described in FIG. 7. The appearance affinity matrix calculation 804 computes appearance affinity scores Aa between detected human bounding boxes from different camera views. Each element of the appearance affinity matrix Ally is calculated by determining the Euclidean distance between the descriptor (feature) vectors of a pair of detected human bounding boxes from view i and view j, and mapping the resulting distances to values between 0 and 1 using a sigmoid function.

[0079] The combined affinity matrix calculation 806 combines the geometric and appearance affinities to produce a unified affinity matrix In one embodiment, each element of the combined affinity matrix A,, is determined according to Ay = SQRT( / 4g!Zx Aa.y), where x denotes element- wise matrix multiplication. The combined affinity matrix A therefore represents the overall likelihood that two detected human body meshes from different views belong to the same person based on both geometric consistency (represented by pose and shape) and appearance similarity (represented by feature).

[0080] The permutation matrix calculation 808 generates a permutation matrix P for establishing correspondences of detected human body meshes across the different views. Each element of the permutation matrix Py, solved separately for each pair of views, has each of its entries representing a binary correspondence (i.e., either 0 or 1) indicating whether a detected human body mesh from view i corresponds to a detected human body mesh from view j.

[0081] The permutation matrix optimization 810 refines the permutation matrix P to produce an optimized permutation matrix Popt. In one embodiment, the permutation matrix optimization 810 applies the alternating direction method of multipliers (ADMM) to solve a convex optimization problem that maximizes affinities while maintaining cycle-consistency across all views, i.e., any two corresponding human body meshes recovered from two views should correspond to the human body mesh (from the same person) in another view. The optimizedpermutation matrix Popt thereby provides cycle-consistent correspondences of human body meshes across multiple camera views. When the optimized permutation matrix Poptachieves the cycleconsistency condition, Popt can be decomposed as YYTwhere Y denotes the correspondences between all the detected human bodies and all the people in the scene across all views. The unknown total number of people in the scene can be determined as the rank of the optimized permutation matrix Popt.

[0082] The permutation for multi-view matching 812 receives the optimized permutation matrix Popt from permutation matrix optimization 810, the perspective mapping vectors from the human detector, and the pose and shape vectors from the HMR model. The permutation for multiview matching 812 determines the correspondences of detected human body meshes (represented by pose, shape, and feature) across all views by decomposing Popt as FT7where Y is the decomposed optimized permutation matrix, while the total number of people in the scene is also determined as the rank of the optimized permutation matrix Popt. The inputs of the permutation for multi-view matching 812 are indexed according to the order of camera view, while its outputs are permuted after multi-view matching and indexed according to the order of distinct detected persons. The permuted pose and shape vectors are provided to an SMPL model 112 to generate permuted 3D human body meshes that are cycle-consistent across the multiple views. The output of the SMPL model 112 and the permuted perspective mapping vectors may be supplied to a crossview matching selector 116 to select valid cross- view matched 3D body meshes in pairs and send to the mesh matching unit 120 to perform pairwise camera calibration and to reject mismatches or outliers. Human body meshes that have no valid matches in other views may be excluded from subsequent multi-camera calibration. In an embodiment, incorrect cross-view matchings are later rejected as outliers by the SfM using RANSAC during pairwise camera calibration performed in the mesh matching unit 120.

[0083] FIG. 9 is a flowchart illustrating a method 900 for performing pairwise calibration of a multi-camera system using human body meshes in accordance with an embodiment of the present disclosure. In an embodiment, the method 900 is implemented by a calibration system such as, but not limited to, the multi-camera automatic calibration system 400 shown in FIG. 4. For instance, as indicated in FIG. 9, the method 900 includes a sequence of steps corresponding to the processing stages of the human estimator 110 (steps 902-904) and the mesh matching unit 120 (steps 906-914) in FIG. 4.

[0084] At step 902, the calibration system performs human detection on image data received from a plurality of cameras (e.g., cameras 102, 104, 106, and 108 in FIG. 4). During this step, the calibration system analyzes the image data to detect one or more humans in each image frame. In an embodiment, the calibration system utilizes the HMR model 114 as described in FIG. 4 to locate each human subject within a view, determine pose and shape parameters («, p) for each detected person, and define a bounding region for subsequent 3D mesh reconstruction.

[0085] In an embodiment, at step 904, the calibration system reconstructs, using the SMPL model 112 in FIG. 4, a 3D human body mesh for each detected individual based on the pose and shape parameters ( / , P) for each detected person. In an embodiment, the reconstructed 3D body meshes are provided as input to the cross-view matching selector 116 for selecting matched mesh pairs for a camera pair as described in FIG. 4.

[0086] In an embodiment, at step 906, the calibration system performs correspondence matching between the 3D human body meshes recovered from a selected pair of cameras. In an embodiment, the calibration system projects selected 3D mesh vertices onto each two-dimensional (2D) image plane to form corresponding keypoints between the selected camera pairs. In an embodiment, adaptive vertex sampling is used to ensure uniform coverage of body parts with reliable geometric features, while minimizing redundant correspondences. The resulting 2D keypoint pairs establish the basis for geometric correspondences used for pairwise calibration.

[0087] At step 908, the calibration system estimates a fundamental matrix between the matched camera pairs using the 2D keypoint correspondences obtained in step 906. The fundamental matrix is a matrix that encodes the epipolar geometry between two different camera views of the same scene. The fundamental matrix serves as an initial representation of the relative geometry between the two cameras.

[0088] In an embodiment, the calibration system, at step 910, refines the correspondence data using an outlier rejection process (e.g., using RANSAC algorithm) to iteratively remove spurious or inconsistent keypoint pairs that violate the epipolar constraint, thereby enhancing geometric accuracy. In some embodiments, an adaptive RANSAC threshold may be used to account for varying baseline distances among camera pairs. The result of this refinement is a set of refined inlier correspondences suitable for precise essential matrix estimation.

[0089] At step 912, the calibration system estimates an essential matrix using the refined inlier correspondences. The essential matrix is a matrix that encodes the true geometric relationship(rotation R and translation / ) between two calibrated cameras. In an embodiment, the calibration system estimates an essential matrix using the refined inlier correspondences by iteratively deriving the matrix that best satisfies the epipolar constraint between camera views, applying singular value decomposition to extract its geometric structure, and enforcing a rank-2 constraint to ensure the matrix represents a physically valid relationship between camera rotation and translation. As stated, the essential matrix determined during this step is only an estimate of the essential matrix of a camera pair because the correspondences, even after refinement, contain imperfect data.

[0090] At step 914, the calibration system estimates camera pose (i.e., rotation and translation components) for the camera pair using the estimated essential matrix obtained at step 912. In an embodiment, the calibration system decomposes the estimated essential matrix into candidate rotation and translation components, and selects the rotation and translation components for which all reconstructed 3D points lie in front (with positive depth) of both cameras in the camera pair as the camera pose. The output of this stage is the relative extrinsic parameters between the two cameras.

[0091] FIG. 10 is a flowchart illustrating a method 1000 for performing pairwise calibration of a multi-camera fisheye system using human body meshes in accordance with an embodiment of the present disclosure. In an embodiment, the method 1000 is implemented by a calibration system such as, but not limited to, the fisheye multi-camera automatic calibration system 600 shown in FIG. 6. For example, as depicted in FIG. 10, the method 1000 includes a sequence of steps corresponding to the processing stages of the human estimator 110 (steps 1002-1006) and the mesh matching unit 120 (steps 1008-1018) described in FIG. 6.

[0092] In step 1002, the calibration system performs spherical human detection on fisheye image data captured by multiple cameras (e.g., cameras 602, 604, 606, and 608 in FIG. 6) to identify humans in the spherical images. In an embodiment, the calibration system outputs, for each detected object, an a-degree bounding FoV centered at angular coordinates (0, cp) together with a class label and confidence score. Only detected objects labeled as “Person” class with confidence score exceeding a preset confidence threshold are selected for calibration in the subsequent steps.

[0093] Next, at step 1004, the calibration system performs perspective mapping. In an embodiment, the calibration system projects each selected a-degree bounding FoV associated witha detected person from the spherical image onto a tangent plane at view angle (0, (p) to create a JFx W pixel undistorted perspective image patch. This operation defines a virtual camera whose optical axis points toward the center of the bounding FoV and whose intrinsic parameters are determined by the perspective mapping. The result is a conventional perspective image patch suitable for mesh reconstruction.

[0094] After perspective patches are generated, at step 1006, the calibration system performs human body mesh recovery. In an embodiment, the calibration system uses the HMR Model 114 to estimate SMPL pose parameters ( / / ) and shape parameters (fi) for each undistorted image patch. The SMPL Model 112 then reconstructs a 3D human body mesh represented by a collection of vertices.

[0095] Once the meshes are reconstructed, at step 1008, the calibration system carries out correspondence matching between the 3D human body meshes recovered from a selected pair of virtual cameras. The calibration system projects selected 3D mesh vertices onto the 2D image planes of each virtual camera and identifies corresponding keypoints between the selected virtual camera pairs. In an embodiment, adaptive vertex sampling provides uniform coverage of body parts while minimizing redundancy. The resulting 2D keypoint pairs form the basis for geometric correspondences used for pairwise calibration.

[0096] In step 1010, the calibration system estimates a fundamental matrix for the virtual camera pair from the 2D keypoint correspondences obtained at step 1008. The fundamental matrix represents the epipolar geometry between the two virtual camera views and serves as an initial model of their relative alignment.

[0097] To improve accuracy, at step 1012, the calibration system refines the correspondence set through outlier rejection using a RANSAC procedure. Keypoints that violate the epipolar constraint are removed, leaving a refined inlier set suitable for precise pose estimation.

[0098] At step 1014, the calibration system estimates an essential matrix from the refined inlier correspondences. In an embodiment, the calibration system computes the essential matrix and applies singular value decomposition with a rank-2 constraint to ensure physical validity.

[0099] The calibration system, at step 1016, decomposes the estimated essential matrix into candidate rotation and translation components and determines the virtual camera pose by selecting the rotation and translation for which all triangulated 3D points have positive depth in both virtualcameras. The output of this stage is the relative extrinsic parameters between the two virtual cameras.

[0100] Finally, at step 1018, the calibration system derives the fisheye-camera pose from the estimated virtual camera pose. In an embodiment, each virtual camera shares the position of its corresponding fisheye camera, while its orientation is defined by the orientation of its corresponding fisheye camera and the mapping angles (0, <p) obtained during spherical human detection in step 1002 and used by perspective mapping in step 1004. Using these relationships, the calibration system computes the extrinsic parameters of the fisheye cameras. The output of this stage is the relative extrinsic parameters between the two fisheye cameras.

[0101] FIG. 11 is a diagram illustrating an apparatus 1100 according to an embodiment of the present disclosure. The apparatus 1100 may be any type of computing device used to implement embodiments of the present disclosure. For instance, in some embodiments, the multi-camera automatic calibration system 400 in FIG. 4, the fisheye multi-camera automatic calibration system 600 in FIG. 6, and the fisheye multi-person multi-camera automatic calibration system 700 in FIG. 7 may be implemented using the apparatus 1100. Additionally, the apparatus 1100 may be configured to execute instructions for performing the method 900 in FIG. 9 and / or the method 1000 in FIG. 10.

[0102] The apparatus 1100 includes receiver units (RX) 1120 for receiving data via ingress ports 1110. The apparatus 1100 also includes transmitter units (TX) 1140 for transmitting via data egress ports 1150. In an embodiment, the RX 1 120 comprises a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of receiving signals. In an embodiment, the TX 1140 comprises a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of transmitting signals.

[0103] The apparatus 1100 includes a memory 1160 for storing the instructions and various data. The memory 1160 can be any type of, or combination of, memory components capable of storing data and / or instructions. For example, the memory 1160 can include volatile and / or nonvolatile memory such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 1160 can also include one or more disks, tape drives, and solid-state drives. In some embodiments, the memory 1160 can be used as an over-flow data storage device to store programs when such programs are selected for execution, and to store instructions and data that are readduring program execution. In some embodiments, the memory 1160 can be processor memory such as cache memory that is integrated with the processor 1130.

[0104] The apparatus 1100 has one or more processors 1130 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU)) to process instructions. The one or more processors 1130 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 1130 are communicatively coupled via a system bus with the ingress ports 1110, RX 1120, TX 1140, egress ports 1150, Input / Output (I / O) 1180, and memory 1160. I / O 1180 provides the communication interfaces for enabling the apparatus 1100 to receive input (e.g., from a keyboard, mouse, or touchscreen) and output information (e.g., to a display or printer).

[0105] The one or more processors 1130 can be configured to execute instructions stored in the memory 1160. For instance, the one or more processors 1130 can be configured to perform any computational, comparison, determination, initiation, configuration, estimation, or any other action corresponding to the claims when the appropriate instruction or model is executed by the processor 1130. In one embodiment, the memory 1160 stores a multi-camera auto calibration module 1170 that includes data, executable instructions, models, and / or one or more sub-modules for implementing the disclosed embodiments. Thus, the inclusion of the multi-camera auto calibration module 1170 substantially improves the functionality of the apparatus 1100.

[0106] The present disclosure may be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0107] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory(SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0108] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0109] Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’ s computer, partly on the user’ s computer, as a standalone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable programinstructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0110] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0111] These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0112] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0113] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, infact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0114] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the disclosure is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0115] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A method, implemented by a computing device, the method comprising: receiving image data captured simultaneously from a plurality of cameras; detecting a human in the image data captured by two or more cameras of the plurality of cameras; reconstructing a three-dimensional (3D) human body mesh of the human based on detection of the human to obtain a set of recovered 3D human body meshes of the human; matching recovered 3D human body meshes from the set of recovered 3D human body meshes to obtain one or more matched pairs of the recovered 3D human body meshes; identifying a matched pair of the recovered 3D human body meshes from the one or more matched pairs for a camera pair from the two or more cameras; and performing pairwise camera calibration for the camera pair using the matched pair of the recovered 3D human body meshes.

2. The method of claim 1, wherein performing the pairwise camera calibration comprises: projecting vertices of the matched pair of recovered 3D human body meshes onto two- dimensional image planes of the camera pair to determine keypoint correspondences; estimating a fundamental matrix for the camera pair using the keypoint correspondences; rejecting outliers from the keypoint correspondences to obtain inlier correspondences; estimating an essential matrix based on the inlier correspondences; decomposing the essential matrix to obtain candidate rotations and translations; and selecting a physically valid rotation and translation from the candidate rotations and translations for which triangulated points have positive depth in both cameras of the camera pair.

3. The method of claim 2, wherein projecting the vertices comprises sampling mesh vertices using adaptive vertex sampling to provide uniform coverage of body parts while reducing redundant correspondences.

4. The method of claim 2, wherein estimating the fundamental matrix is performed on normalized image coordinates.

5. The method of claim 2, wherein rejecting outliers is performed using a Random Sample Consensus (RAN SAC) algorithm.

6. The method of claim 2, wherein estimating the essential matrix comprises performing singular value decomposition with a rank-2 constraint.

7. The method of claim 1, further comprising repeating the performing of the pairwise camera calibration for multiple camera pairs of the plurality of cameras to obtain a set of relative pose parameters between camera pairs.

8. The method of claim 7, further comprising performing pose accumulation and global optimization to compute globally consistent camera poses for the plurality of cameras and minimizing a reprojection error across the plurality of cameras.

9. The method of claim 8, wherein minimizing the reprojection error comprises performing bundle adjustment to jointly refine relative pose parameters for the plurality of cameras.

10. The method of claim 1, wherein detecting the human comprises processing the image data using a neural network trained to identify human regions or human keypoints.

11. The method of claim 1, wherein the plurality of cameras comprises fisheye cameras, and wherein the method further comprises: performing spherical human detection on fisheye images to identify the human within a distorted field of view; and mapping a detected region of interest to a perspective image patch by projecting a fisheye image region onto a tangent plane at a viewing direction to define a virtual camera view.

12. The method of claim 11, further comprising reconstructing the 3D human body mesh using the perspective image patch corresponding to the virtual camera view.

13. The method of claim 11, further comprising deriving a fisheye camera pose from a virtual camera pose by compensating an orientation of a fisheye camera with mapping angles used in the projecting.

14. The method of claim 1, further comprising performing calibration when multiple persons are present by: reconstructing respective 3D human body meshes for the multiple persons in images captured by the plurality of cameras; and determining correspondences between the respective 3D human body meshes that represent a same person across different cameras to produce matched pairs usable for performing the pairwise camera calibration.

15. The method of claim 14, wherein determining the correspondences comprises multi-view matching, and wherein the multi-view matching comprises: computing a geometric affinity matrix; computing an appearance affinity matrix; combining the geometric affinity matrix and appearance affinity matrix to produce a combined affinity matrix; and optimizing a permutation matrix to enforce correspondence consistency across multiple views.

16. The method of claim 15, wherein the geometric affinity matrix is based at least in part on difference between pose and shape of human body meshes, and wherein the appearance affinity matrix is based at least in part on learned re-identification features.

17. The method of claim 15, wherein the multi-view matching is applied to images captured by perspective cameras, fisheye cameras, or a combination thereof.

18. The method of claim 14, wherein determining the correspondences further comprising enforcing cycle-consistency constraints among the multiple persons captured across the plurality of camera.

19. The method of claim 1, wherein reconstructing the 3D human body mesh comprises generating a mesh representation based on estimated human pose and shape parameters using a human mesh recovery model and a parametric human body model.

20. The method of claim 5, wherein the Random Sample Consensus algorithm utilizes an adaptive inlier threshold that varies according to a baseline between the camera pair.

21. The method of claim 6, further comprising selecting, from a decomposed essential matrix, one of multiple candidate rotations and translations based on a positive-depth constraint.

22. The method of claim 9, wherein the bundle adjustment jointly refines camera extrinsic parameters for the plurality of cameras while holding camera intrinsic parameters fixed.

23. An apparatus, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the apparatus to: receive image data captured simultaneously from a plurality of cameras; detect a human in the image data captured by two or more cameras of the plurality of cameras; reconstruct a three-dimensional (3D) human body mesh of the human based on detection of the human to obtain a set of recovered 3D human body meshes of the human; match recovered 3D human body meshes from the set of recovered 3D human body meshes to obtain one or more matched pairs of the recovered 3D human body meshes; identify a matched pair of the recovered 3D human body meshes from the one or more matched pairs for a camera pair from the two or more cameras; and perform pairwise camera calibration for the camera pair using the matched pair of the recovered 3D human body meshes.

24. The apparatus of claim 23, wherein performing the pairwise camera calibration causes the one or more processors to:project vertices of the matched pair of recovered 3D human body meshes onto two- dimensional image planes of the camera pair to determine keypoint correspondences; estimate a fundamental matrix for the camera pair using the keypoint correspondences; reject outliers from the keypoint correspondences to obtain inlier correspondences; estimate an essential matrix based on the inlier correspondences; decompose the essential matrix to obtain candidate rotations and translations; and select a physically valid rotation and translation from the candidate rotations and translations for which triangulated points have positive depth in both cameras of the camera pair.

25. The apparatus of claim 24, wherein projecting the vertices comprises sampling mesh vertices using adaptive vertex sampling to provide uniform coverage of body parts while reducing redundant correspondences.

26. The apparatus of claim 24, wherein estimating the fundamental matrix is performed on normalized image coordinates.

27. The apparatus of claim 24, wherein rejecting outliers is performed using a Random Sample Consensus (RANSAC) algorithm.

28. The apparatus of claim 24, wherein estimating the essential matrix comprises performing singular value decomposition with a rank-2 constraint.

29. The apparatus of claim 23, wherein the instructions further cause the one or more processors to repeat the performing of the pairwise camera calibration for multiple camera pairs of the plurality of cameras to obtain a set of relative pose parameters between camera pairs.

30. The apparatus of claim 29, wherein the instructions further cause the one or more processors to perform pose accumulation and global optimization to compute globally consistent camera poses for the plurality of cameras and to minimize a reprojection error across the plurality of cameras.

31. The apparatus of claim 30, wherein minimizing the reprojection error comprises performing bundle adjustment to jointly refine relative pose parameters for the plurality of cameras.

32. The apparatus of claim 23, wherein detecting the human comprises processing the image data using a neural network trained to identify human regions or human keypoints.

33. The apparatus of claim 23, wherein the plurality of cameras comprises fisheye cameras, and wherein the instructions further cause the one or more processors to: perform spherical human detection on fisheye images to identify the human within a distorted field of view; and map a detected region of interest to a perspective image patch by projecting a fisheye image region onto a tangent plane at a viewing direction to define a virtual camera view.

34. The apparatus of claim 33, wherein the instructions further cause the one or more processors to reconstruct the 3D human body mesh using the perspective image patch corresponding to the virtual camera view.

35. The apparatus of claim 33, wherein the instructions further cause the one or more processors to derive a fisheye camera pose from a virtual camera pose by compensating an orientation of a fisheye camera with mapping angles used in the projecting.

36. The apparatus of claim 23, wherein the instructions further cause the one or more processors to perform calibration when multiple persons are present by: reconstructing respective 3D human body meshes for the multiple persons in images captured by the plurality of cameras; and determining correspondences between the respective 3D human body meshes that represent a same person across different cameras to produce matched pairs usable for performing the pairwise camera calibration.

37. The apparatus of claim 36, wherein determining the correspondences comprises multi-view matching, and wherein the multi-view matching causes the one or more processors to:compute a geometric affinity matrix; compute an appearance affinity matrix; combine the geometric affinity matrix and appearance affinity matrix to produce a combined affinity matrix; and optimize a permutation matrix to enforce correspondence consistency across multiple views.

38. The apparatus of claim 37, wherein the geometric affinity matrix is based at least in part on epipolar consistency or reprojection error between candidate mesh correspondences, and wherein the appearance affinity matrix is based at least in part on learned re-identification features.

39. The apparatus of claim 37, wherein the multi-view matching is applied to images captured by perspective cameras, fisheye cameras, or a combination thereof.

40. The apparatus of claim 36, wherein determining the correspondences further cause the one or more processors to enforce cycle-consistency constraints among the multiple persons captured across the plurality of cameras.

41. The apparatus of claim 23, wherein reconstructing the 3D human body mesh comprises generating a mesh representation based on estimated human pose and shape parameters using a human mesh recovery model and a parametric human body model.

42. The apparatus of claim 27, wherein the Random Sample Consensus algorithm utilizes an adaptive inlier threshold that varies according to a baseline between the camera pair.

43. The apparatus of claim 28, wherein the instructions further cause the one or more processors to select, from a decomposed essential matrix, one of multiple candidate rotations and translations based on a positive-depth constraint.

44. The apparatus of claim 31, wherein the bundle adjustment jointly refines camera extrinsic parameters for the plurality of cameras while holding camera intrinsic parameters fixed.

45. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium and that, when executed by one or more processors of an apparatus, cause the apparatus to perform any one of the methods in claims 1-22.

Images