Cross-Spectral Feature Mapping for Camera Calibration

Abstract

The method includes a first camera sensitive to light in a first spectrum and having a first light source capturing a first image of the real-world scene; a second camera sensitive to light in a second spectrum and having a second light source capturing a second image of the real-world scene; identifying at least one feature in the first image; identifying at least one feature in the second image that matches the at least one feature identified in the first image using a machine learning (ML) model; mapping pixels in the first image and the second image to light rays in three-dimensional (3D) space based on the matched at least one feature; and calibrating the first camera and the second camera based on the mapping.

Description

Technical Field

[0001] Field Embodiments relate to the geometric calibration of two or more cameras.

Background Art

[0002] Background Geometric camera calibration is the process of determining the position and internal parameters (such as focal length) of a camera or a set of cameras. Geometric calibration provides a mapping between camera pixels and rays in three-dimensional (3D) space. Calibration is determined by finding pairs of pixels in different camera views that correspond to the same point in the real-world scene and adjusting the internal parameters of each camera to align the pixel pairs (e.g., pixels in the image of the first camera and pixels in the image of the second camera are mapped to be the same in the real-world scene).

Summary of the Invention

[0003] Summary In general aspects, a device, a system, a non-transitory computer-readable medium (storing computer-executable program code executable on a computer system), and / or a method are capable of executing a process by the method. The method includes a first camera having sensitivity to light of a first spectrum and having a first light source capturing a first image of a real-world scene, a second camera having sensitivity to light of a second spectrum and having a second light source capturing a second image of the real-world scene, identifying at least one feature in the first image, using a machine learning (ML) model to identify at least one feature in the second image that matches at least one feature identified in the first image, mapping pixels in the first image and the second image to light rays in a three-dimensional (3D) space based on the matching at least one feature, and calibrating the first camera and the second camera based on the mapping.

[0004] Implementations may include one or more of the following features. For example, the first camera may be a near infrared (NIR) camera and the second camera may be a visible light camera. An ML model may be used to identify at least one feature in the first image. An algorithm may be used to identify at least one feature in the first image. An ML model may be used to match at least one feature in the first image with at least one feature in the second image, and a score may be assigned to at least one pixel in the second image based on the likelihood that at least one pixel in the second image matches a pixel of at least one feature in the first image. An algorithm may be used to match at least one feature in the first image with at least one feature in the second image, and a score may be assigned to at least one pixel in the second image based on the likelihood that at least one pixel in the second image matches a pixel of at least one feature in the first image, and a direction may be assigned to at least one pixel in the second image based on a prediction of the position of a target pixel.

[0005] Using an algorithm, at least one feature in the first image may be identified, and matching at least one feature in the first image with at least one feature in the second image may include using a first ML model to select candidate features from at least one feature in the first image, matching at least one pixel in the second image with the pixels of the candidate features, and based on the likelihood that at least one pixel matches one of the at least one feature in the first image, assigning a score to at least one matching pixel in the second image, using a second ML model to predict the direction of the position of the target pixel, and assigning a direction to at least one matching pixel in the second image. Calibrating the first camera and the second camera may be based on at least one matching pixel in the second image associated with the candidate feature having the highest score and the direction of at least one matching pixel in the second image having the highest score, and the direction may be based on at least one matching pixel in the second image having the highest score and neighboring pixels. The method may further include selecting at least one search window in the second image based on a previous calibration. The machine learning model may be trained with data captured from a calibrated multi-camera system.

[0006] Exemplary embodiments will be more fully understood from the following detailed description of the specification and the accompanying drawings. In the accompanying drawings, like elements are denoted by like reference numerals. These elements are given by way of example only and thus do not limit the exemplary embodiments.

Brief Description of the Drawings

[0007]

Figure 1A

Figure 1B

Figure 1C

Figure 1D

Figure 1E

Figure 1F

Figure 1G

Figure 1H

Figure 1I

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0008] Note that these drawings are intended to show the general characteristics of the methods, structures, and / or materials utilized in certain exemplary embodiments and to supplement the description provided below. However, these drawings are not to scale and may not accurately reflect the exact structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values or characteristics encompassed by the exemplary embodiments. For example, the relative thicknesses and positions of molecules, layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of like or the same reference numbers in the various drawings is intended to indicate the presence of like or the same elements or features.

[0009] Detailed Description of Embodiments Feature matching for geometric camera calibration can be difficult in systems that include cameras sensitive to different parts of the light spectrum. For example, a system that includes a combination of a visible light camera and a near-infrared (NIR) camera. Feature matching can be difficult because the visual appearance of an object can vary significantly across different light spectra. This problem can be exacerbated in situations where the spectra have different lighting conditions because the appearance of points within a real-world scene can change dramatically depending on the incident illumination. In the calibration of a mixed-spectrum camera system, it is typically necessary to use a specially designed calibration target that has fiducial markings that are easily detectable across different parts of the spectrum.

[0010] The use of calibration targets may not be desirable in systems that require minimal technical support during use (e.g., three-dimensional (3D) teleconferencing systems). Exemplary implementations described herein use, for example, a machine learning (ML)-based approach to solve the problem of finding corresponding feature points between, for example, visible light images and near-infrared (NIR) images. In the NIR image, a set of candidate feature points can be selected. Candidate features can represent pixels for which accurate localization is easy (e.g., corners, transitions, spots, etc.). For each candidate feature, a search window is defined in the target red-green-blue (RGB) image. A machine learning model (e.g., a neural network) scoring function can be used to assign a high score to pixels in the search window that are likely to correspond to the candidate NIR feature and a low score to other pixels, thereby assigning scores to each RGB pixel in the search window.

[0011] A second ML model (e.g., a second neural network) can be used to predict the exact match position (e.g., x, y position) for each pixel in the search window. The estimated offset for each pixel in the search window (e.g., prediction of how far the exact match position (e.g., position of the exact match pixel) is from the current RGB pixel along the x and y axes of the image). If it is found that the RGB pixels in the search window have a sufficiently high score (e.g., are likely to match), the estimated offsets of the RGB pixel and its neighboring pixels can be averaged to find the target match pixel position, creating a match between the NIR and the RGB. The first and second ML models can be trained using data captured from a well-calibrated multi-camera system in which the correct matching feature pairs between the NIR camera and the RGB camera are accurately determined.

[0012] FIG. 1A is a diagram showing a camera and a scene according to at least one exemplary embodiment. FIG. 1A is used to describe a real-world scene used to identify features in an image that can be used for calibration of a multi-camera system including cameras responsive to different types of light sources (e.g., IR and visible light). According to an exemplary implementation, the real-world scene does not include a specially designed calibration target (e.g., including distinguishable features for use in a camera calibration process).

[0013] As shown in FIG. 1A, the scene includes a first camera 5 and a second camera 10. By way of example, two cameras are shown. However, an exemplary implementation may include more than two cameras in a three-dimensional (3D) teleconferencing system. The first camera 5 and the second camera 10 may be configured to capture an image of the scene shown as including a portion in contact with the floor 110 of the wall 105. The wall may include a door 115. The wall 105 and Door 115 toThe scene to be included may include a portion of the image that includes features 120-1, 120-2, and 120-3. Each of the portions of the image that includes features 120-1, 120-2, and 120-3 can be selected using a search algorithm and / or an ML model (e.g., a neural network). The ML model can be trained to select a portion of an image (of a real-world scene). The portions of the image that include features 120-1, 120-2, and 120-3 can be corners, transitions, spots, etc. that can be precisely located (or positioned). The features can include adjacent pixels having a color (e.g., NIR or RGB) gradient. In other words, the features can be portions of an image (e.g., an NIR image) having at least one color transition from one pixel to at least one adjacent pixel. For example, the portion of the image that includes feature 120-1 may include corner 122-1 of door 115, the portion of the image that includes feature 120-2 may include a distinguishable spot (e.g., door handle 122-2), and the portion of the image that includes feature 120-3 may include corner of door 115 and transition 122-3 from door 115 to floor 110. The search algorithm and / or ML model used to identify the features can use an image captured in the light spectrum (e.g., NIR) in which it is most difficult to identify the features.

[0014] Camera 5 (e.g., NIR camera) may include a light source 125 configured to generate at least one light ray 130 in the light spectrum associated with camera 5 (e.g., NIR). The scene further includes light 135 and at least one light ray 140 in the light spectrum associated with camera 10 (e.g., visible light). Although light source 125 and light source 135 are shown inside and outside the camera respectively, exemplary implementations may include external light sources and camera light sources alone and in combination. Light rays 145 and 150 are light rays reflected from the real-world scene and are detected by the sensors of camera 5 and camera 10 respectively, and image points (e.g., pixels) are generated in the image (e.g., by the sensors of the camera) based on light rays 145, 150. Light rays 145 and 150 may correspond to (or be reflected from) the same point in the real-world scene associated with feature 120-1. In an exemplary implementation, prior to calibration, light rays 145 and 150 may not be used to generate pixels in the image at the same position (e.g., x, y position) in both camera 5 and camera 10. Thus, camera 5 and camera 10 can be calibrated to align the pixels (in the image) generated by camera 5 based on light ray 145 and the pixels (in the image) generated by camera 10 based on light ray 150 to the same position in their respective images.

[0015] Figure 1B is a two-dimensional (2D) diagram showing a portion of the scene shown in Figure 1A according to at least one exemplary embodiment. Figure 1B is used to illustrate a portion of an image that may include pixels used during calibration. In an exemplary implementation, the pixels used for calibration may be pixels within a first image that have corresponding pixels in a second image. Figure 1B shows a portion of the image shown in Figure 1A that includes feature 120-1. The portion of the image that includes feature 120-1 may include door corner 122-1. Figure 1B shows the portion of the image that includes feature 120-1 as part of a 2D image. This 2D image may be an RGB image (captured using camera 10), and the portion of the image that includes feature 120-1 may be identified using an NIR image (captured using camera 5). This 2D diagram shows pixel 155, which may be a pixel identified as a pixel to be used in camera calibration in the portion of the image that includes feature 120-1. The pixels used in camera calibration may be pixels located within an image captured by camera 5 that have corresponding (e.g., matching) pixels captured by camera 10 (shown in Figure 1A).

[0016] Figures 1C and 1D are used to show a portion of a camera sensor and the interpreted positions of detected light rays. Figure 1C shows a camera sensor associated with camera 5. Sensor position 160-1 shown in Figure 1C may correspond to a pixel (e.g., an NIR pixel) within an NIR image (not shown) captured by camera 5. Light ray 145 can cause camera 5 to generate a pixel (not shown) within the image using sensor position 160-1 during the image capture process.

[0017] Figure 1D shows a camera sensor associated with camera 10. Sensor position 160-2 shown in Figure 1D may correspond to pixel 155 (e.g., an RGB pixel). As shown in Figure 1D, light ray 150 can cause camera 10 to generate pixel 155 using sensor position 160-2 during the image capture process.

[0018] The position (e.g., x, y coordinates) of the pixels (not shown) of the image captured using camera 5 should be the same as the pixel position of pixel 155. Therefore, sensor positions 160-1 and 160-2 should be used to generate pixels having the same x, y coordinates in the corresponding images. However, as seen in FIGS. 1C and 1D, sensor positions 160-1 and 160-2 do not have the same x, y coordinates. This indicates a multi-camera system that requires calibration. In other words, a calibrated multi-camera system must have the same sensor position (e.g., sensor position 160-3 associated with pixel 155 and the corresponding pixel captured using camera 5).

[0019] Figures 1E - 1G are used to illustrate the 2D positions of pixels on an image before calibrating a multi - camera system. Figure 1E shows a 2D coordinate system representing a portion of an image having a pixel 165 - 1 located at X1, Y1 corresponding to sensor position 160 - 1 (of camera 5). Figure 1F shows a two - dimensional coordinate system representing a portion of an image having a pixel 165 - 2 located at X2, Y2 corresponding to sensor position 160 - 2 (of camera 10). Figure 1G shows a 2D coordinate system representing the overlay of the 2D coordinate system of Figure 1E and the 2D coordinate system of Figure 1F. Pixel 165 - 1 and pixel 165 - 2 can represent the same point in the real - world scene (as coincident pixels). In other words, pixel 165 - 1 and pixel 165 - 2 can represent a point within the real - world scene having the same 3D coordinates (x, y, z coordinates). Thus, the 2D image containing pixel 165 - 1 captured using camera 5 and the 2D image containing pixel 165 - 2 captured using camera 10 should share the same position (e.g., x, y coordinates) in the overlaid 2D coordinate system of Figure 1G. As can be seen in Figure 1G, pixel 165 - 1 and pixel 165 - 2 do not share the same position. Therefore, the cameras should be calibrated to align the pixels of the image captured using camera 5 with the pixels of the image captured using camera 10 (e.g., such that pixel 165 - 1 and pixel 165 - 2 have the same 2D coordinates in their respective images).

[0020] Calibration may involve adjusting calibration parameters such that calculations related to ray 145 and ray 150 are associated with the same target pixel position. The target pixel position should be the same position (e.g., x, y coordinates) in a 2D coordinate system representing a portion of the image.

[0021] FIG. 1H is used to illustrate the 2D positions of pixels on an image after calibration of a multi-camera system. FIG. 1H shows a 2D coordinate system representing the overlay of the 2D coordinate system of FIG. 1E and the 2D coordinate system of FIG. 1F after the camera calibration process. As shown in FIG. 1H, pixel 165-1’ and pixel 165-2’ share the same position X3, Y3 (also shown in FIGS. 1C and 1D). Pixel 165-1’ and pixel 165-2’ represent pixel 165-1 and pixel 165-2 after the calibration parameters have been adjusted such that rays 145 and 150 are associated with the same target pixel position. By calibrating pixel 165-1 and pixel 165-2 to the target pixel position, it is possible to interpret the 2D positions that result in the interpretation that the rays intersect at the same point in the 3D real-world scene coordinate system in the processing of sensor readings (e.g., by a camera) associated with rays 145 and 150.

[0022] FIG. 1I is used to illustrate the 3D positions of points in space corresponding to the 2D pixel positions of the calibrated cameras described above. FIG. 1I shows the 3D coordinate system of the real-world scene after camera calibration, and point 170 indicates the intersection point. Thus, the target pixel position is the position of a pixel in the 2D coordinate system used to adjust the calibration parameters of the camera such that rays (e.g., rays 145’ and 150’) in the 3D coordinate system of the real-world scene intersect at a point (e.g., point 170) in the real-world scene.

[0023] By adjusting the calibration parameters, the interpreted 2D positions of the detected light rays (e.g., light ray 145 and light ray 150) can be changed within the camera so that they are associated with different 2D detection positions where the sensor positions associated with the light rays are different. The calibration parameters can include internal parameters and external parameters. The internal parameters can include the effective focal length, or the distance from the image plane to the projection center, the lens distortion coefficients, the magnification of x, and the shift of the origin of the acquired image due to the scanning and / or acquisition timing error of the camera. The external parameters can be defined by the 3D position and orientation of the camera with respect to a defined world coordinate system.

[0024] In an exemplary implementation, the internal parameters are considered to be within a specified range and the external parameters are adjusted. For example, parameters that change the x, y, z coordinates of points within the real-world scene can be elements of the calibration. Further, parameters that change the x-axis, y-axis, and z-axis (e.g., orientation) coordinates of the coordinate planes in the real-world scene can also be elements of the calibration. Camera calibration is described using two cameras for illustrative purposes. However, the exemplary implementation can include more than two cameras in a three-dimensional (3D) video conferencing system. For example, two or more NIR cameras and / or two or more RGB cameras can be used. Further, for illustrative purposes, a single matching pixel is described. The exemplary implementation can include the use of multiple (e.g., 10s, 100s, 1000s, etc.) pixels in camera calibration.

[0025] Reference is made to machine learning (ML) models, the use of ML models, and the training of ML models. An ML model can include the use of algorithms such as convolutional neural networks, recurrent neural networks, decision trees, random forests, k-nearest neighbor methods, and the like. For example, a convolutional neural network (CNN) can be used for pixel matching, pixel position determination, pixel identification, etc. A CNN architecture can include an input layer, one or more feature extraction layers, and one or more classification layers.

[0026] The input can receive three-dimensional (e.g., x, y, color) data (e.g., image data). The one or more feature extraction layers can include one or more convolutional layers and one or more pooling layers. The one or more convolutional layers and one or more pooling layers can find features within the image and progressively build higher-order features. The one or more feature extraction layers can be learning layers. The one or more classification layers can generate class probabilities or scores (e.g., indicating the likelihood of a match).

[0027] Training (e.g., training of the one or more feature extraction layers) can include, for example, supervised learning and unsupervised learning. Supervised learning includes a target / result variable (e.g., ground truth or dependent variable) to be predicted from a given set of predictor (independent) variables. Using these sets of variables, a function that can map the input to the desired output is generated. The training process continues until the model achieves the desired accuracy level based on the training data. Unsupervised learning machine learning uses algorithms to make inferences from a dataset consisting of input data without using labeled responses. Unsupervised learning can sometimes include clustering. Other types of training (such as hybrid training and reinforcement training) can also be used.

[0028] As described above, the training of the ML model can be continued until a desired level of accuracy is reached. The determination of the accuracy level may involve the use of a loss function. For example, the loss function may include hinge loss, logistic loss, negative log-likelihood, etc. To indicate that a sufficient level of accuracy in the ML model training has been reached, the loss function can be minimized. Regularization can also be used. By regularization, overfitting can be prevented. Overfitting can be prevented by making the weights and / or the changes in the weights small enough to prevent training (e.g., endless training).

[0029] Figure 2 is a block diagram showing a data flow according to at least one exemplary embodiment. The data flow is related to camera calibration in a multi-camera system. As shown in Figure 2, the data flow 200 includes a camera 5, a camera 10, a feature identifier 215 block, a feature matching 220 block, a mapping from rays to pixels 225 block, and a calibration 230 block. In the data flow 200, a first image is captured by the camera 5 and a second image is captured by the camera 10. Each image can be an image of a real-world scene (e.g., substantially the same real-world scene). According to an exemplary implementation, the real-world scene does not include a specially designed calibration target (e.g., including distinguishable characteristics for use in a camera calibration process). For example, each image can be of the scene shown in Figure 1A. The first image can be an NIR image and the second image can be an RGB image. The camera 5 can transmit the first image to the feature identifier 215 block, where a plurality of features in the first image (e.g., the NIR image) can be identified. For example, as shown in Figure 1A, the portion including the features 120-1, 120-2, 120-3 of the image may include the identified features 122-1, 122-2, 122-3. The plurality of identified features can be transmitted to the feature matching 220 block. The camera 10 transmits the second image to the feature matching 2 20It can be transmitted to the block, where the pixels of the identified multiple features can be matched (for example, located and matched) with the pixels of the multiple features in the second image. For example, the pixel 155 shown in FIG. 1B is an RGB pixel matched in the NIR image and can be used during calibration.

[0030] The multiple matching features from both the first image and the second image are transmitted from the feature matching 220 block to the ray-to-pixel mapping 225 block. The ray-to-pixel mapping 225 block can map rays in 3D space to pixels in 2D space for multiple matched features related to both the first image and the second image. For example, the ray 145 shown in FIG. 1C and the ray 150 shown in FIG. 1D can be mapped to pixels based on the sensor position and the corresponding pixel positions. The rays mapped to pixels can be used by the calibration 230 block to calibrate camera 5 and camera 10. The calibration 230 block can adjust the calibration parameters to align the positions of the matching pixels (from the matching features) so that the pixels in the first image and the pixels in the second image are substantially in the same position in the 2D space of the image. For example, the pixels 165-1’ and 165-2’ as shown in FIG. 1H can be aligned matching pixels.

[0031] The feature identifier 215 block can be configured to identify features in the image. The feature identifier 215 block can use corner and edge detection. Corner and edge detection can include the use of a Harris corner detector. The Harris corner detector is based on the local autocorrelation function of the signal, and the local autocorrelation function measures the local change of the signal with patches shifted slightly in different directions. To find corners in the input image, this technique analyzes the directional average intensity. The mathematical form of the Harris corner detector identifies the intensity differences for displacements in all directions (u, v).

[0032] The feature identifier 215 block can identify features using a machine learning (ML) model. The ML model can be trained using data (e.g., images) captured using a calibrated multi-camera system. The ML model can be a convolutional neural network. The ML model can use classification to identify a portion of an image as containing (or being) candidate features. In an exemplary implementation, camera 5 is a NIR camera configured to capture NIR images. The NIR images can be input into the ML model. The ML model can output classifications for multiple portions of the NIR image. The output can include a unique identifier for each portion of the NIR image, the location and / or dimensions (s) of each portion of the NIR image, and can indicate each portion of the NIR image as containing candidate features or not containing candidate features. Candidate features can include at least one pixel (s) (e.g., in the NIR image) that can be easily located precisely (e.g., indicating the location of pixel (s) in the NIR image) and can include, for example, corners, transitions, spots, etc.

[0033] The feature matching 220 block can be configured to use an ML model to identify matching features identified as candidate features in a first image in a second image. The ML model can be trained using data (e.g., images) captured using a calibrated multi-camera system. The ML model can be a convolutional neural network. The ML model can use scoring to identify pixels (s) in the second image as potentially matching pixels (s) in the first image. For example, a high score indicates that the pixel is likely to match, and a low score indicates that the pixel is less likely to match.

[0034] In an exemplary implementation, the first image is a NIR image (captured by camera 5), and the second image is an RGB image (captured by camera 10). The feature matching 220 block receives data including unique identifiers for each part of the NIR image, the position and / or dimensions (plural possible) of each part of the NIR image, and can indicate each part of the NIR image as including candidate features or not including candidate features. Data including parts of the NIR image including candidate features can be input into the ML model. A score (e.g., a score indicating the likelihood of a match) can be assigned to each pixel associated with a part of the NIR image including candidate features by the ML model.

[0035] For each candidate feature, a search window can be defined in the second image (e.g., the RGB image). Each pixel within the search window can have a score assigned to it using the ML model, where a high score indicates a pixel likely corresponding to a pixel (plural possible) within the candidate feature and a low score in other images. A second ML model (e.g., a second neural network) can be used to predict the exact match position (e.g., the x,y position) of each pixel within the search window. An estimated offset (e.g., a prediction of how far the exact match pixel is from the current pixel along the x-axis and y-axis of the image) can be generated for each pixel within the search window. The estimated offset of the match pixel (e.g., a pixel having a score passing a criterion (e.g., exceeding a threshold)) can be calculated. For example, if a pixel within the search window is found to have a sufficiently high score (e.g., a high likelihood of a match), the estimated offsets of the pixel and its neighboring pixels can be averaged to obtain the estimated offset of the best match pixel (as the target match pixel), and a match between the NIR and the RGB can be generated. The match (e.g., the position of a pixel within the second image or the RGB image) can be output from the second ML model along with the estimated offset. In one example, the window (plural possible) can be based on a previous calibration. For example, the position and dimensions of the window can be based on the position and dimensions of the window determined (and stored in memory) during a previous calibration.

[0036] The above ML model(s) can be trained using data (e.g., images) captured using a calibrated multi-camera (e.g., NIR and RGB) system. Training may include generating scores for pixels related to candidate features. The ground truth data may include the number of features, the positions of the pixels within the features, the pixel scores, and the offsets. Training may include adjusting the weights (e.g., neural network weights) associated with the ML model until the score output of the ML model passes a criterion based on comparison with the ground truth data.

[0037]

Number

[0038] Returning to FIG. 2, calibration block 230 can be configured to calibrate cameras 5 and 10 with respect to each other. The calibration can include aligning pixels (within the images) generated by camera 5 based on a ray (in the real world) and pixels (within the images) generated by camera 10 based on a ray (in the real world) to have the same position within their respective images. The calibration can include adjusting calibration parameters such that a first ray (e.g., R1, ray 145) and a second ray (e.g., R2, ray 150) are associated with a target pixel position (P). The target pixel position can be the point that is the intersection of the first ray and the second ray in the real-world scene coordinate system. Adjusting the calibration parameters such that the first ray and the second ray are interpreted as intersecting at a point in real space (e.g., 3D space) can include shifting the processed sensor positions associated with the first ray and the second ray in 2D by comparing them to an uncalibrated camera(s). The calibration parameters can be included in the camera calibration matrix M. Thus, by modifying the calibration matrix M, points p1 and p2 can be translated (as matching pixels) such that rays R1, R2 intersect at point P. In an exemplary implementation, there must be a large number of matching pixels (e.g., hundreds of matching pixels).

[0039] The calibration parameters can include intrinsic parameters and extrinsic parameters. The intrinsic parameters can include the effective focal length, or the distance from the image plane to the projection center, lens distortion coefficients, the magnification of x, and a shift of the origin of the acquired image due to the camera's scanning and / or acquisition timing error. The extrinsic parameters can be defined by the 3D position and orientation of the camera with respect to a defined world coordinate system.

[0040] In an exemplary implementation, the internal parameters are considered to be within a specified range and the external parameters are adjusted. For example, parameters that vary the x, y, z coordinates of points within a real-world scene can be elements of calibration. Further, parameters that vary the x-axis, y-axis, z-axis (e.g., orientation) coordinates of a coordinate plane within a real-world scene can be elements of calibration.

[0041] FIG. 3 is a block diagram showing a videoconference system according to at least one exemplary embodiment. The elements shown in FIG. 3 are related to (or include) the camera calibration of a videoconference system as shown in FIG. 2. As shown in FIG. 3, the videoconference system 300 includes at least one processor 305, at least one memory 310, a camera interface 315, a feature identifier 215 block, a feature matching 220 block, a mapping from rays to pixels 225 block, and a calibration 230 block. The feature identifier 215 block, the feature matching 220 block, the mapping from rays to pixels 225 block, and the calibration 230 block have been described above.

[0042] At least one processor 305 is utilized to execute instructions stored in at least one memory 310, thereby enabling the realization of various features and functions described herein, or additional or alternative features and functions. The at least one processor 305 may be a general-purpose processor. The at least one processor 305 may be a graphics processing unit (GPU) and / or an audio processing unit (APU). The at least one processor 305 and the at least one memory 310 may be utilized for various other purposes. In particular, the at least one memory 310 may represent an example of various types of memory and related hardware and software that can be used to implement any one of the modules described herein.

[0043] At least one memory 310 may be configured to store data and / or information related to the video conferencing system 300. For example, at least one memory 310 may be configured to store code related to calibrating a camera using the features of an identified real-world scene. According to an exemplary implementation, the real-world scene does not include a specially designed calibration target (e.g., including features that can be identified for use in a camera calibration process). For example, at least one memory 310 may be configured to store code related to at least one trained ML model. At least one memory 310 may be a non-transitory computer-readable medium having code that, when executed by a processor 305, causes the processor 305 to execute one or more of the techniques described herein. At least one memory 310 may be a shared resource. For example, the model training system 300 may be an element of a larger system (e.g., a server, a personal computer, a mobile device, etc.). Thus, at least one memory 310 may be configured to store data and / or information related to other elements within the larger system.

[0044] Figures 4 and 5 are flowcharts illustrating methods according to exemplary embodiments. The methods described with respect to Figures 4 and 5 may be executed due to the execution of software code stored in a memory (e.g., a non-transitory computer-readable storage medium) associated with an apparatus and executed by at least one processor associated with the apparatus.

[0045] However, alternative embodiments are contemplated, such as a system embodied as a dedicated processor. The dedicated processor can be a Graphics Processing Unit (GPU) and / or an Audio Processing Unit (APU). The GPU can be a component of a graphics card. The APU can be a component of a sound card. The graphics card and / or the sound card can also include video / audio memory, a Random Access Memory Digital-to-Analogue Converter (RAMDAC), and driver software. The driver software can be software code stored in the memory mentioned above. The software code can be configured to execute the method described herein.

[0046] The methods described below are described as being executed by a processor and / or a dedicated processor, but the methods are not necessarily executed by the same processor. In other words, at least one processor and / or at least one dedicated processor may execute the methods described below with respect to FIGS. 4 and 5.

[0047] FIG. 4 shows a block diagram of a method for calibrating a camera according to at least one exemplary embodiment. As shown in FIG. 4, in step S405, a first image is captured by a first camera. For example, the first camera may be sensitive to light of a first spectrum (e.g., IR, NIR, etc.) and may have a first light source (e.g., an IR or NIR flash associated with the first camera). In an exemplary implementation, the first camera may be an NIR camera and the first image can be an NIR image.

[0048] In step S410, a second image is captured by a second camera. For example, the second camera may be sensitive to light of a second spectrum (e.g., visible light) and may have a second light source (e.g., indoor light, sunlight, etc.). In an exemplary implementation, the second camera may be a visible light or RGB camera, and the first image can be an RGB image.

[0049] In step S415, features are identified in the first image. Feature identification may include the use of corner and edge detection. Corner and edge detection may include the use of a Harris corner detector. The Harris corner detector is based on the local autocorrelation function of a signal, and the local autocorrelation function measures the local change of a signal having patches shifted slightly in different directions. To find corners in an input image, this technique analyzes the directional average intensity. The mathematical form of the Harris corner detector identifies the intensity differences for displacements in all directions (u, v).

[0050] Alternatively, a machine learning (ML) model can be used to identify features of an image. The ML model can be trained using data (e.g., images) captured using a calibrated multi-camera system. The ML model can be a convolutional neural network. The ML model can use classification to identify that a portion of an image contains (or is) a candidate feature. In an exemplary implementation, the camera is a NIR camera configured to capture a NIR image. The NIR image can be input into the ML model. The ML model can output classifications for multiple portions of the NIR image. The output can include a unique identifier for each portion of the NIR image, the position and / or dimensions (s) of each portion of the NIR image, and can indicate each portion of the NIR image as containing a candidate feature or not containing a candidate feature. A candidate feature can include at least one pixel (s) (e.g., of the NIR image) that can be easily identified in terms of its exact location (e.g., indicating the position of the pixel (s) within the NIR image) and can include (e.g., a corner, a transition, a spot, etc.).

[0051] In step S420, features in the second image that match the features identified in the first image are identified. For example, using an ML model, the matching features identified as candidate features in the first image can be identified in the second image. The ML model can be trained using data (e.g., images) captured using a calibrated multi-camera system. The ML model can be a convolutional neural network. The ML model can use scoring to identify pixels (if any) in the second image that may match pixels (if any) in the first image. For example, a high score can indicate that the pixel is likely to match, and a low score can indicate that the pixel is less likely to match.

[0052] In an exemplary implementation, the first image is an NIR image and the second image is an RGB image. The ML model can use data including unique identifiers for each part of the NIR image, the position and / or dimensions (if any) of each part of the NIR image, and can indicate each part of the NIR image as including candidate features as an input to the ML model. In another implementation, the first image is an RGB image and the second image is an NIR image.

[0053] For each candidate feature, a search window can be defined in a second image (e.g., an RGB image). Each pixel within the search window can be assigned a score using an ML model, where a high score indicates a pixel likely corresponding to a pixel(s) within the candidate feature and a low score for other pixels. A second ML model (e.g., a second neural network) can be used to predict the exact match position (e.g., x, y position) of each pixel within the search window. An estimated offset for each pixel within the search window (e.g., a prediction of how far the exact match pixel is from the current pixel along the x and y axes of the image) can be generated. The estimated offset of a match pixel (e.g., a pixel having a score passing a criterion (e.g., exceeding a threshold)) can be calculated. For example, if a pixel within the search window is found to have a sufficiently high score (e.g., likely to match), the estimated offsets of the pixel and its neighboring pixels can be averaged to obtain the estimated offset of the best match pixel (as the target match pixel), and a match between the RGB and the NIR can be generated. The match (e.g., the position of a pixel within the second image or RGB image) can be output from the second ML model along with the estimated offset. In one example, the window(s) can be based on a previous calibration. For example, the position and dimensions of the window can be based on the position and dimensions of the window determined (and stored in memory) during a previous calibration.

[0054]

Number

[0055] Returning to FIG. 4, in step S430, based on the mapping, the first camera and the second camera are calibrated. For example, the calibration may include aligning a pixel (in the image) generated by camera 5 based on a ray (in the real world) and a pixel (in the image) generated by camera 10 based on a ray (in the real world) to have the same position in their respective images. The calibration may include adjusting the calibration parameters such that a first ray (e.g., R1, ray 145) and a second ray (R2, ray 150) are associated with a target pixel position (P). The target pixel position can be the same position (e.g., x, y coordinates) in a 2D coordinate system representing a part of the image. The target pixel position can be associated with the camera sensor position and the processed interpretation of the pixel associated with the camera sensor position.

[0056] The calibration can include adjusting the calibration parameters such that the first ray and the second ray are interpreted as intersecting at a point in real space (e.g., 3D space), and can include shifting the processed sensor positions associated with the first ray and the second ray 2D compared to the uncalibrated camera(s). The calibration parameters can include a camera calibration matrix M. The calibration parameters can include internal parameters and external parameters. The internal parameters can include the effective focal length, or the distance from the image plane to the projection center, lens distortion coefficients, the magnification of x, and the shift of the origin of the acquired image due to the camera scanning and / or acquisition timing error. The external parameters can be defined by the 3D position and orientation of the camera with respect to a defined world coordinate system.

[0057] In an exemplary implementation, the internal parameters are considered to be within a specified range and the external parameters are adjusted. For example, parameters that change the x, y, z coordinates of points within a real-world scene can be elements of calibration. Further, parameters that change the x-axis, y-axis, and z-axis (e.g., orientation) coordinates of a coordinate plane in a real-world scene can be elements of calibration.

[0058] FIG. 5 shows a block diagram of a method for matching pixels according to at least one exemplary embodiment. As shown in FIG. 5, in step S505, candidate features are selected from a first image. For example, the candidate features can include at least one pixel (e.g., a corner, transition, spot, etc.) that can be easily identified to accurately locate (e.g., indicate the position of a pixel(s) in the NIR image), such as in an NIR image. The candidate features can be one of a plurality of features identified using an ML model. The first image can be captured by a first camera. The first camera can be sensitive to light of a first spectrum (e.g., IR, NIR, etc.) and can have a first light source (e.g., an IR or NIR flash associated with the first camera). In an exemplary implementation, the first camera can be an NIR camera and the first image can be an NIR image. In another implementation, the first camera and the second camera can be sensitive to light of the same spectrum. For example, an exemplary implementation can reduce the influence of view-dependent effects associated with highly reflective surfaces and / or complex micro-shapes (e.g., present in an image including hair).

[0059] In step S510, the RGB pixels in the second image are matched with the candidate features. For example, using an ML model, the matching features identified as candidate features in the first image can be identified in the second image. The second camera may be sensitive to light of the second spectrum (e.g., visible light) and may have a second light source (e.g., indoor light, sunlight, etc.). In an exemplary implementation, the second camera can be a visible light or RGB camera, and the first image can be an RGB image.

[0060] In step S515, a score is assigned to the matching RGB pixels. For example, the ML model can use scoring to identify pixels (if any) in the second image that may match pixels (if any) in the first image. For example, a high score can indicate that the pixel is likely to match, and a low score can indicate that the pixel is less likely to match.

[0061] In an exemplary implementation, the first image is an NIR image and the second image is an RGB image. The ML model can use data including unique identifiers for each part of the NIR image, the position and / or dimensions (if any) of each part of the NIR image, and can indicate each part of the NIR image as including candidate features as an input to the ML model.

[0062] For each candidate feature, a search window can be defined in the second image (e.g., RGB image). Each pixel within the search window can be assigned a score using the ML model, where a high score indicates a pixel that is likely to correspond to a pixel (if any) within the candidate feature, and a low score for other pixels.

[0063] In step S520, the direction of the position of the target matching pixel (or target pixel) is predicted. For example, when it is found that the pixels within the search window have a sufficiently high score (e.g., a high likelihood of matching), the estimated offset of each pixel within the search window can be determined. By averaging the estimated offsets of the best matching pixel and its neighboring pixels, the position of the best matching pixel (as the target matching pixel) can be obtained, and a match between the RGB of the NIR can be generated.

[0064] In step S525, a direction is assigned to the matched RGB pixel. For example, a second ML model (e.g., a second neural network) can generate an estimated offset (e.g., a prediction of how far the target matching pixel is from the current pixel (e.g., the pixel having a score passing (e.g., exceeding) a threshold) along the x-axis and y-axis of the image). The match (e.g., the position of the pixel in the second image or RGB image) can be output from the second ML model together with the estimated offset. In one example, the window(s) can be based on previous calibration. For example, the position and dimensions of the window can be based on the position and dimensions of the window determined (and stored in memory) during previous calibration.

[0065] Embodiments may include a device, a system, a non-transitory computer-readable medium (storing computer-executable program code executable on a computer system), and / or a method. The method is executable to perform a process that includes: a first camera having sensitivity to light of a first spectrum and having a first light source capturing a first image of a real-world scene; a second camera having sensitivity to light of a second spectrum and having a second light source capturing a second image of the real-world scene; identifying at least one feature in the first image; using a machine learning (ML) model to identify at least one feature in the second image that matches at least one feature identified in the first image; mapping pixels in the first image and the second image to rays in three-dimensional (3D) space based on the matching at least one feature; and calibrating the first camera and the second camera based on the mapping.

[0066] Realization examples may include one or more of the following features. For example, the first camera may be a near-infrared (NIR) camera, and the second camera may be a visible light camera. An ML model may be used to identify at least one feature in the first image. An algorithm may be used to identify at least one feature in the first image. An ML model may be used to match at least one feature in the first image with at least one feature in the second image, and a score may be assigned to at least one pixel in the second image based on the likelihood that at least one pixel in the second image matches a pixel of at least one feature in the first image. An algorithm may be used to match at least one feature in the first image with at least one feature in the second image, and a score may be assigned to at least one pixel in the second image based on the likelihood that at least one pixel in the second image matches a pixel of at least one feature in the first image, and a direction may be assigned to at least one pixel in the second image based on a prediction of the position of a target pixel.

[0067] Using an algorithm, at least one feature in a first image may be identified, and matching at least one feature in the first image with at least one feature in a second image may include using a first ML model to select candidate features from at least one feature in the first image, matching at least one pixel in the second image with the pixels of the candidate features, assigning a score to at least one matching pixel in the second image based on the likelihood that the at least one pixel matches one of the at least one feature in the first image, using a second ML model to predict the direction of the position of a target pixel, and assigning a direction to at least one matching pixel in the second image. Calibrating the first camera and the second camera may be based on at least one matching pixel in the second image associated with a candidate feature having the highest score and the direction of at least one matching pixel in the second image having the highest score, and the direction may be based on at least one matching pixel in the second image having the highest score and neighboring pixels. The method may further include selecting at least one search window in the second image based on a previous calibration. The machine learning model may be trained with data captured from a calibrated multi-camera system.

[0068] FIG. 7 illustrates an example of a computer device 700 and a mobile computer device 750 that can be used with the technology described herein. Computing device 700 is intended to represent various forms of digital computers, such as a laptop, desktop, workstation, personal digital assistant, server, blade server, mainframe, and other appropriate computers. Computing device 750 is intended to represent various forms of mobile devices, such as a personal digital assistant, cellular phone, smartphone, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit examples of the inventions described and / or claimed herein.

[0069] Computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 that connects to memory 704 and high-speed expansion port 710, and a low-speed interface 712 that connects to low-speed bus 714 and storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 may be interconnected using various buses and may be mounted on a common motherboard or in other manners as appropriate. Processor 702 can process instructions for execution within computing device 700, and the instructions may include instructions stored in or on storage device 706 that display graphical information for a GUI on an external input / output device such as display 716 coupled to high-speed interface 708. In some implementations, multiple memories and multiple types of memories may be used, along with multiple processors and / or multiple buses as appropriate. Also, multiple computing devices 700 may be connected, and each device may provide a portion of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0070] Memory 704 stores information within computing device 700. In one implementation, memory 704 is one or more volatile memory units. In another implementation, memory 704 is one or more non-volatile memory units. Also, memory 704 may be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0071] Storage device 706 can provide large-capacity storage to computing device 700. In one implementation, storage device 706 may be a computer-readable medium, such as a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a device in a storage area network or other configuration, or may include such a computer-readable medium. A computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 704, storage device 706, or memory on processor 702.

[0072] The high-speed controller 708 manages bandwidth-intensive operations for the computing device 700, while the low-speed controller 712 manages operations that are not as bandwidth-intensive. Such an assignment of functions is merely exemplary. In one implementation, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 710 that can receive various expansion cards (not shown). In this implementation, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, etc., or to a networking device such as a switch or router, e.g., via a network adapter.

[0073] The computing device 700 may be implemented in many different forms as shown in the figure. For example, it may be implemented as a standard server 720, or multiple times as a group of such servers. It may also be implemented as part of a rack server system 724. Additionally, it may be implemented in a personal computer such as a laptop computer 722. Or, components from the computing device 700 may be combined with other components within a mobile device (not shown) such as the device 750. Each of such devices may include one or more of the computing devices 700, 750, and the overall system may be composed of multiple computing devices 700, 750 that communicate with each other.

[0074] Computing device 750 includes, among several components, in particular, a processor 752, a memory 764, input / output devices such as a display 754, a communication interface 766, and a transceiver 768. The device 750 may be provided with a storage device such as a microdrive or other device to further provide storage. Each of the components 750, 752, 764, 754, 766, and 768 is interconnected using various buses, and some of these components may be mounted on a common motherboard or in other suitable manners as appropriate.

[0075] Processor 752 can execute instructions within computing device 750, including instructions stored in memory 764. The processor may be implemented as a chipset of separate multiple analog and digital processors. The processor may provide coordination of other components of the device 750, such as, for example, control of the user interface, applications executed by the device 750, and wireless communication by the device 750.

[0076] Processor 752 may communicate with the user through control interface 758 and display interface 756 coupled to display 754. Display 754 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. Display interface 756 may include appropriate circuitry for driving display 754 to display graphic information and other information to the user. Control interface 758 may receive commands from the user, convert these commands, and send them to processor 752. Additionally, an external interface 762 may be provided to communicate with processor 752 to enable proximity area communication between device 750 and other devices. External interface 762 may provide, for example, wired communication in one implementation, wireless communication in other implementations, or multiple interfaces may be used.

[0077] Memory 764 stores information within computing device 750. Memory 764 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Also, an extended memory 774 may be provided and connected to device 750 via an extended interface 772, and the extended interface 772 may include, for example, a SIMM (Single In Line Memory Module) card interface. Such extended memory 774 may provide additional storage space for device 750, or may store applications or other information for device 750. Specifically, extended memory 774 may include instructions for executing or complementing the above processes, and may also include secure information. For this reason, extended memory 774 may be provided, for example, as a security module for device 750, and may be programmed using instructions that enable secure use of device 750. In addition, a secure application may be provided via the SIMM card together with additional information, such as by placing identification information on the SIMM card in a non-hackable manner.

[0078] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation example, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 764, extended memory 774, or memory on processor 752, which may be received via, for example, transceiver 768 or external interface 762.

[0079] Device 750 may communicate wirelessly via a communication interface 766, which may include a digital signal processing circuit as needed. The communication interface 766 may provide communication under various modes or protocols, such as GSM (registered trademark) voice telephone, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA (registered trademark), CDMA2000, or GPRS. Such communication may be performed, for example, via a radio frequency transceiver 768. In addition, short-range communication may be performed, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 770 may provide additional wireless data related to navigation and location to device 750, and the data may be appropriately used by an application running on device 750.

[0080] Also, device 750 may communicate audibly using an audio codec 760 that can receive speech information from a user and convert it into usable digital information. Similarly, the audio codec 760 may generate audible sound for the user, such as via a speaker in the handset of device 750. Such sound may include sound from a voice call, recorded sound (e.g., voice message, music file, etc.), and further sound generated by an application running on device 750.

[0081] Computing device 750 may be implemented in many different forms as shown. For example, it may be implemented as a mobile phone 780. It may also be implemented as part of a smartphone 782, a personal digital assistant, or other similar mobile device.

[0082] Exemplary embodiments may include various modifications and alternative forms, and such embodiments are shown by way of example in the drawings and are described in detail herein. However, it is not intended to limit the exemplary embodiments to the specific forms disclosed, but rather, the exemplary embodiments are to be understood to cover all modifications, equivalents, and alternatives within the scope of the claims. Throughout the description of the drawings, like numbers refer to like elements.

[0083] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system. The programmable system may include an implementation in one or more computer programs that are executable and / or interpretable on the programmable system. The programmable system may include at least one programmable dedicated or general-purpose processor coupled to a storage system for storing and retrieving data, at least one input device, and at least one output device. Various implementations of the systems and techniques described herein may be realized as, and / or generally referred to herein as, circuits, modules, blocks, or systems that combine software and hardware aspects. For example, a module may include functions / acts / computer program instructions that are executed on a processor (e.g., a processor formed on a silicon substrate, a GaAs substrate, etc.) or other programmable data processing device.

[0084] Some of the above-exemplified embodiments have been described as processes or methods shown as flowcharts. Although a flowchart describes operations as a sequential process, many of the operations can be performed in parallel, simultaneously, or concurrently. Further, the order of the operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not included in the drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0085] The above-described method, some of which is shown by a flowchart, may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. When implemented as software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine or computer-readable medium such as a storage medium. A processor(s) may execute the necessary tasks.

[0086] The specific structural and functional details disclosed herein are merely representative for purposes of describing exemplary embodiments. However, the exemplary embodiments may be embodied in many alternative forms and should not be construed as limited to only the embodiments described herein.

[0087] Terms such as "first," "second," etc. may be used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0088] When an element is said to be connected or coupled to another element, it will be understood that the one element can be directly connected or coupled to the other element or that intervening elements may be present. In contrast, when an element is said to be directly connected or directly coupled to another element, intervening elements are absent. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" and "directly between", "adjacent" and "directly adjacent", etc.).

[0089] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the terms "comprises", "comprising", "includes" and / or "including" specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0090] Note that in some alternative implementations, the functions / acts described may occur in a different order than shown in the figures. For example, two figures shown in succession may in fact be executed simultaneously or, depending upon the related functions / acts, may be executed in the reverse order.

[0091] Unless otherwise defined, all terms (including technical and scientific terms) used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. Further, terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning that coincides with the meaning in the context of the relevant art, and it will be understood that they shall not be interpreted in an idealized or overly formal sense unless explicitly so defined herein.

[0092] Some of the above exemplary embodiments and corresponding detailed descriptions are presented from the perspective of symbolic representations of operations related to software, or algorithms, and data bits in a computer memory. These descriptions and representations are for those of ordinary skill in the art to effectively convey the essence of their research to others outside the art. An algorithm, as used herein and as commonly used, is considered to be a self-consistent series of steps that result in a desired outcome. The steps require physical operations on physical quantities. Although not necessarily so, typically these quantities take the form of optical, electrical, magnetic signals capable of being stored, transferred, combined, compared, and other operations. Referring to such signals as bits, values, elements, symbols, characters, terms, numbers, etc. has proven to be mainly convenient from common usage.

[0093] In the above exemplary embodiments, references to acts and symbolic representations of operations that may be implemented as program modules or functional processes (e.g., in the form of a flowchart) include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types and may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware includes one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate arrays (FPGAs), computers, and the like.

[0094] However, it should be recognized that all of these and similar terms are to be associated with appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified or apparent from the discussion, terms such as processing, computing, calculating, determining, displaying, etc. refer to the actions and processes of a computer system, or similar electronic computing device, that manipulate and transform data represented as physical electronic quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the memories or registers of the computer system, or other such information storage, transmission, display devices.

[0095] Note that the software implementation of the exemplary embodiments is typically encoded on some form of non-transitory program storage medium or realized via some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory, or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be a twisted pair, coaxial cable, fiber optic, or other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0096] Finally, while the appended claims set forth certain combinations of the features described herein, the scope of the disclosure is not limited to the specific combinations claimed below, but instead extends to any combination of the features or implementations disclosed herein, whether or not that specific combination is presently specifically recited in the appended claims.

Claims

**Claim 1** A first camera having sensitivity to light of a first spectrum and having a first light source captures a first image of a real-world scene; A second camera having sensitivity to light of a second spectrum and having a second light source captures a second image of the real-world scene; Identifying at least one feature in the first image; Using a machine learning (ML) model to identify at least one feature in the second image that matches the at least one feature identified in the first image; Based on the matched at least one feature, mapping each of a first pixel in the first image and a second pixel in the second image to a first ray and a second ray respectively in a three-dimensional (3D) space; Calibrating the first camera and the second camera based on the mapping, wherein the first ray and the second ray intersect at a position of a target pixel in the real-world scene, the calibration including calibrating the first camera and the second camera such that a position of the first pixel in the first image and a position of the second pixel in the second image are the same. A method. **Claim 2** The method according to claim 1, wherein the first camera is a near-infrared (NIR) camera and the second camera is a visible light camera. **Claim 3** The method according to claim 1 or 2, wherein an ML model is used to identify the at least one feature in the first image. **Claim 4** The method according to any one of claims 1 to 3, wherein an algorithm is used to identify the at least one feature in the first image. **Claim 5** Using an ML model to collate the at least one feature in the first image with the at least one feature in the second image, A score is assigned to the at least one pixel of the second image based on the likelihood that the at least one pixel of the second image matches the pixel of the at least one feature in the first image. The method according to any one of claims 1 to 4. **Claim 6** The method according to any one of claims 1 to 5, further comprising selecting at least one search window in the second image based on a previous calibration. **Claim 7** The method according to any one of claims 1 to 6, wherein the ML model is trained with data captured from a calibrated multi-camera system. **Claim 8** A three-dimensional (3D) video conferencing system, comprising: a first camera; a second camera; a memory including a code segment representing a plurality of computer instructions; a processor configured to execute the code segment, wherein the plurality of computer instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7 for calibration of the first camera and the second camera. A three-dimensional (3D) video conferencing system. **Claim 9** A program comprising instructions that, when executed, cause a processor of a computer system to perform the method according to any one of claims 1 to 7. **Claim 10** The program according to claim 9, wherein the light of the first spectrum and the light of the second spectrum are light of the same spectrum.

Images