Method and system for determining spatial coordinates of a 3D reconstruction of at least a portion of a physical object

Abstract

The present invention provides a method and system for determining the spatial coordinates of a 3D re-construction of at least a portion of a physical object. The method comprises: acquiring a first image including at least a portion of a first physical object from a first camera system; acquiring a second image from a second camera system, the second image including at least a portion of a second physical object associated with a known geometric property, wherein the first camera system and the second camera system have a known spatial relationship; determining the scale of at least a portion of the second physical object based on the second image and the known geometric property of the at least a portion of the second physical object; and determining the absolute scale characteristic of the first physical object based on the known spatial relationship and the scale of the at least a portion of the second physical object.

Description

[0001] This application is a divisional application of the invention patent application with application number 201480083523.5, filed on November 21, 2014, and invention name “Method and system for determining the spatial coordinates of a 3D heavy component in absolute spatial proportion of at least a portion of a physical object”. Technical Field

[0002] The present disclosure relates to a method and system for determining spatial coordinates of a 3D re-construction of at least a portion of a physical object at an absolute spatial scale. Background Art

[0003] Computer vision methods involving image analysis are often used, for example, for navigation, object recognition, 3D reconstruction, and augmented reality applications. Images can be captured by a single camera or by different cameras. Image feature (such as corners and edges) detection and image feature extraction are common steps in various computer vision methods or algorithms, such as image-based recognition, image-based tracking, image-based reconstruction, image-based classification, and image distortion. For example, visual simultaneous localization and mapping (SLAM) is a well-known computer vision method that reconstructs a real environment through one or more cameras and tracks this one or more cameras. Assuming that one or more cameras have captured at least two images, typical SLAM methods include: feature detection, description, matching, triangulation, and (global) graph optimization.

[0004] The structure of the object is determined from a set of images captured by a monocular capture device, in such a way that a reconstructed spatial (or geometric) structure is formed in accordance with scale. That is, the reconstruction uses spatial units for which the scale factor of the absolute spatial units (such as the unit meter) is unknown. In many applications, it is necessary to obtain, for example, a reconstructed structure with absolute units, also referred to as a reconstructed structure "at absolute scale". This often requires knowledge of at least one distance at absolute scale, for example, the distance between multiple parts of the object, or the distance between the position of the camera relative to the object when the corresponding images for reconstruction were taken.

[0005] Therefore, a common problem of various SLAM and SfM systems is that the geometric model of the real environment is reconstructed based on the scale as an uncertain factor. If the position of the real objects and the pose of the camera taking the reconstructed image are also unknown, the absolute spatial scale of the scene cannot be determined. For example, according to Figure 2aThe two images shown - one taken from the front I (W1) and one from the right I (W2) - do not allow to tell whether it is a real car of full size or a realistic small mini car. Therefore, it is not possible to tell how many meters (in the case of a full-size car) or just a few centimeters (in the case of a mini car) the cameras that took the two images were separated from each other. However, if other information about the absolute spatial scale of the cameras (e.g., the two cameras are 2.34 meters apart) or the absolute spatial scale of parts of the object (e.g., the car headlights are 3.45 meters apart) is known, reconstruction can be performed in absolute scale.

[0006] In cases where the absolute spatial scale of the scene cannot be determined, the SLAM system can randomly assign a scale, for example, by determining the initial keyframe from pixel disparity measurements in image space and assuming some common real-world distance for the baseline between the two corresponding camera poses. As a result, the reconstructed 3D features have coordinates in a coordinate system relative to the geometric model, where the scale factor of the model relative to the absolute coordinates is unknown when the absolute coordinates are in the real world, for example, millimeters, centimeters, meters, or inches. In addition, the camera positions calculated from the recovered geometric model are also scale-dependent, see reference [4].

[0007] The scale factor is uncertain, which poses a challenge to determine the actual movement of the camera in absolute scale in vision-based navigation of (for example) robotic systems or vehicles, and in augmented reality applications to correctly overlay virtual visual information relative to the camera image of the real environment. As an example, a visual navigation application can determine the shape of the camera motion (e.g., the camera moves on a circular path), but it cannot determine the translational portion (e.g., distance or position) in absolute scale, for example, if the radius of the circle is 1 meter or 10 meters. As another example, consider an augmented reality application that superimposes a piece of virtual furniture that is spatially registered on a live video of the environment. If the camera is tracked in a coordinate system at a random (i.e., arbitrary) scale, the superimposed virtual furniture will also have an arbitrary scale. A virtual cupboard that is 2 meters high may appear to be three times the height of a 1-meter-high table, or it may appear to be half the height of the table, depending on the arbitrary scale chosen during reconstruction. Obviously, this is undesirable. Instead, a virtual cupboard that is 2 meters high should appear to be twice the height of the real table that is 1 meter high next to it. The scale of real objects and virtual objects augmented by superposition in the camera should be consistent. To achieve this, it is necessary to know the (correct) absolute scale of the geometric model of the real environment.

[0008] Moreover, the problem of scale factor uncertainty is particularly evident in cases where multiple geometric models of multiple real objects have been created separately by the same visual SLAM system for simultaneously tracking multiple real objects, such as in reference [8]. Typically, a random scale value is applied to each of the multiple geometric models. If the SLAM system switches between multiple geometric models, the scale may change, thus severely affecting the user experience in computer vision applications such as augmented reality.

[0009] Various methods have been proposed for determining the correct scaling factor that defines the actual size of the reconstructed geometric model of the real environment as it would be in the real world.

[0010] For example, Davison et al. in [1] proposed to introduce a calibration object with known absolute spatial dimensions into the absolute scale determination scene of the SLAM system. As a result, they needed to change the scene appearance because they used the same camera to capture the calibration object and to capture the scene for SLAM reconstruction. The user also needs to obtain the calibration object.

[0011] In reference [5], Lemaire et al. proposed using a stereo camera system (i.e., two cameras displaced by overlapping camera cones) to solve the problem of determining absolute scale in SLAM systems. However, using stereo cameras only solves part of the problem, because the displacement between the two cameras must be significant relative to the distance of the environment in order to reliably calculate the depth of the environment. It is also necessary to know the displacement between the two cameras in absolute scale, that is, in units such as millimeters, centimeters, meters, or inches.

[0012] Reference

[14] also discloses a scheme for estimating absolute scale using a multi-camera setup with overlapping camera frustums. However, the displacement between the two cameras must be significant relative to the ambient distance in order to reliably compute the ambient depth.

[0013] In reference [6], Lieberknecht et al. integrated depth information into monocular vision-based SLAM by using RGB-D cameras to achieve accurate reconstruction of scaled geometric models. RGB-D cameras can provide absolute depth information associated with image pixels. The absolute scale can be determined based on the known depth information at the absolute scale. However, compared with common monocular RGB cameras, RGB-D camera devices are not commonly used in handheld devices (e.g., mobile phones, tablets, or PDAs). Similarly, if there is a lot of infrared ambient light, such as in outdoor environments during the day, active stereo depth cameras based on projecting infrared light into the scene cannot operate reliably.

[0014] In reference [7], Klein et al. solved the problem of scale estimation in the following way: when a camera captures two images required for 3D triangulation, a baseline (i.e., the distance in absolute scale) between the two camera positions is manually defined, and the 3D triangulation is used to reconstruct the environment.

[0015] Absolute scale can also be estimated using a fusion of sensors and an inertial measurement unit (IMU), as disclosed in reference [9]. One problem with this approach is that inaccurate sensor values ​​lead to inaccurate estimates of scale. Expensive (i.e., computationally intensive) techniques, such as "Kalman filtering" or "bundle adjustment", are used to solve this problem, but the accuracy of IMUs integrated with off-the-shelf devices (such as mobile phones) is usually not sufficient to accurately estimate absolute scale.

[0016] Therefore, it is necessary to provide a method and system for determining the spatial coordinates of a 3D heavy component at an absolute spatial scale of at least a portion of a real object, which method and system can reconstruct the real object at an absolute scale, or determine a scale factor that maps the coordinates of a heavy component at an arbitrary scale to an absolute scale. Summary of the invention

[0017] According to a first aspect, the present invention discloses a method for determining the spatial coordinates of a 3D re-construction of at least a portion of a physical object in absolute spatial scale, comprising the following steps:

[0018] a) receiving image information of a first image, the first image including at least a portion of a first physical object captured by a first camera

[0019] b) receiving image information of a third image, the third image including the at least a portion of the first object captured by the third camera,

[0020] c) receiving image information of a second image, the second image comprising at least a first portion of a second object captured by a second camera,

[0021] d) receiving image information of a fourth image, the fourth image including at least a second portion of the second object captured by a fourth camera,

[0022] wherein the viewing cone of the first camera and the viewing cone of the second camera do not overlap, and wherein the viewing cone of the third camera and the viewing cone of the fourth camera do not overlap,

[0023] e) providing a first spatial transformation between the first camera and the second camera, and providing a second spatial transformation between the third camera and the fourth camera,

[0024] f) providing first scale information and second scale information, the first scale information indicating an absolute spatial scale of at least a first portion of the second object, and the second scale information indicating an absolute spatial scale of at least a second portion of the second object,

[0025] g) determining a pose of at least a portion of the second camera and a pose of at least a portion of the fourth camera according to the second image, the first scale information, the fourth image, and the second scale information, wherein the pose of at least a portion of the second camera and the pose of at least a portion of the fourth camera are defined in a second common coordinate system,

[0026] h) determining a pose of at least a portion of the first camera according to a pose of at least a portion of the second camera and the first spatial transformation, and determining a pose of at least a portion of the third camera according to a pose of at least a portion of the fourth camera and the second spatial transformation, wherein the pose of at least a portion of the first camera and the pose of at least a portion of the third camera are defined in a second common coordinate system,

[0027] i) determining spatial coordinates of a 3D component of at least a portion of the first object in absolute spatial scale according to the first image, the third image, the pose of at least a portion of the first camera, and the pose of at least a portion of the third camera.

[0028] According to one embodiment, the pose of at least a portion of the second camera, the pose of at least a portion of the fourth camera, the pose of at least a portion of the first camera, and the pose of at least a portion of the third camera all include translation information.

[0029] According to one embodiment, step i) includes: determining the posture of the first camera and the posture of the third camera based on the first image and the third image, wherein the posture of the first camera and the posture of the third image are defined in a first common coordinate system; determining first translation information indicating at least one distance based on the translation information of the posture of the first camera and the translation information of the posture of the third camera in the first common coordinate system; determining second translation information indicating at least one distance based on the posture of at least a part of the first camera and the posture of at least a part of the third camera in a second common coordinate system; determining a scale factor based on the first translation information and the second translation information; determining the spatial coordinates of a 3D heavy component at an arbitrary spatial scale of at least a part of the first object in the first common coordinate system based on the first image, the third image, the posture of the first camera, and the posture of the third camera; and transforming the determined spatial coordinates of the 3D heavy component from the arbitrary spatial scale of the first common coordinate system to the absolute spatial scale of the second common coordinate system based on the scale factor.

[0030] According to one embodiment, step i) further includes: determining the posture of the first camera and the posture of the third camera based on the first image and the third image, wherein the posture of the first camera and the posture of the third camera are defined in a first common coordinate system; providing the posture of at least a part of the first camera and the posture of at least a part of the third camera defined in a second common coordinate system; updating the translation information related to the posture of the first camera and the translation information related to the posture of the third camera in the first common coordinate system based on the posture of at least a part of the first camera and the posture of at least a part of the third camera in the second common coordinate system; determining the spatial coordinates of the 3D heavy component of at least a part of the first object in absolute spatial proportion based on the first image, the third image, the updated posture of the first camera and the updated posture of the third camera.

[0031] For example, the posture of at least a portion of the second camera includes 3DoF translation information and 3DoF rotation information, the posture of at least a portion of the fourth camera includes 3DoF translation information and 3DoF rotation information, the posture of at least a portion of the first camera includes 3DoF translation information and 3DoF rotation information, and the posture of at least a portion of the third camera includes 3DoF translation information and 3DoF rotation information.

[0032] According to a second aspect, the present invention discloses a method for determining the spatial coordinates of a 3D re-construction of at least a portion of a first physical object at an absolute spatial scale, comprising the following steps:

[0033] a) receiving image information of a first image, the first image comprising at least a portion of a first physical object captured by a first camera,

[0034] b) receiving image information of a third image, the third image including at least a portion of the first object captured by a third camera,

[0035] c) receiving image information of a second image, the second image comprising at least a first portion of a second object captured by a second camera,

[0036] d) receiving image information of a fourth image, the fourth image including at least a second portion of the second object captured by a fourth camera,

[0037] wherein the viewing cone of the first camera and the viewing cone of the second camera do not overlap, and wherein the viewing cone of the third camera and the viewing cone of the fourth camera do not overlap,

[0038] e) providing first scale information and second scale information, the first scale information indicating an absolute spatial scale of at least a first portion of the second object, and the second scale information indicating an absolute spatial scale of at least a second portion of the second object,

[0039] f) determining a pose of at least a portion of the second camera and a pose of at least a portion of the fourth camera according to the second image, the first scale information, the fourth image and the second scale information, wherein the pose of at least a portion of the second camera and the pose of at least a portion of the fourth camera are defined in a second common coordinate system, wherein the pose of at least a portion of the second camera includes translation information, and the pose of at least a portion of the fourth camera includes translation information,

[0040] g) determining a position and posture of the first camera and a position and posture of the third camera according to the first image and the third image, wherein the position and posture of the first camera and the position and posture of the third camera are defined in a first common coordinate system,

[0041] i) the method further comprises

[0042] i0) determining second translation information indicating at least one distance according to a pose of at least a portion of the second camera and a pose of at least a portion of the fourth camera,

[0043] i1) determining first translation information indicating at least one distance according to the translation of the first camera pose and the translation of the third camera pose,

[0044] i2) determining a scaling factor according to the first translation information and the second translation information,

[0045] i3) determining the spatial coordinates of a 3D heavy component of at least a part of the first object in an arbitrary proportion according to the first image, the third image, the pose of the first camera and the pose of the third camera,

[0046] i4) transforming the determined spatial coordinates of the 3D heavy component from an arbitrary spatial scale to an absolute spatial scale according to a scale factor,

[0047] or

[0048] k) In place of steps i0)-i4), the method further comprises

[0049] k1) updating the translation information related to the pose of the first camera and the translation information related to the pose of the third camera to be in absolute space scale according to the pose of at least a part of the second camera and the pose of at least a part of the fourth camera,

[0050] k2) determining spatial coordinates of a 3D component of at least a portion of the first object in absolute spatial scale based on the first image, the third image, the updated position of the first camera, and the updated position of the third camera.

[0051] The following embodiments can also be implemented according to the first aspect or the second aspect.

[0052] According to one embodiment, the second common coordinate system is related to the second physical object, and step g) (first aspect) or step f) (second aspect) respectively include: determining the pose of at least a part of the second camera based on the second image and the first scale information; determining the pose of at least a part of the fourth camera based on the fourth image and the second scale information.

[0053] According to one embodiment, the second common coordinate system is associated with a selected camera, the selected camera is one of the second camera and the fourth camera, wherein the other of the second camera and the fourth camera is an unselected camera, and step g) (first aspect) or step f) (second aspect) respectively include: providing a pose associated with the selected camera in the second common coordinate system; determining a pose associated with the unselected camera based on the second image, the first scale information, the fourth image, and the second scale information.

[0054] According to one embodiment, the first common coordinate system is associated with the first object or one of the first camera and the third camera.

[0055] According to one embodiment, at least the first part of the second object and at least the second part of the second object are the same part, and the first proportion information and the second proportion information are the same, or at least the first part of the second object and at least the second part of the second object are different parts, and the first proportion information and the second proportion information are different.

[0056] According to one embodiment, the first camera and the second camera face opposite directions, and the third camera and the fourth camera face opposite directions.

[0057] According to one embodiment, the second physical object is a human face, and step g) (first aspect) or step f) (second aspect) is respectively implemented by a face tracking method, wherein the first scale information and / or the second scale information is provided as at least one distance in absolute scale between the positions of at least two facial features.

[0058] According to one embodiment, the second physical object is a human face, step g) (first aspect) or step f) (second aspect) is implemented by a face tracking method, respectively, and the first scale information and / or the second scale information is provided as at least one depth information in absolute scale of the position of at least a part of the second physical object in at least one of the second image and the fourth image.

[0059] According to one embodiment, at least one of the second camera and the fourth camera provides depth information at an absolute scale, and step g) (first aspect) or f) (second aspect) is implemented using visual odometry at an absolute scale, respectively, and the first scale information and / or the second scale information is provided by depth information.

[0060] According to one embodiment, the first camera and the third camera are the same camera at different points in time, and the second camera and the fourth camera are the same camera at different points in time.

[0061] According to one embodiment, the second object is a human face.

[0062] According to one embodiment, the first scale information and / or the second scale information comprises a pupillary distance.

[0063] According to one embodiment, the method further comprises providing intrinsic camera parameters of at least a portion of the second camera and intrinsic camera parameters of at least a portion of the fourth camera.

[0064] Preferably, the user carrying the first to fourth cameras is stationary.For example, the method further comprises the step of determining when the user is stationary.

[0065] According to one embodiment, the second camera and the fourth camera each comprise an infrared camera.

[0066] According to one embodiment, the scaling factor is determined based on a set of pose pairs N, where N>2.

[0067] According to one embodiment, the method further uses a generic face model for the second object.

[0068] According to one embodiment, the method further uses a probability distribution-based face model for the second object.

[0069] According to one embodiment, the method also uses a calibrated, in particular adapted, reconstructed or manually measured face model with respect to a second object.

[0070] According to one embodiment, the method further comprises face recognition or classification to select a model associated with the second object.

[0071] According to one embodiment, the method further comprises: detecting user input for starting the method and providing instructions to the user to perform a movement using the second camera, the movement being measured based on images captured using the second camera; receiving user interaction, the user interaction triggering determination of spatial coordinates of the 3D heavy component.

[0072] According to one embodiment, the second object is considered to be stationary relative to the first object when capturing the first image, the second image, the third image and the fourth image.

[0073] According to one embodiment, the first image, the second image, the third image and the fourth image are selected based on a method of determining that the second object is stationary relative to the first object while capturing the first image, the second image, the third image and the fourth image.

[0074] According to one embodiment, determining the second translation information indicating at least one distance based on the posture of at least a portion of the second camera and the posture of at least a portion of the fourth camera only considers the distance between the second object and the second camera and the distance between the second object and the fourth camera.

[0075] According to one embodiment, steps a) to i2) in the second aspect are repeatedly performed on multiple groups of first images, second images, third images and fourth images, wherein different image groups may or may not overlap with each other, thereby resulting in multiple scale factors in step i2), and the method also includes determining a single scale factor from the multiple scale factors and using the single scale factor to transform the spatial coordinates in step i4).

[0076] According to one embodiment, steps a) to i1) in the second aspect are repeatedly performed on multiple groups of first images, second images, third images and fourth images, wherein different image groups may or may not overlap with each other, thereby resulting in a variety of first translation information in step i1) and a variety of second translation information in step i0), wherein step i2) determines the scaling factor based on the multiple first translation information in step i1) and the multiple second translation information in step i0).

[0077] According to an embodiment, the first scale information and / or the second scale information is provided as at least one distance in comparison scale between positions of at least two facial features according to a generic face model, eg a face model based on a probability distribution.

[0078] According to one embodiment, the first scale information and / or the second scale information is provided as at least one distance in absolute scale between the positions of at least two facial features of a specific face based on a previous calibration of at least one distance in absolute scale between the positions of at least two facial features of the specific face, for example by adaptive fitting, reconstruction, manual measurement.

[0079] According to one embodiment, the first scale information and / or the second scale information is provided as at least one distance in absolute scale between positions of at least two facial features according to a model selected based on visual face recognition.

[0080] According to one embodiment, the first scale information and / or the second scale information is provided as at least one distance in absolute scale between positions of at least two facial features according to a model selected based on visual face classification for determining facial characteristics such as age, gender, race, weight, or height from a data set of generic face models for different face categories.

[0081] According to one embodiment, the 3D reconstruction of the first physical object at absolute scale is used to determine a camera pose at absolute scale. For example, the camera pose at absolute scale is used to overlay digital information in an augmented reality application.

[0082] According to one embodiment, the method is triggered by user input. According to another embodiment, the method is automatically triggered.

[0083] According to another aspect, the present invention discloses a system for determining spatial coordinates of at least a portion of a first object in an absolute spatial scale for 3D reconstruction, comprising a processing system for executing the steps listed in the aspects and embodiments disclosed above.

[0084] According to a preferred embodiment, when there is a capture device that can capture both a human face and a physical object, we thus use the face to determine the distance in absolute proportion, which can be used to achieve the purpose of the present invention. The capture device can be a single camera or a set of rigidly connected cameras, such as a single camera or a set of rigidly connected cameras in a commonly used mobile phone. For such a mobile phone with a front camera and a rear camera, the front camera can capture the user's face, while the rear camera captures the (first) physical object.

[0085] Generally speaking, the present invention relates to the problem of how to determine the spatial coordinates of a 3D reconstructed structure of a first physical object at an absolute scale, which problem can be solved by observing the first physical object and the second physical object with an imaging device from each of at least two viewing angles where the spatial coordinates of at least two points of the second physical object at an absolute scale are known, which enables the determination of the distance in absolute spatial scale between the at least two viewing angles, and the determination of the absolute spatial scale used for reconstructing the spatial coordinates of the first physical object.

[0086] In addition, the inventors have found that the user's face can be captured by the front camera while the object or environment is captured by the rear camera. It also includes: realizing that the absolute proportion of the face varies limitedly, thus providing good proportion constraints for measuring in absolute proportion. If the spatial characteristics of a specific user's face are unknown, a general face model can be used to determine the absolute spatial proportion. This introduces an error that depends on the degree of variation in the spatial characteristics of the face that form the estimate. Even if the presented spatial characteristics are different from the actual characteristics, which to some extent forms a more inaccurate absolute proportion estimate, multiple proportion estimates using the same characteristics and the same face will always form similar absolute proportions. As a result, the estimated proportion may be inaccurate but accurate, that is, consistent. If a calibrated (i.e., measured) specific face is used, the proportion of the face will be accurately grasped in absolute proportion.

[0087] Furthermore, it has been found that no additional known objects are required in the scene. By using the front and rear cameras of a modern handheld device, the face of the person providing the absolute scale-related information need not be part of the portion of the scene captured and reconstructed by the front camera, and can instead be captured while the user is operating the application. As a result, the user's face does not become part of the scene reconstruction, in contrast to a scheme where a marker or known object is added to the scene and then reconstructed as part of the scene.

[0088] The method described in the present invention can, for example, be used in conjunction with visual simultaneous localization and mapping (SLAM), such as the SLAM disclosed in reference [1], which is a well-known technique for creating a geometric model of a real environment using one or more cameras without any prior knowledge of the environment. Another common term for the same technique is structure from motion (SfM). A geometric model with at least depth information is also referred to as a 3D map of the real environment. The creation of a geometric model of the environment is also referred to as a (3D) reconstruction of the real environment. The created (usually referred to as reconstructed) geometric structure can be represented by multiple 3D features, such as point features or edge features. The 3D features describe the physical 3D features (also referred to as structure) of the real environment. The real environment can also be referred to as a real scene, a real object, or can be understood to include one or more real objects.

[0089] The reconstructed geometric model can be used in different applications. For example, it can be displayed virtually on a display screen or printed out using a 3D printer. It can also serve as a basis for measuring the spatial distance between multiple points on a physical object.

[0090] The reconstructed geometric model can also be used to determine the pose (i.e., position and / or orientation) of the camera relative to the object based on the current image of the object captured by the camera. By matching the extracted 2D features of the current camera image with the 3D features present in the geometric model, for example, through local image feature descriptors (reference

[20] ), multiple 2D-3D correspondences can be established. The camera position and orientation in the coordinate system of the geometric model can then be calculated based on these correspondences. This process is called camera pose estimation, sometimes also called camera tracking. The problem of tracking a camera relative to an object can also be stated as the problem of tracking an object relative to the camera. If one of the two problems is solved, the solution to the second problem is the inverse of the solution to the first problem. Therefore, when discussing the overall concept, camera tracking and object tracking can be used interchangeably.

[0091] Visual SLAM performs camera tracking and environment reconstruction simultaneously. Visual SLAM can facilitate many applications, such as visual navigation of robotic systems or vehicles. In particular, visual SLAM is a promising technology for supporting augmented reality (AR) systems or applications (see reference [3]) in unknown real environments.

[0092] The purpose of the present invention is to determine the absolute scale of the reconstructed geometric model of the real environment so that the coordinate system of the geometric model is in absolute scale, that is, there is a known scale factor that maps the coordinate system units of the reconstructed geometric model to absolute spatial units in reality. For example, the model can be scaled down to millimeters so that the model units of the model correspond to millimeters of the real object. In this case, if two points in the model are 56 units apart, the corresponding points on the real object are 56mm apart from each other. It can be implemented to define the model in absolute scale so that the units in the model coordinate system correspond to any real distance (for example, 12.34 meters) as long as the distance is known.

[0093] Thus, at least one scaling factor may be determined that may be used to scale a plurality of 3D features defined in a coordinate system associated with a geometric model describing a physical object.

[0094] In one embodiment, the present invention determines a scale factor that scales the coordinates of an existing physical model defined at an arbitrary scale to a coordinate system defined at an absolute scale.

[0095] In another embodiment, the invention determines the distance between at least two cameras in absolute scale, which makes it possible to reconstruct a 3D model of a physical object visible in the cameras in absolute scale from the images of the two cameras.

[0096] We can determine a scaling factor between a spatial coordinate system in which the spatial characteristics of a face are defined in terms of real-world units of measurement (e.g., centimeters). If a coordinate system defining the location of a feature is already given, the scaling factor can be used to scale the coordinate system (and the spatial coordinates of the feature, respectively) to have a one-to-one ratio with respect to real-world units of measurement (e.g., 1 unit = 1 mm). If a coordinate system defining the location of a feature has not yet been determined, we can use the scaling factor to directly initialize the coordinate system (and the spatial coordinates of the feature, respectively) to have a one-to-one ratio with respect to real-world units of measurement (e.g., 1 unit = 1 mm).

[0097] In addition, the coordinate system can be maintained. For operations that require absolute scale (i.e., representing real-world scale), the determined spatial scale factor can be used to extract real-world spatial distances or to transform the spatial coordinates of a pose or feature into a coordinate system with a one-to-one scale relative to real-world units of measure.

[0098] In another embodiment, two poses in absolute scale are provided for two images of a first camera, for example, a rear camera at two points in time, which is used for scene structure reconstruction by triangulation. Two poses in absolute scale can be provided by face tracking based on images captured by a different camera facing the user, hereinafter referred to as the front camera, with a known spatial transformation relative to the first camera.

[0099] In many possible scenarios, a user holds a mobile device equipped with a rear camera. The front camera can be used to capture images of the surrounding environment, for example, to overlay the surrounding environment images with virtual objects in augmented reality applications or for other image processing applications. In such applications, it is often required to reconstruct the real objects in the surrounding environment and / or estimate the camera's pose or the camera's motion relative to the real objects or the environment.

[0100] In such as Figure 6 In the scenario shown where we have an additional front-facing (also called user-facing) camera, we can use an image of the face of a user, who is operating the application and is therefore already present in the real environment, to estimate the absolute scale of the map (i.e., the 3D reconstruction) created from the images captured by the world-facing camera.

[0101] This has several advantages: the user's face is available, so no additional geometry or objects need to be added, and the user's face is captured by a camera facing the user, so no cumbersome equipment is required. Since the user is usually looking at the screen to experience the application, the camera facing the user can always capture the user's face, while the camera facing the world can capture a view of the real environment. Since the user's face is always available as long as the user is facing or looking at the display of the handheld device, automatic updating or re-estimation of scale can be supported. The geometry of the face is also limited to the range of geometric variations, thereby effectively representing and constraining the size and scale of facial features of most people. That is, with only the user's face and the capture device, the scale estimation can be done by anyone, without the need for additional known objects. Specific users can also have their faces specially calibrated to achieve higher accuracy. It is also possible to incorporate a face recognition program that can recognize a large number of people, for example, the face recognition program in reference

[19] , which is used to recognize users appearing in the image of the camera facing the user, and then select the corresponding correct absolute size of the user's face from a preset database. Accessible via a network connection, the face recognition program can be run locally or remotely. Accessible via a network connection, a preset database containing the correct absolute dimensions of a user's face can be provided locally or remotely.

[0102] Another embodiment uses a virtual face classification method, for example, as disclosed in reference

[12] , to determine characteristics such as a person's age, gender, race, weight, height, etc., and then uses a generic face model for a specific human class.

[0103] Two cameras (e.g., a world-facing camera and a user-facing camera) may be used in conjunction with the following assumptions: the spatial relationship between the coordinate systems of the two cameras is known, e.g., a rigid body transformation. The world-facing camera may be used to determine: the pose of the world-facing camera relative to the real environment in a coordinate system associated with the real environment; and / or an object coordinate system associated with a physical object located in the real environment. This will enable the desired calibration between virtual objects, which may be superimposed on the camera image, and physical objects, which are physical objects in the real environment in the image captured by the world-facing camera. Assuming that the transformation between the coordinate systems of the two cameras is known, the absolute scale information of the user's face captured by the user-facing camera may be transformed into the real environment coordinate system and / or the object coordinate system. This will enable SLAM reconstruction at absolute scale by the world-facing camera.

[0104] For each view of a camera setup including a user-facing camera and a rigidly connected world-facing camera, we can determine a pair of two poses: a pose of the user-facing camera at absolute spatial scale relative to the user's face, and a pose of the world-facing camera at arbitrary scale relative to a first object. Given a spatial transformation between the user-facing camera and the world-facing camera, we can determine the pose of the world-facing camera at absolute spatial scale relative to the user's face by transforming the pose of the user-facing camera at absolute scale relative to the user's face by the spatial transformation between the user-facing camera F and the world-facing camera.

[0105] Given two such transformed poses obtained from two different viewpoints of a dual-camera setup, we can determine the translation distance D_abs between the two poses of the world-facing camera at absolute scale. Given two poses of the world-facing camera relative to a first object defined at an arbitrary scale, we can determine the translation distance D_arb between the two poses at an arbitrary scale.

[0106] Finally, the scale factor from the arbitrary scale of the coordinate system relative to the real object to the absolute scale can be determined as the ratio of D_abs to D_arb: S = D_abs / D_arb.

[0107] Without referring to the attached drawings, the absolute spatial distance between two camera poses PW1 and PW2 of a first camera WC (capturing images of at least a portion of a real object for SLAM reconstruction) is determined in the following manner: by performing image-based camera pose estimation relative to the face of at least two images, observing the spatial translation and rotation of a second camera FC (capturing images of at least a portion of a face) belonging to the same step device C, wherein at least one facial image I(F1) is captured at camera pose PF1 of camera FC, i.e., camera WC is at camera pose PW1 at this point in time, and another facial image I(F2) is captured at camera pose PF2 of camera FC, i.e., camera WC is at camera pose PW2 at this point in time. At least a portion of the real object can be captured by camera WC in images I(W1) and I(W2) at camera poses PW1 and PW2, respectively. Images I(W1) and I(W2) can be used in real object reconstruction or camera pose estimation applications (e.g., SLAM). For the method of determining the spatial characteristics of a real object in absolute proportion, the absolute spatial distance between the two camera poses PW1 and PW2 should not be zero.

[0108] In one embodiment, the scale estimation is performed not only based on two pairs of corresponding poses PW and PF (i.e., PW1 and PF1, and PW2 and PF2), which are determined by corresponding four images captured at these poses (essentially as Figure 2a and Figure 2b As shown), it is also performed based on multiple pairs of poses PW and PF, each pair of poses being determined by the captured images. Multiple scale estimates, each based on two pairs of poses (W_i and F_i, and W_j and F_j), can be merged by (for example) a model fitting method, such as a median, an average, or RANSAC. The model fitting method may additionally consider whether certain pairs of poses are suitable for scale estimation, for example based on the minimum distance between the poses or the uncertainty score and quality score of the pose measurements. The consistency between the difference from F_i to F_j when the camera facing the user is rotated and the difference from W_i to W_j when the camera facing the world is rotated can also be used as (for example) an uncertainty score and / or a quality score of the pose measurements and as an indicator of whether the second object has moved relative to the first object. The rotation part between the two poses of the camera facing the user can also be used to determine whether these poses are suitable for scale estimation. Ignoring the true transformation between the camera facing the user and the camera facing the world and assuming that their origins are the same, the rotation part between the two poses of the camera facing the user can introduce scale estimation errors (see Figure 8a , 8b and 8c), and therefore preferably only has negligible / small rotations. The two complete pose trajectories (one for the user-facing camera and one for the world-facing camera) can also be used to estimate the probability that the face did not move during capture. This can be estimated (for example) by the following method as disclosed by Umeyama (reference

[10] ): calibrate the two trajectories and calculate the residual error after alignment. If the residual error is above a certain threshold, this may indicate that the head has moved relative to the object. In this case, the determined scale factor can be discarded and the calibration can be restarted. The consistency between the rotation rate of the user-facing camera and the rotation rate of the world-facing camera can also be used to support the correction of the two trajectories, and the consistency of the rotation rates can be taken into account when calculating the residual error.

[0109] Modern handheld mobile devices, such as mobile phones, tablet computers, or tablet computers may be equipped with two cameras (e.g., a camera facing the user and a camera facing the world), pointing in two opposite directions. The display screen of the mobile device usually faces the same direction as the camera facing the user.

[0110] One possible implementation of the present invention is to estimate the absolute scale by an image of the user's face captured by a camera facing the user. The absolute scale is then used for reconstruction and physical tracking at absolute scale by another camera (a world-facing camera pointing in the opposite direction to the camera facing the user and typically with a known spatial transformation relative to the camera facing the user). The two cameras can be attached to a handheld or mobile device, such as a mobile phone, tablet computer, or tablet computer. In addition, a display device, such as an LCD screen, can be attached to the mobile device.

[0111] The two cameras of the mobile device may have a fixed spatial relationship, eg, a rigid body transformation, which may be determined by a calibration procedure, eg, hand-eye calibration, by using at least one known marker or another tracking system.

[0112] Commonly used advanced solutions require special cameras (with depth sensors based on active stereo or passive stereo or time of flight) or other devices in order to estimate the absolute scale of the real object. This definitely limits the applicability of these solutions.

[0113] One approach to estimating the absolute scale of a real-world SLAM map is to detect known objects directly in the camera image used as input to the SLAM method (as described in reference [1]); and to infer the absolute scale of the map using the known absolute scale of the known objects. A problem with this approach is that it requires the acquisition of known objects and an additional setup step in which additional known objects are added to the scene. This changes the original scene and requires the camera to be oriented toward the known objects.

[0114] Compared to the state-of-the-art techniques that use special objects like planar markers that must be placed prominently in the room and captured by the SLAM camera for scale estimation (like the one used in [1]), the user's face has a significant advantage: no special care is needed to keep the object in the field of view of the camera facing the user during the entire reconstruction process.

[0115] Another significant difference of the present invention compared to the solution of adding objects with known absolute spatial properties to the scene to be reconstructed (reference [1]) is that the present invention does not rely on capturing the known object with the same camera used to reconstruct the real object or scene, but instead uses a second camera to capture the face. Thus, the face does not become part of the reconstructed structure, which is contrary to adding a real object to the scene.

[0116] In contrast to solutions such as reference [1] that add known objects to the scene and thus require a computer equipped with a camera, a user, the physical object to be reconstructed, and additional special known objects that the user will need to carry with him for calibration, the present invention only requires a computer equipped with a camera, a user, and the physical object to be reconstructed.

[0117] According to various embodiments of the present invention, a significant advantage of determining the absolute scale of the SLAM heavy component is that the absolute distance between two or more facial features or reference points is explicitly used (e.g., the distance between the two eyes of the face, or the distance between the eyes and the mouth on the face, or the distance between the left and right corners of the eye). When the user observes the display device, these distances can be identified in the user image captured by a camera facing the user (i.e., a camera pointing to the user, or a camera pointing to the user image reflected by one or more optical instruments (e.g., mirrors or optical lenses) for capturing the user's face). This enables the application of prior knowledge about the human face and its absolute spatial characteristics. Facial images are usually captured by a camera facing the user and are used to estimate the absolute scale, rather than relying on other objects in the scene whose geometry is known for scale estimation. By explicitly using a face that is usually or always available during the entire period when the user observes the display screen (e.g., using face-specific features), scale estimation can be performed at any time without any impact on the scene. In addition, the range of variation in the geometry of the face between all people is limited, and by focusing on the face, specialized algorithms can be used to estimate the scale through the user's face. Face regions that are particularly suitable for scale estimation may be pre-learned and / or pre-defined. Face regions may be registered during real-time tracking by an established u-pose tracking algorithm. Face regions that may adversely affect scale estimation, e.g., because these regions may differ significantly in size and shape from person to person, may be considered and excluded from scale estimation (like nose size or ear size).

[0118] In one embodiment, the normal of the display device of the mobile device and the optical axis of the camera facing the user preferably have the same direction. In this case, when the user observes the visualization information (e.g., the augmented scene) on the display device, the camera facing the user can usually or always capture the user's face. Therefore, the absolute scale can always be estimated based on the face image.

[0119] For example, a processing system according to the present invention is at least partially included in a mobile device (such as a mobile phone, a wearable computer, a tablet computer, a mobile computer, commonly referred to as a laptop, or a head-mounted display, such as for optical see-through augmented reality applications) and / or a server computer adapted to communicate with the mobile device. The processing system may be included in only one of these devices, for example, in the mobile device or in the server computer, or it may be a distributed system in which one or more processing tasks are distributed and processed by one or more processing devices in the processing system, which are dispersed and communicate through point-to-point communication or through a network.

[0120] According to one embodiment, a system includes a mobile device including one or more cameras and, for example, a display screen.

[0121] Any steps, embodiments, aspects and examples described herein in connection with the method may be implemented similarly or similarly by a processing system (software and / or hardware) for performing the corresponding steps, embodiments, aspects or examples. Any processing device used within the processing system may be configured in this manner by software and / or hardware and may communicate with one or more cameras, displays and / or any other components via a communication network, for example, via a server computer or point-to-point communication.

[0122] According to another aspect, the present invention also relates to a computer program product comprising a software code segment, which is adapted to perform the method according to the present invention. Specifically, the software code segment is contained on a non-transient computer readable medium. The software code segment can be loaded into the memory of one or more processing devices (such as microprocessors) described herein. Any processing device (such as one or more microprocessors) used can communicate via a communication network, for example, via a server computer or a point-to-point communication method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] Aspects and embodiments of the present invention will now be described with respect to the accompanying drawings, in which:

[0124] Figure 1 A flow chart showing a method according to an embodiment of the present invention is shown,

[0125] Figure 2a and Figure 2b shows a possible implementation of the invention and the problem of scale ambiguity when using monocular SLAM or SfM,

[0126] Figure 3 Another embodiment of the present invention is shown.

[0127] Figure 4 The involved coordinate systems and transformations according to an embodiment of the present invention are shown.

[0128] Figure 5 shows a capture device according to an embodiment of the present invention, the capture device comprising a user-facing camera and a world-facing rigidly attached camera in different poses,

[0129] Figure 6 An exemplary embodiment of the present invention implemented with a handheld device is shown.

[0130] Figure 7 An embodiment of a graphical user interface that guides a user through an exemplary scale estimation process is shown.

[0131] Figure 8a ,8b and 8c show the effect of the spatial transformation between the first camera and the second camera,

[0132] Fig. 9 Another embodiment of the present invention is shown.

[0133] Fig.10 Another embodiment of the present invention is shown.

[0134] Fig.11 Another embodiment of the present invention is shown.

[0135] Fig.12 Another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0136] The structure of the object is determined based on a set of images captured by a monocular capture device, which forms a reconstructed spatial (or geometric) structure based on the scale. This means that the reconstruction uses spatial units for which the absolute spatial units (such as the unit meter) are unknown. In many applications, it is necessary to obtain a reconstructed structure with absolute units, also known as a reconstructed structure with "absolute scale". For this purpose, knowledge of at least one distance in absolute scale can be used: the distance between multiple parts of the object when the corresponding image is taken for reconstruction, or the distance between the position of the camera relative to the object. This distance in absolute scale can be (for example) the eye distance of a specific person or the universal eye distance or any other spatial characteristic of the facial reference point. When there is a capture device that captures the face and the object, the embodiments disclosed herein use the face to determine the distance in absolute scale. The capture device can be a single camera or a group of rigidly connected cameras, for example, a single camera or a group of rigidly connected cameras in a mobile phone. The front camera usually captures the user's face, while the rear camera captures the object.

[0137] Advantageously, the present invention utilizes the user's face (user faces are not uniform but most people have similar characteristics), which can be preferably used in handheld augmented reality programs.

[0138] The present invention enables reconstruction of the structure of a physical object or environment at an absolute spatial scale (hereinafter also referred to as absolute scale). This (for example) enables camera pose tracking at absolute scale, which enables superposition of virtual objects, which are defined at absolute scale to maintain a consistent scale with the reconstructed physical object. Camera pose tracking also enables real space measurement, thereby enabling calculation processes based on absolute spatial scale, such as physical simulation (e.g., gravity acceleration) or collision detection between virtual objects and real objects (e.g., whether an object is spatially consistent with reality).

[0139] Instead of using known objects (i.e., markers) that need to be added to the scene, embodiments of the present invention use the user's face that is always present. Contrary to other solutions, the present invention does not require any user input, the present invention does not require inertial sensors, and provides more accurate results than consumer-grade inertial sensors.

[0140] Figure 1 A flow chart of a method according to an embodiment of the present invention is shown. In a first step S1, there are provided: a first image including at least a portion of a first physical object captured by a first camera, a second image including at least a portion of a second physical object captured by a second camera, a third image including at least a portion of the first physical object captured by a third camera, and a fourth image including the second physical object captured by a fourth camera. In a second step S2, there are provided: a spatial transformation in absolute proportion between the first camera and the second camera, and a spatial transformation in absolute proportion between the third camera and the fourth camera. In a third step S3, an absolute spatial proportion of at least a portion of the second physical object is provided. In a fourth step S4, at least a portion of the posture of the second camera in absolute proportion is determined based on the absolute proportion information of the second image and at least a portion of the second physical object, and at least a portion of the posture of the fourth camera in absolute proportion is determined based on the absolute proportion information of the fourth image and at least a portion of the second physical object.

[0141] In another embodiment, it is also possible not to determine two complete poses related to the second physical object (e.g., a human face), i.e., the second camera pose for capturing the second image and the fourth camera pose for capturing the fourth image, but only to determine the difference between the two poses, in other words, only determine the pose of the fourth camera relative to the second camera.

[0142] In another embodiment, it is also possible to determine only the translation distance between two poses, ie to determine only the distance between the fourth camera pose and the second camera pose.

[0143] In the next step S5, the posture of at least a part of the first camera in absolute proportion is determined based on the posture of the second camera and the spatial transformation between the first camera and the second camera, and the posture of at least a part of the third camera in absolute proportion is determined based on the posture of the fourth camera and the spatial transformation between the third camera and the fourth camera.

[0144] In another embodiment, it is also possible not to determine two complete poses related to the first physical object, namely the first camera pose for capturing the first image and the third camera pose for capturing the third image, but only to determine the difference between the two poses, in other words, only determine the pose of the third camera relative to the first camera.

[0145] In another embodiment, it is also possible to determine only the translation distance between two poses, ie to determine only the distance between the third camera pose and the first camera pose.

[0146] This distance can then be used together with the estimated distance between the second camera and the fourth camera to determine the absolute scale of the first camera (e.g., the so-called world-facing camera or rear-facing camera). For example, the scale factor can define the true size of the reconstructed geometric model of the real environment, or be used to map the coordinates of a reconstructed component at an arbitrary scale to an absolute scale.

[0147] In step S6, the spatial coordinates of the 3D re-construction (also referred to as the geometric model) of the first physical object in absolute scale may be determined by the pose of at least a portion of the first camera, the pose of at least a portion of the third camera, the first image and the third image, or by the difference between the two poses, or by the distance between the two poses.

[0148] In another embodiment, an absolute scale factor of the first camera (eg, the world-facing camera) is calculated based on the distance between the first camera and the third camera (in the first common coordinate system) and the distance between the second camera and the fourth camera (in the second common coordinate system).

[0149] Figure 2a and Figure 2b A possible implementation of the present invention is shown together with the problem of scale ambiguity when using monocular SLAM or SfM. Figure 2a A top-down scene diagram is shown, which includes a large vehicle O1 (eg, a real car for driving), four cameras W1, F1, W2, and F2 (corresponding to a first camera, a second camera, a third camera, and a fourth camera, respectively), and a user U1. Figure 2a Also shown in four small figures are images I(W1), I(F1), I(W2), and I(F2) taken by four cameras W1, F1, W2, and F2, respectively. Figure 2b A top-down scene diagram is shown, which includes a small car O2 (eg, a child-type toy car), four cameras W3, F3, W4, and F4 (corresponding to a first camera, a second camera, a third camera, and a fourth camera, respectively), and a user U1. Figure 2b Images I(W3), I(F3), I(W4), and I(F4) taken by four cameras W3, F3, W4, and F4, respectively, are also shown in four small figures.

[0150] Assume that, despite the obvious difference in size between the large car O1 and the small car O2, the images I(W1) and I(W3) and the images I(W2) and I(W4) are substantially identical. This is because the poses of cameras W3 and W4 are scaled relative to W1 and W2 in the same way that O2 is scaled relative to O1. This illustrates the problem of scale ambiguity. It is impossible to determine the absolute size of the car based on just one or more images, and therefore, it is impossible to distinguish between the large car O1 and the small car O2 based on images I(W1), I(W2), I(W3) or I(W4). Therefore, if the size of the car is unknown, it is also impossible to determine the distance between the camera positions of W1 and W2 or W3 and W4 in absolute scale based on the images. However, if the absolute distance between the camera positions of W1 and W2 is known, for example, 1 meter, it is also impossible to determine the absolute size of the car. Determining the absolute size of real objects will be beneficial in many applications. Therefore, according to the present invention, it is advantageous to determine the absolute size of real objects and reconstruct them in absolute scale.

[0151] In addition to the car, user U1 (i.e., person) is also located Figure 2a and Figure 2b In the scene shown. According to an embodiment of the present invention, we have captured images I(F1), I(F2), I(F3), and I(F4) of user U1 including his face. In one embodiment, the spatial transformations between the camera poses of F1 and W1, F2 and W2, F3 and W3, and between F4 and W4 are known and may be the same if the images are captured by two rigidly connected cameras, that is, W1, W2, W3, and W4 are the same physical camera at different points in time, and F1, F2, F3, and F4 are the same physical camera at different points in time. In another embodiment, W1, W2, W3, and W4 are different cameras, and F1, F2, F3, and F4 are different cameras.

[0152] Since user U1 is Figure 2a and Figure 2b, the sizes in are not changed, and the corresponding images I(F1) and I(F2) are different from images I(F3) and I(F4), respectively. The difference in face position between I(F1) and I(F2) is greater than that between I(F3) and I(F4), which indicates that the movement between cameras W1 and W2 is greater than that between W3 and W4. Now assume that at least one spatial characteristic of at least one facial reference point on the user's face is given in absolute proportion, for example, the interpupillary distance is known to be 63 mm (the eye distance of a universal face model is known to be 63 mm). Assume that the reference point (e.g., the eye center position) can be automatically located in images I(F1) and I(F2) by means of a face or eye detection algorithm. Other facial points may also be used, such as one or more of the following: position (corner position, center position, or border area), size, shape, contour, area, proportion, ratio, distance between left and right eyes, and appearance of left and right eyes (pupil, iris, cornea, sclera, inner canthus, outer canthus, center, upper and lower eyelids, eyelashes, ...), nose bridge, nose (nose tip, nose dorsum, nose wing, nostril columella, ...), philtrum, lips, left and right ears, left and right eyebrows, teeth, left and right cheeks, jaw, neck, Adam's apple. Then, the distance between the position of camera F1 and the position of camera F2 can be determined in absolute proportion based on the positions of the facial reference points in images I(F1) and I(F2), because these reference points have known spatial characteristics in absolute proportion. In the case where the spatial transformation between cameras W1 and F1 and the spatial transformation between W2 and F2 are known, the distance between the position of camera W1 and the position of camera W2 can also be calculated in absolute proportion. The same applies to W3, F3, W4 and F4.

[0153] The camera images I(W1) and I(W2) together with the known distance between the cameras W1 and W2 that took these images ultimately enable the reconstruction of the object O1 in absolute scale, or the determination of a scale factor. An existing reconstruction of O1 at any scale is mapped to absolute scale, i.e., to a coordinate system that has a known relationship to real-world spatial units of measurement, such as meters or inches.

[0154] As a result, the heavy components of O1 are different in proportion from the heavy components of O2, which enables differentiation between O1 and O2.

[0155] Figure 3Another embodiment of the present invention is shown. A top-down scene diagram at different time points is shown, the upper part of the figure shows the scene at the first time point, and the lower part of the figure shows the scene at the second time point. The small figure in the left part of the figure shows the images captured by the corresponding cameras. Camera W5 is used to capture an image I(W5) of the car O1, while camera F5 with a known spatial transformation relative to camera W5 captures an image I(F5) of the user U1 and his face. The two cameras are then moved away from the car towards the user, thereby forming camera poses F6 and W6 and corresponding images I(F6) and I(W6). Similarly, it is impossible to determine the absolute size of the car, that is, the size of the car in reality, given only the camera images I(W5) and I(W6). We assume that the intrinsic parameters (especially the focal length) of cameras F5, W5, F6, W6 are known. We further assume that in a specific face model or a general face model for eye detection or face detection, the pupil distance P of user U1 with absolute units (e.g., centimeters) is known. Furthermore, the distance between the pupils of the user (P5 and P6) can be determined in the images (F5) and I(F6) in pixel coordinates. In the special case where the imaging plane of camera F5 is at the same distance to both pupils (left and right), the distance P5 together with the intrinsic camera parameters of F5 is sufficient to calculate the distance D5 between the point between the pupils and the center of the F5 camera in absolute scale, i.e., a distance with absolute real units, such as meters, because D5 = (f*P) / P5, where f is the focal length of camera F5 in pixels (this formula is used if the face is centered on the camera's principal point). Similarly, when the imaging plane of camera F6 is at the same distance to both pupils (left and right), the distance D6 between user U1 and camera F6 can be determined based on the focal length of F6, the pupil distance P6 in image I(F6), and the pupil distance P of user U1 in absolute scale. If the user U1 has not moved relative to the car O1 in this scene, the difference between the distances D5 and D6 can be used to determine the distance between the position of the camera W5 and the position of the camera W6 in absolute scale. This enables the determination of the absolute scale of the car O1 taking into account the camera images I(W5) and I(W6).

[0156] Figure 4An embodiment of the coordinate systems and transformation processes involved is shown. An object coordinate system O is defined in absolute units with a known scale factor (such as meters) relative to the actual distance. A coordinate system W is associated with the world-facing camera (such as W1, W2, etc.) and a coordinate system F is associated with the user-facing camera (such as F1, F2, etc.). When SfM or SLAM is performed on the images of the world-facing camera, the camera's coordinate system W is defined in arbitrary scale, as described above. It is assumed that the transformation between W and F (i.e., the rotation R and translation T (in absolute scale)) is known. The transformation can be static and calibrated offline (i.e., before being used in the scene), for example, if the two cameras are the user-facing camera and the world-facing camera of a mobile phone or tablet computer, the coordinate system F is defined in absolute scale because its pose is estimated by facial features in the user's face coordinate system U, which is defined in absolute scale, and the facial features are also defined in absolute scale. One of the ideas of the present invention is to now exploit the fact that F can be defined in absolute scale and the rigid body transformation between F and W is known. As a result, in this embodiment, W can also be defined in absolute scale, which is not possible without the coordinate systems F and U, ie without a camera facing the user and the user's face.

[0157] Figure 5 An embodiment of a capture device is shown, which includes a user-facing camera and a rigidly attached world-facing camera in different poses. The world-facing camera and the user-facing camera can be at least two different cameras or a single camera, for example, an omnidirectional camera with a 360° capture field of view. In this case, the camera images of the respective cameras can be defined as two different parts of the overall image. For a single camera, the poses F and W can be the same.

[0158] The cameras used at poses F8 and W8 may be the same cameras used at poses FF7 and W7 at different points in time, or they may be different cameras used simultaneously or at different points in time. The image from the camera facing the user enables determination of spatial distance information (a scalar value) in absolute scale between camera positions F7 and F8, and inferred spatial distance d (also a scalar value) in absolute scale between the position of camera W7 and the position of camera W8, because the transformation between F7 and W7 and the transformation between F8 and W8 are known.

[0159] Figure 6An exemplary embodiment of the present invention is shown as being implemented with a handheld device H9, such as a mobile phone, smart phone, phablet, or tablet computer, including a user-facing camera F9 that can capture the user's face U9 and a world-facing camera W9 that can capture physical objects O9. In addition, the handheld device H9 can have a processing device P9 and a wireless network unit. Any steps of any embodiment disclosed herein can be performed locally in the handheld device H9 by the processing device P9, or sent to a remote server computer S9 or another mobile device via the wireless network unit. The user-facing camera, the world-facing camera, and the display screen of the handheld device can have a known spatial relationship.

[0160] According to one embodiment, the processing system described herein may be at least partially included in the handheld device H9 and / or in a server computer S9 adapted to communicate (e.g., wirelessly) with the handheld device H9. The processing system may be included in only one of these devices, for example, in the handheld device H9 or in the server computer S9, or may be a distributed system in which one or more processing tasks (performing one or more method steps) are issued and processed by one or more processing devices (such as microprocessors), which are spatially dispersed and communicate with each other.

[0161] With such a system arrangement, the user U9 can reconstruct the real object O9, in this embodiment a chair in absolute scale, by moving the handheld device H9 to at least two different viewing angles, while the head U9 does not move relative to the real object O9. This allows the determination of the pose of U9 in absolute scale when the O9 heavy component is used as a reference for the camera pose estimation. This also allows the superposition of a virtual object (e.g. a virtual pillow) on the camera image I91 of the world-facing camera W9 in absolute scale (see image I92) rather than in an erroneous arbitrary scale (see image I93). This makes the augmentation more realistic, with a consistent scale in image I92, while in image I93 the virtual pillow is too small relative to the real chair.

[0162] Figure 7An example of a graphical user interface for guiding a user to perform a scale estimation according to an embodiment of the present invention is shown. In this embodiment, the method is performed in a mobile phone D2, which includes a display device D1 (a touch screen in this case), a camera F10 facing the user, and a world-facing camera on the back of the phone that is not visible in the accompanying drawings. In this embodiment, the world-facing camera repeatedly captures images of a physical object (car) placed behind the mobile phone D2. These images are displayed on the display device D1. In an initial state S11, the user can start the scale estimation process by touching a button G1 on the display screen. Thereafter, the camera F10 facing the user is used to determine the position of the camera facing the user in absolute proportion relative to the user's face. In addition, thereafter, the world-facing camera is used to determine the position of the camera facing the world in arbitrary proportion relative to the physical object (car). Then, in stage S12, the button appearance changes, prompting it to be deactivated (see the empty button G2). Additional graphical user interface elements G3 and G4 appear, providing instructions for camera movement to perform the process and visualize the progress of the process. In addition, a label G5 can be displayed on the display device to provide text instructions or information. After the user moves the phone (and thus the world-facing camera) to state S13, the progress indicator G6 adjusts accordingly and shows that progress has been made. Once the camera movement is large enough to make a scale estimate (S14), the progress indicator G7 shows that the goal has been achieved. In addition, a label G8 can notify the user of the success or failure of the calibration. At this point, the user-facing camera and the world-facing camera can stop capturing and / or tracking. In addition, the application can be in the initial state S11 again, except that the absolute scale of the real object (car) can now be estimated, and the world-facing camera can determine the pose of the world-facing camera relative to the real object (car) in absolute scale. By touching button G1 again, the user can trigger another scale estimation process.

[0163] Figure 8a , 8b 8c show the effect of the spatial transformation between the first camera and the second camera. The first camera and the third camera can be implemented by the same physical camera (e.g., a world-facing camera, such as a world-facing camera of a mobile phone), and the second camera and the fourth camera can be implemented by the same physical camera (e.g., a user-facing camera, such as a user-facing camera of a mobile phone). It is found that the two cameras (e.g., the world-facing camera and the user-facing camera) together can form a dual camera, where the camera frustums between the two cameras do not overlap.

[0164] Figure 8a , 8b8c show three different arrangements of such dual cameras, which differ in the length of the spatial translation between the first camera and the second camera (and thus between the third camera and the fourth camera). Each sub-image contains a top-down scene image, the scene includes four cameras, wherein each sub-image shows the dual camera in two poses, wherein the first camera and the third camera have the same pose in all sub-images. The origins of the cameras are shown as OR1, OR2, OR3 and OR4. In each top-down view, the absolute spatial distance between the camera origins OR1 and OR3 (above) and the absolute spatial distance between the camera origins OR2 and OR4 (below) are marked. The absolute spatial distance between the camera origins is noteworthy because it contains absolute scale information.

[0165] The rotational movement between the first camera (index "1") and the third camera (index "3") causes a translational movement between the second camera (index "2") and the fourth camera (index "4"), which depends on the length of the spatial translation between the first camera and the second camera and causes the following difference: the difference between the absolute spatial distance length between OR1 and OR3 and the absolute spatial distance length between OR2 and OR4.

[0166] exist Figure 8a , where the spatial translation length between the first camera and the second camera is zero, the absolute spatial distance between the camera origins OR1 and OR3 (above) and the absolute spatial distance between the camera origins OR2 and OR4 (below) are the same.

[0167] exist Figure 8b , where the length of the spatial translation between the first camera and the second camera is quite small compared to the length of the translation movement between the second camera and the fourth camera, the absolute spatial distance between the camera origins OR1 and OR3 (above) and the absolute spatial distance between the camera origins OR2 and OR4 (below) are similar but not exactly the same, and can be considered equal when very small errors can be introduced.

[0168] exist Figure 8c , where the spatial translation length between the first camera and the second camera is not small compared to the translation length between the second camera and the fourth camera, the absolute spatial distance between the camera origins OR1 and OR3 (above) and the absolute spatial distance between the camera origins OR2 and OR4 (below) are quite different and cannot be considered equal without introducing large errors.

[0169] According to the first spatial transformation between the first camera and the second camera (see step e) above), the pose (P1F) of at least a portion of the first camera relative to the pose P2F is determined (see step h)). Similarly, according to the second spatial transformation between the third camera and the fourth camera (see step e)), the pose (P3F) of at least a portion of the third camera relative to the pose P4F is determined (see step h)).

[0170] Based on the pose of the second camera, the spatial transformation T1 between the first camera and the second camera, the pose of the fourth camera, and the spatial transformation T2 between the third camera and the fourth camera, the absolute spatial distance between the pose of the first camera and the pose of the third camera is determined, where T1 and T2 may be the same. This transformation may be a 6D rigid body transformation, which may include a calibration process of the relative spatial position and orientation between the first camera and the second camera in an absolute scale in the coordinate system. This can be determined by a method such as that disclosed in reference

[11] . In this case, the translational motion between camera 2 and camera 4 caused by the rotational motion between camera 1 and camera 3 is taken into account, and the calculation process of the absolute spatial distance between the pose of the first camera (camera 1) and the pose of the third camera (camera 3) is accurate.

[0171] According to another embodiment (see the second aspect of the invention described above), the method regards the spatial distance between camera F (second camera) and camera W (first camera) as zero. Thus, the translational movement between the pose of the third camera and the pose of the first camera caused by the movement between the pose of the second camera and the pose of the fourth camera is ignored. For the translational part of the pose calculated by the first camera and the pose calculated by the third camera, the error caused is less than or equal to the actual distance between the first camera and the second camera. For the absolute spatial distance between OR2 and OR4, the error caused is less than or equal to twice the actual distance between the first camera and the second camera. The error caused also depends on the rotation amount between the pose of the second camera and the pose of the fourth camera. A 180° rotation causes a higher error. Only a basic translational movement occurs between the pose of the second camera and the pose of the fourth camera, that is, there is only a negligible rotation part, which will only cause a negligible error.

[0172] This means that for a camera arrangement where the distance between the first camera and the second camera is small compared to the translational motion between the second camera and the fourth camera, the translational part of the spatial transformation between the first camera and the second camera can be considered as the identity transformation. This also means that for a camera motion having only a rotational part (for the first camera and the second camera) that is negligible compared to the translational motion between the second camera and the fourth camera, the translational part of the spatial transformation between the first camera and the second camera can be considered as the identity transformation.

[0173] Fig. 9 Another embodiment of the invention is shown, in which the pose of at least a portion of a first camera (the world-facing camera) is used as input to a 3D reconstruction method, such that the method can reconstruct an object O9 in absolute scale. The poses P2F and P4F of two (e.g.) user-facing cameras C2 (second camera) and C4 (fourth camera) relative to the coordinate system of the user's face U can be determined in absolute scale by a face tracking algorithm. Given a spatial transformation T1 between cameras C1 (first camera) and C2 (second camera) and a spatial transformation T2 between cameras C3 (third camera) and C4 (fourth camera), the pose P1F of (e.g.) world-facing camera C1 and the pose P3F of (e.g.) user-facing camera C3 can be determined in absolute scale in the coordinate system of the user's face U by a cascade method. This embodiment then utilizes the poses of the two world-facing cameras C1 and C3 determined in absolute scale in a common coordinate system to create a 3D reconstructed object O9 in absolute scale by establishing a correspondence between the camera images of cameras C1 and C3 and triangulating their depths. In this embodiment, the 3D reconstruction method does not estimate the camera pose, but uses the provided camera pose in absolute scale.

[0174] In another embodiment, a 3D reconstruction method estimates the 3D structure of the scene and the camera poses, and the solution space of the camera poses is limited to pose pairs whose translation distance is equal to the known translation distance between the two camera poses C1 and C3, which is the result of the above process.

[0175] Fig.10Another embodiment of the present invention is shown, in which a scaling factor between a first coordinate system defined in an arbitrary scale and a second coordinate system defined in an absolute scale is determined. In this embodiment, the poses P2F and P4F of two cameras C2 and C4 facing the user relative to the coordinate system of the user's face U can be determined in absolute scale by a face tracking algorithm. Given the spatial transformation T1 between C1 and C2 and the spatial transformation T2 between C3 and C4, the pose P1F of the camera C1 facing the world and the pose P3F of the camera C3 facing the user can be determined in absolute scale in the coordinate system of the user's face U by a cascade method. Based on this, the translation distance between the position of camera C1 relative to P1F and the position of camera C3 relative to P3F can be determined in absolute scale as D_abs. The camera images of cameras C1 and C3 facing the world can be used to determine the pose P1W of camera C1 in the coordinate system OA associated with the physical object O10 at an arbitrary scale, and the pose P3W of camera C3 in the coordinate system associated with the physical object O10 at an arbitrary scale according to the 3D re-component of the physical object O10. According to these two poses, the translation distance between the position of camera C1 relative to P1W and the position of camera C3 relative to P3W can be determined as D_arb at an arbitrary reconstruction scale of the physical object O10. Finally, according to the camera pose determined by face tracking at an absolute scale, after determining the translation distance between the position of camera C1 and the position of camera C3 in the coordinate system of the physical re-component as D_arb at an arbitrary scale, and determining the translation distance between cameras C1 and C3 at an absolute scale as D_abs, the scale factor between the arbitrary scale and the absolute scale of the coordinate system of the physical re-component can be determined as D_abs / D_arb. The scale factor may be used to scale the coordinates of the 3D re-component of the physical object O10 so that it is defined at the determined scale after scaling.

[0176] In another embodiment, it is assumed that the spatial transformations T1 and T2 have a very small translation part that is treated as zero, and therefore, the translation distance D_abs in absolute scale relative to the user coordinate system is calculated between the two user-facing cameras C2 and C4, instead of calculating D_abs in absolute scale between the poses of the world-facing cameras C1 and C3 relative to the user coordinate system.

[0177] Fig.11 Another embodiment of the present invention is shown, and according to the embodiment of the present invention, it is shown how to merge the input data in more than four images into an estimated final scale factor.

[0178] According to an embodiment of the present invention, box 1101 corresponds to a separate estimation process of the scaling factor based on the four provided images I(C1), I(C2), I(C3), I(C4) and the two provided transformations T1 and T2.

[0179] Image I(C1) is an image captured by a first camera C1, showing a portion of a first object. Image I(C2) is an image captured by a second camera C2, showing a portion of a second object. T1 is a known transformation between the pose of camera C1 and the pose of camera C2. Based on image I(C1) showing a portion of the first object, a pose P1W of camera C1 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C2) showing a portion of the second object, a pose P2F of camera C2 in a second common coordinate system at an arbitrary scale is determined. The pose P1F of camera C1 in the second common coordinate system is determined by the transformation of pose P2F through the provided transformation T1. The transformation is optional, i.e., the translation part of T1 can be regarded as an identity transformation (i.e., the translation part is zero or can be ignored), so that the translation part of P1F is equal to the translation part of P2F.

[0180] Image I(C3) is an image captured by the third camera C3, showing a portion of the first physical object. Image I(C4) is an image captured by the fourth camera C4, showing a portion of the second physical object. T2 is a known transformation between the pose of camera C3 and the pose of camera C4. Based on image I(C3) showing a portion of the first physical object, the pose P3W of camera C3 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C4) showing a portion of the second physical object, the pose P4F of camera C4 in a second common coordinate system at an arbitrary scale is determined. The pose P3F of camera C3 in the second common coordinate system is determined by the transformation of pose P4F through the provided transformation T2. ​​This transformation is optional, that is, the translation part of T2 can also be regarded as an identity transformation (that is, the translation part is zero or can be ignored), so that the translation part of P3F is equal to the translation part of P4F.

[0181] According to the translation part of the posture P1W of the first camera C1 in the first common coordinate system, the translation part of the posture P1F of the first camera C1 in the second common coordinate system, the translation part of the posture P3W of the third camera C3 in the first common coordinate system, and the translation part of the posture P3F of the third camera C3 in the second common coordinate system, the scale factor S1 for scaling the first common coordinate system from an arbitrary scale to an absolute scale can be determined.

[0182] According to an embodiment of the present invention, block 1102 corresponds to another separate estimation process of the scale factor based on the four provided images I(C5), I(C6), I(C7), I(C8) and the two provided transformations T3 and T4. The images can be different images or partially the same images as used in block 1101. For example, I(C5) and I(C6) can be the same as I(C3) and I(C4), respectively. In another embodiment, I(C7) and I(C8) can be the same as I(C3) and I(C4), respectively.

[0183] Image I(C5) is an image captured by camera C5, showing a portion of a first physical object. Image I(C6) is an image captured by camera C6, showing a portion of a second physical object. T3 is a known transformation between the pose of camera C5 and the pose of camera C6. Based on image I(C5) showing a portion of the first physical object, the pose P5W of camera C5 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C6) showing a portion of the second physical object, the pose P6F of camera C6 in a second common coordinate system at an arbitrary scale is determined. The pose P5F of camera C5 in the second common coordinate system is determined by the transformation of pose P6F through the provided transformation T3. This transformation is optional, that is, the translation part of T3 can also be regarded as an identity transformation, so that the translation part of P5F is equal to the translation part of P6F.

[0184] Image I(C7) is an image captured by camera C3, showing a portion of a first physical object. Image I(C8) is an image captured by camera C8, showing a portion of a second physical object. T4 is a known transformation between the pose of camera C7 and the pose of camera C8. Based on image I(C7) showing a portion of the first physical object, the pose P7W of camera C7 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C8) showing a portion of the second physical object, the pose P8F of camera C8 in a second common coordinate system at an arbitrary scale is determined. The pose P7F of camera C7 in the second common coordinate system is determined by the transformation of pose P8F through the provided transformation T4. This transformation is optional, that is, the translation part of T4 can also be regarded as an identity transformation, so that the translation part of P7F is equal to the translation part of P8F.

[0185] According to the pose P5W of camera C5 in the first common coordinate system, the pose P5F of camera C5 in the second common coordinate system, the pose P7W of camera C7 in the first common coordinate system, and the pose P7F of camera C7 in the second common coordinate system, the scale factor S2 for scaling the first common coordinate system from an arbitrary scale to an absolute scale can be determined.

[0186] Finally, the individual ratio estimates S1 and S2 may be combined into a final ratio estimate S by fitting a mathematical model, for example, by one or more methods such as average, mean, median, probability maximization or RANSC.

[0187] The above example includes two separate ratio estimates S1 and S2, and combines them into a final ratio estimate. Of course, similarly, more than two separate calculation processes, for example, 3, 4, 5 or 100, etc., can be performed, and all estimates can be combined.

[0188] Fig.12 Another embodiment of the present invention is shown, and according to the embodiment of the present invention, it is shown how to merge the input data in more than four images into an estimated final scale factor.

[0189] According to an embodiment of the present invention, this example is based on six provided images I(C1), I(C2), I(C3), I(C4), I(C5), I(C6) and three provided transformations T1, T2 and T3.

[0190] Image I(C1) is an image captured by the first camera C1, showing a portion of a first physical object. Image I(C2) is an image captured by the second camera C2, showing a portion of a second physical object. T1 is a known transformation between the pose of camera C1 and the pose of camera C2. Based on image I(C1) showing a portion of the first physical object, the pose P1W of camera C1 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C2) showing a portion of the second physical object, the pose P2F of camera C2 in a second common coordinate system at an arbitrary scale is determined. The pose P1F of camera C1 in the second common coordinate system is determined by the transformation of pose P2F through the provided transformation T1. This transformation is optional, that is, the translation part of T1 can also be regarded as an identity transformation, so that the translation part of P1F is equal to the translation part of P2F.

[0191] Image I(C3) is an image captured by the third camera C3, showing a portion of the first physical object. Image I(C4) is an image captured by the fourth camera C4, showing a portion of the second physical object. T2 is a known transformation between the pose of camera C3 and the pose of camera C4. Based on image I(C3) showing a portion of the first physical object, the pose P3W of camera C3 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C4) showing a portion of the second physical object, the pose P4F of camera C4 in a second common coordinate system at an arbitrary scale is determined. The pose P3F of camera C3 in the second common coordinate system is determined by the transformation of pose P4F through the provided transformation T2. ​​This transformation is optional, that is, the translation part of T2 can also be regarded as an identity transformation, so that the translation part of P3F is equal to the translation part of P4F.

[0192] Image I(C5) is an image captured by camera C5, showing a portion of a first physical object. Image I(C6) is an image captured by camera C6, showing a portion of a second physical object. T3 is a known transformation between the pose of camera C5 and the pose of camera C6. Based on image I(C5) showing a portion of the first physical object, the pose P5W of camera C5 in a first common coordinate system at an arbitrary scale is determined. Based on image I(C6) showing a portion of the second object, the pose P6F of camera C6 in a second common coordinate system at an arbitrary scale is determined. The pose P5F of camera C5 in the second common coordinate system is determined by the transformation of pose P6F through the provided transformation T3. This transformation is optional, that is, the translation part of T3 can also be regarded as an identity transformation, so that the translation part of P5F is equal to the translation part of P6F.

[0193] Based on the poses P1W, P3W and P5W, i.e. the poses of cameras C1, C3 and C5 in the first common coordinate system, and the poses P1F, P3F and P5F, i.e. the poses of cameras C1, C3 and C5 in the second common coordinate system, the scale factor S for scaling the first common coordinate system from an arbitrary scale to an absolute scale can be determined in the following way: by (for example) one or more methods, such as iterative closest point (ICP), Umeyama method, or Kabsch method, or other least squares methods and / or RANSAC model fitting, point set based registration method, mathematical model fitting.

[0194] Fig.12 It is shown that 3 cameras are used to capture the first object and 3 cameras are used to capture the second object. Of course, even more than 3 camera poses in the first common coordinate system and 3 poses corresponding to the corresponding cameras in the second common coordinate system can also be used, for example, 4, 5, 6 or 100 pose pairs in the first and second common coordinate systems.

[0195] Other embodiments of the invention are further disclosed below without explicit reference to the drawings or figures.

[0196] According to other embodiments, the method assumes that the user's face is stationary positioned relative to the object to be tracked or reconstructed while capturing images useful for making a scale estimation.

[0197] According to other embodiments, the method detects when the user's face is stationary relative to the object, and then only uses images captured when the user's face is stationary relative to the object for scale estimation. This can be done (for example) by comparing the epipolar geometry of the two poses of the camera facing the user with the characteristic movement in the camera facing the world or vice versa. Another approach to determine whether the head has moved relative to the first object is based on a set of corresponding poses of the camera facing the user and the camera facing the world. These poses can be converted into 3D points representing the camera positions. An algorithm for determining a similarity transformation between the two sets of points is (for example) disclosed by Umeyama

[10] . After alignment, the residual error can be calculated. If the residual error exceeds a certain threshold, the two sets of corresponding poses are considered to be independent of the rigid body transformation. This indicates that the head (i.e., face) has moved relative to the first object.

[0198] According to another embodiment, the method processes and compensates for the movement of the face relative to the first physical object by estimating the movement of the head relative to the physical object through visual tracking to estimate the movement of the face in the camera image of the camera facing the user and the movement of the first physical object in the camera image of the camera facing the world.

[0199] According to another embodiment, the method processes and compensates for the motion of the face relative to the real object by estimating the motion of the head relative to the real object by visual tracking to separately estimate the motion of the face and the motion of the background in the camera image of the world-facing camera.

[0200] According to another embodiment, calculating the absolute spatial distance between cameras W1 and W2 using the poses of cameras F1 and F2 may include: calibrating the relative spatial position and orientation between coordinate systems F and W in a coordinate system at an absolute scale, for example, by a method such as that disclosed in reference

[11] .

[0201] According to another embodiment, the transformation between the coordinate systems F and W is a 6DoF (DoF: Degree of Freedom) rigid body transformation, including 3D rotation and 3D translation.

[0202] According to a further embodiment, the method provides and takes into account the spatial distance between the coordinate systems F and W.

[0203] According to another embodiment, the method regards the spatial distance between coordinate systems F and W as zero, thereby ignoring the translational movement of the posture of coordinate system (camera) F caused by the rotation of coordinate system (camera) W and vice versa, which results in the postures of cameras F1 and F2 corresponding to the postures of cameras W1 and W2 having an error less than or equal to the actual distance between coordinate system (camera) F and (camera) W.

[0204] According to a further embodiment, the method considers the orientation difference between coordinate systems F and W to be 180 degrees, ie the optical axes of the respective cameras are parallel and the cameras are facing in opposite directions.

[0205] According to one embodiment, the method uses a facial fiducial tracker like that disclosed in reference

[15] to perform face correction and determine the 2D positions of facial features and fiducials in the image.

[0206] According to one embodiment, the method uses a face tracking method to transmit a full 6D pose of the face facing the user camera, as disclosed in reference

[16] , where a statistical anthropometric 3D rigid model is used as an approximation of the human head. The transmitted 6D pose is then modified in scale and translation, possibly according to other specifications of the size of specific facial features (such as eye distance).

[0207] According to one embodiment, the method uses a face tracking method based on a pre-acquired, fitted or configured model of the user's specific face to deliver a full 6D pose of the user-facing camera associated with the face at absolute scale.

[0208] According to one embodiment, the method uses a face tracking method that delivers a 3D pose containing the orientation of the face relative to a camera facing the user (as disclosed in reference

[17] ), and uses the 3D pose together with detected facial features to infer additional pose translation information. The viewing angle between two positions corresponding to facial features can be used, for example, together with the actual distance between these two positions required to compensate for head rotation to make up for the distance from the camera to the face. The change in distance between different poses can be used to infer absolute scale information of the camera movement.

[0209] According to one embodiment, the method uses a face tracking method that delivers a 1D pose containing the left-right orientation (i.e., yaw orientation) of the face relative to a camera facing the user (such as disclosed in reference

[17] ), and uses this 1D pose together with detected facial features to infer additional pose translation information, assuming that these facial features are distributed in the horizontal line of the face, such as eye centers. The viewing angle between the two eye centers can be used together with the actual eye distance required to compensate for the 1D left-right rotation (yaw) of the head to make up the distance from the camera to the face / eyes. The change in distance between different poses can be used to infer absolute scale information of the camera movement.

[0210] According to another embodiment, the method assumes that the poses of cameras F1 and F2 are limited to being in front of the face, wherein the pose of F2 differs from the pose of F1 when (almost) only translating towards or away from the face, and the distance from the camera to the face / eyes is compensated using the detected eye positions, the relative viewing angle between the eyes, and the actual eye distance required, the eye positions being detected in the captured image by the method disclosed in reference

[18] , for example. This is also Figure 3 The visual angle between the two eyes can be calculated by the dot product of the corresponding viewing directions giving the cosine of the visual angle. The distance to the face can then be calculated as follows: divide half the eye distance by the tangent angle of half the visual angle.

[0211] The distance changes between the corresponding poses of cameras F1 and F2 and W1 and W2 can be used to infer the absolute scale information of the coordinate systems of cameras W1 and W2, respectively.

[0212] According to one embodiment, for at least one facial feature, a spatial characteristic is provided in absolute spatial units, e.g., the interpupillary distance of a human, from which the characteristic can be given as a single value or a probability distribution. The characteristic can be determined individually for a specific user or for multiple people. Similarly, multiple values / probability distributions can be defined for different groups of people (gender, race, age, etc.), and the relevant groups can be selected by manual user input or other automatic labeling or classification procedures (e.g., as disclosed in reference

[12] ).

[0213] According to one embodiment, a specific face may be automatically calibrated based on:

[0214] Absolute scale information provided to a second camera (e.g., a camera facing the user). Absolute scale information may be provided, for example, by depth information of a second object, the depth of which is obtained by defocus, time of flight, structured light, active illumination, brightness, photographic light, laser rangefinder, multi-frequency phase shift, interferometry, or passive stereo. Stereo with a small footprint, as is the case with mobile phones, works more reliably with a second camera (e.g., a camera facing the user) where the second object is a face close to the camera, typically no closer than 50 cm, as opposed to stereo with a first camera (e.g., a camera facing the world) where the first object is located at an arbitrary distance from the face. This is because the size of the baseline required for a certain depth "resolution" in the depth of the captured object depends on the distance of the captured object from the camera.

[0215] According to one embodiment, a specific face can be manually calibrated or a generic model can be used. Thus, the statistical model can also be used to determine the uncertainty in the proportion estimate, which indicates the degree to which the spatial characteristics of the face vary in different human bodies.

[0216] Manual calibration of the interpupillary distance can be done, for example, with a mirror or ruler. Face the mirror, keep your head upright, and position yourself in front of the mirror. Place the ruler horizontally in front of your face, as close as possible to the eyes, with the measurement marks visible in the mirror. The following measurement should be performed without moving the head or the ruler: With one eye closed and the other eye open, the measurement on the ruler below the center of the pupil can be read. The process can be repeated with the other eye (closing the previously open eye and opening the previously closed eye). The difference between the two readings is the interpupillary distance. Semi-automatic calibration of the interpupillary distance or other facial features in absolute scale can be performed, for example, by using a dual-camera setup, with camera pose estimation in absolute scale from images captured by the rear camera (e.g., marker-based tracking, object tracking, or SLAM in absolute scale). At the same time, the facial features to be calibrated are tracked in the camera facing the user. When the user's face is stationary relative to the actual object used for tracking by the rear camera, the absolute scale can be converted to facial features by using the known transformation between the camera poses.

[0217] Based on statistics such as disclosed in reference

[13] , a general model may (for example) contain means and probability distributions of absolute spatial properties of facial features, such as interpupillary distance. Such a model may also include non-deterministic information about certain values. The general model may also include multiple (joint) probability distributions of multiple measurements, such that the absolute proportion is determined according to the position in the parameter space of the maximum probability in the joint probability distribution, the joint spatial distribution being based on the properties of the observed facial features.

[0218] According to one embodiment, the method combines absolute scale estimates from different sources (including at least one absolute scale estimate based on at least one image of the user's face) and any of the following: IMU, GPS, known objects in the scene, defocused depth, manual input, passive stereo, flight time, structured light, active lighting, brightness, projected reduction depth estimation, historical data of scale estimates from previous object reconstructions, which historical data of scale estimates may be combined with object classification to only consider previous object reconstructions of previous objects.

[0219] According to an embodiment of the present invention, the estimation of proportion can be performed as an iterative process over multiple pose pairs, each pose pair comprising two poses at two time points. Inconsistencies between different measurements can be detected, and the best consistent value of proportion can be obtained by combining the different measurements by, for example, averaging, median, or maximum values ​​in a histogram, and may be weighted according to, for example, uncertainty in the individual proportion estimation processes, age, or inconsistency between measurements. Combining different measurements can also be performed by model fitting methods, such as implementing a Bayesian filter like a Kalman filter to infer absolute proportions.

[0220] Likewise, for many use cases, it can be assumed that the distance between the user's face and the user-facing camera has limited variation, for example, where the user is close to the device because the user is holding the device, or because the user is next to the display device to experience an AR application, and thus close to the user-facing camera. The limited distance variation makes the scenario more robust to the scale estimation process for standard cameras. It also realizes that with a camera facing the user, the depth is only estimated within a small range, and the depth of more distant objects, such as the house next door, cannot be estimated.

[0221] One possible implementation of the invention includes a combination with a depth camera facing the user that enables scale estimation through the user's facial appearance to rely on more accurate information about the actual face model geometry and the spatial dimensions of the decision. This enables the extraction of more details and increases the need to rely on statistics from a standard face base model, or to fit and warp a generic face model, or to configure user-specific scale values. Another advantage of using a depth camera is that face detection and pose estimation can be performed in very dark or very bright environments or in environments where the metric changes are very strong and / or frequent. In such environments, a standard monocular camera with a small dynamic range will most likely not be able to detect a face and estimate its pose.

[0222] Another possible embodiment of the present invention includes a second camera and / or a fourth camera, the second camera and / or the fourth camera being a depth camera (e.g., a depth camera facing the user), the depth camera implementing a scale estimation using a visual odometry method based on depth information through any physical object that appears in the depth camera's frustum. This forms a camera pose in absolute scale, even without utilizing any facial features.

[0223] In another embodiment, the second camera and / or the fourth camera (e.g., the camera facing the user) is an infrared camera, which is well suited for face detection and tracking, or imaging in at least one of the following bands: extreme ultraviolet, near ultraviolet, near infrared, mid-infrared, long-wave infrared, or far infrared.

[0224] In another embodiment, the second camera and / or the fourth camera (e.g., the camera facing the user) is composed of multiple cameras, such as a passive stereo camera or any other combination of two or more cameras, which may image different bands of the spectrum, such as visible light, extreme ultraviolet, near ultraviolet, near infrared, mid-infrared, long-wave infrared, or far infrared.

[0225] According to one embodiment, the method may further include: detecting a first user input.

[0226] According to one embodiment, the user is asked to perform a certain movement with the second camera (e.g., the camera facing the user), for example, a translation of 30 cm, which is measured based on the image captured by the second camera (e.g., the camera facing the user). The application can visualize the process while performing the process, such as Figure 7 shown.

[0227] The absolute scale estimation process may be triggered by a user interaction, which may be a button press, a screen touch, voice recognition, and / or motion recognition.

[0228] In another embodiment, the method is performed without any interaction with the user interface. In this embodiment, the method is performed in the background and determines the scale estimate when the camera motion is suitable for scale estimation.

[0229] Furthermore, the invention relates to a user interaction method implemented on a computer, as described herein, for enabling a user to interact with the method according to the invention.

[0230] The user input may include one or more user interactions. The user interaction may be one of: speaking, running, jumping, blinking, and / or moving any part of the user's body. The user interaction may also be pressing a button, touching a screen, speaking into a microphone, looking, or gesturing. The user interaction may also be: placing an object into the field of view of the camera so that the object can be identified based on an image of at least a portion of the object captured by the camera.

[0231] The user input can also be a specific camera movement that can be performed while the button is pressed. Such specific movements can be: moving the camera towards and away from the face, or moving the camera up and down, or moving the camera left and right.

[0232] The mobile phone, as used herein, comprises at least one camera for capturing images. The mobile phone also comprises a processing device, which can be used to perform any of the steps according to the present invention as described herein. The mobile phone also comprises a touch screen, which can display a graphical user interface so that a user can touch or press a physical button or a button displayed on the graphical user interface.

[0233] Embodiments of the present invention are described herein with reference to the use of a mobile device or handheld device, such as a mobile phone, but in principle the present invention can be applied with any processing device, such as implemented in a conventional computer device including one or more microprocessors for performing the corresponding steps (such as a wearable computer, a tablet computer, a mobile computer, a so-called laptop computer, or a head-mounted display, such as a computer device for optical see-through augmented reality applications). The steps described herein can also be performed by a network of processing devices (such as a computer network) and / or a network of mobile devices communicating with a server computer.

[0234] Embodiments of the present invention may be used in a variety of applications, including: augmented reality applications that enable the placement and display of virtual objects in a real environment; navigation applications that use a camera to estimate the user's position and / or movement; programs that simulate the spatial relationship of a captured object to other objects or the environment; or measurement applications designed to measure the distance between multiple points on a reconstructed object at absolute scale.

[0235] For example, embodiments of the present invention can be used in augmented reality applications. The heavy component of the real object in absolute proportion is used as a basic condition to determine the position of the camera relative to the object according to the 2D-3D correspondence established between the camera image and the real object reconstruction model. Such augmented reality applications can overlay virtual 3D objects, such as virtual chairs, in real-time views so that the objects appear to be stationary relative to the real objects. This requires mastering the camera position (i.e., position and orientation) relative to the real object. Since the heavy component of the real object is defined in absolute proportion according to embodiments of the present invention, the position can also be estimated in absolute proportion, which realizes the superposition of virtual objects in absolute proportion. Thus, if the real object is a table and the virtual object is a chair, when the virtual chair is placed next to the real table, the virtual chair will appear with the same size as the real table. On the contrary, when using heavy components in arbitrary proportions, the chair can be 5 times the height of the table or one-fifth the height of the table, both of which lead to unrealistic appearances. In augmented reality applications that should provide users with the opportunity to evaluate whether an object is spatially suitable for a real environment by overlaying a virtual model, it is particularly important to have a heavy component of the real object or environment in absolute proportion. For example, such an application can be used to visually assess whether a sofa will fit in a certain living room or whether it is too big. Without knowing the absolute proportions as provided by the invention, this would not be possible.

[0236] In another embodiment, embodiments of the present invention can be used in measurement applications. Embodiments of the present invention are used to reconstruct physical objects in absolute proportions. The software application can provide the user with the following options: select points on the heavy component, for example, by clicking these points with a pointing device (such as a mouse). The application can calculate the (Euclidean) distance between the two selected points and display the distance to the user. If such an application acts on a heavy component in absolute proportion, as provided by the present invention, the calculated distance is also in absolute proportion, for example, millimeters. Conversely, when using a heavy component in arbitrary proportions, the distance measured on the heavy component can only be used to compare the relative distances of each other on the object, but all measurements will be independent of absolute units, such as millimeters or inches.

[0237] In another embodiment, embodiments of the present invention may be used in visual navigation for estimating the user's motion based on visual odometry to locate the user and provide feedback. In order to update the position relative to the coordinate system, it is necessary to estimate the motion in absolute scale in which the map and navigation information are stored. Without absolute scale, only the covered trajectory can be determined, and the actual distance information covered cannot be determined. Similarly, due to changes in scale, the shape may be severely deformed after a period of time. Indoor navigation therefore requires that the measurements remain robust and consistent over the long term. Similarly, speed is also important, since speed refers to distance divided by time, which directly depends on the scale of the heavy component. Without heavy components in absolute scale, the speed information obtained by motion estimation is not in absolute scale.

[0238] Other applications can also benefit from repeatable reconstruction scales, even if the absolute relationship with respect to the actual distances is unknown. This repeatability is achieved by the proposed method. When (for example) reconstructions are created for several parts of a scene separately, it is necessary that the individual images are defined with the same scale. This makes it easier to merge parts of the scene. Repeatable scales also make it possible to overcome the problem of scale changes that may occur during longer reconstruction processes.

[0239] In general, further explanation of terms is given below, and the following other aspects and embodiments can be applied in conjunction with various aspects of the present invention.

[0240] As used herein, a camera is an image capture device for capturing image information of a physical object. Intrinsic camera parameters of at least one camera or multiple cameras may be provided or calibrated. The present invention may be applied in the following cases: image information is received from any camera that provides an image of a physical object. It is not limited to cameras for providing color images in RGB format. It may also be applicable to any other color format and also to monochrome images, for example, to cameras that provide grayscale format images. The camera used may also provide images with depth data. There is no need to provide depth data at the same resolution as the (color / grayscale) image. Cameras that provide images with depth data are generally referred to as RGB-D cameras. An RGB-D camera system may be a time-of-flight (TOF) camera system or a camera system using structured light. At least one camera or multiple cameras may also capture light that is invisible to the human eye. For example, at least one camera or multiple cameras may be an infrared camera that captures infrared light.

[0241] A physical object can be any object that has a physical geometric shape in reality. A physical object can also include one or more physical objects. For example, a physical object can be the interior of a room or a vehicle. Physical objects can also be, for example, a chair, a building, a mountain, a tree, or a table. The interior of a room (as a physical object) can also include a chair and / or a table. In this article, a physical object can also be referred to as a real environment or a real scene. A physical object can also be an arrangement of multiple physical objects. For example, an indoor object can be an arrangement of chairs, a table, and a sofa.

[0242] A geometric model of a real environment (3D graphics or also called 3D reconstruction) can be created by triangulating 2D observations that exist in multiple images captured by one or more cameras. Triangulation is a common method in 3D reconstruction procedures based on camera images, also known as structure from motion (SfM), see reference [2].

[0243] The camera pose describes the spatial relationship or rigid transformation between a camera at a specific position and a reference coordinate system. The reference coordinate system can be relative to a physical object or to a camera at another position. The spatial relationship or rigid transformation describes at least one translation, or at least one rotation, or a combination of these in 3D space, or at least one distance.

[0244] The 3D feature represents or describes the physical 3D features of the corresponding object or at least a portion of the corresponding object. The 3D features include, for example, but not limited to: points, edges, lines, segments, corners and / or any other geometric shapes.

[0245] To describe the geometry of an object, the positions of points, other geometric elements, and their other characteristics such as orientation are uniquely identified in a coordinate system. A particular element (eg, position) is defined by an ordered number or coordinate tuple.

[0246] We refer to this below as the Cartesian coordinate system, keeping in mind that representations of geometric figures in other coordinate systems (like, for example, polar coordinates or homogeneous coordinates) can be converted to the Cartesian coordinate system and vice versa through coordinate transformations, which describe the relationship between coordinate systems and provide formulas for representing specific coordinates in one coordinate system by corresponding coordinates in other coordinate systems.

[0247] In Cartesian coordinates, each element of a coordinate tuple corresponds to the signed distance of a point to the corresponding hyperplane. The distance between two coordinate tuples can itself be defined as the Euclidean distance, which describes the length of the line segment connecting the two coordinate tuples. The distance itself is thus also given as a 1D coordinate.

[0248] The meaning of the units along the coordinate axes should be defined. The measurement unit is a quantity that is a factor of an existing quantity representing the property (e.g., length). When the relationship between the coordinate system units defining the measured values ​​and the real spatial reference units is known, we characterize the measured values ​​(e.g., spatial coordinates defining the location of the measured values, and coordinates defining the spatial distance and the length between the spatial coordinates, respectively), and assign them absolute spatial units, also called absolute spatial scales. The real spatial reference units can, for example, be meters (metric) or any other units with a fixed, known conversion ratio to meters.

[0249] Measurements can be at an absolute scale by setting the spatial base units of the coordinate system, where measurements are defined directly in real-world spatial reference units, or by specifying the spatial scale of the coordinate system relative to the real-world spatial reference units.

[0250] An example of a measurement without absolute spatial units but in a coordinate system at an arbitrary scale is given, for example, by reconstructing 3D points by triangulating corresponding point features in two images captured at different camera poses in space, where the length of the baseline between the two camera poses is unknown. Although the coordinate axis units are the same, the units themselves are not determined. That is, although the ratio of two distances in the coordinate system is correct, the absolute scale of the entire coordinate system relative to reality is unknown. The coordinate system and the heavy components themselves are therefore considered to be in scale. Absolute scale does not refer to the absolute translation compensation value of the object, which depends on the origin of the coordinate system, but to the absolute size.

[0251] Facial features and base points may include: position (corner position, center position, or border area), size, shape, contour, area, proportion, ratio, distance between the left eye and the right eye, and appearance of the left eye and the right eye (pupil, iris, cornea, sclera, inner canthus, outer canthus, center, upper and lower eyelids, eyelashes, ...), nose bridge, nose (nose tip, nose dorsum, nose wing, nostril columella, ...) (size, shape), philtrum, lips, left and right ears, left and right eyebrows, teeth, left and right cheeks, jaw, neck, Adam's apple, skin structure and firmness (such as pores), facial and head hair, etc.

[0252] Visual odometry refers to a method for determining the position and orientation of a camera by analyzing related camera images. When depth information with absolute scale (e.g., the distance of one pixel on the imaging surface in millimeters) is provided for at least one pixel in a camera image, visual odometry can determine the camera pose (i.e., position and orientation) at absolute spatial scale. The term visual odometry is often used interchangeably with SLAM.

[0253] References :

[0254] 1. Davison, Andrew J. et al. "MonoSLAM: Real-time single camera SLAM." Pattern Analysis and Machine Intelligence, IEEE Transactions on 29.6 (2007): pp. 1052-1067.

[0255] 2.Hartley, Richard and Andrew ZissermanMultiple view geometry incomputervision. Volume 2. Cambridge, 2000.

[0256] 3.Azuma, Ronald et al. “Recent advances in augmented reality.”

[0257] Computer Graphics and Applications, IEEE 21.6(2001): pages 34-47.

[0258] 4. Strasdat, Hauke, JMM Montiel, and Andrew J. Davison. “Scaledrift-awarelarge scale monocular SLAM.” Proceedings of Robotics:

[0259] Science and Systems (RSS). Vol. 2. No. 3. 2010.

[0260] 5. Lemaire, Thomas et al. “Vision-based slam: Stereo and monocular approaches.” International Journal of Computer Vision 74.3 (2007):

[0261] pp. 343 - 364.

[0262] 6. Lieberknecht, Sebastian et al. “RGB-D camera-based parallel tracking and meshing.” Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on. IEEE, 2011.

[0263] 7. Klein, Georg, and David Murray. “Parallel tracking and mapping for small AR workspaces.” Mixed and Augmented Reality, 2007. WACV

[0264] 2007. 6th IEEE and ACM International Symposium on. IEEE, 2007. 8. Castle, Robert, Georg Klein, and David W. Murray. “Video-rate localization in multiple maps for wearable augmented reality.”

[0265] Wearable Computers, 2008. ISWC 2008. 12th IEEE International Symposium on. IEEE, 2008.

[0266] 9. Nützi, Gabriel et al. "Fusion of IMU and vision for absolute scale estimation in monocular SLAM." Journal of intelligent & robotics systems 61.1 - 4 (2011): pp. 287 - 299.

[0267] 10. Umeyama, Shinji. "Least - squares estimation of transformation parameters between two point patterns." Pattern Analysis and Machine Intelligence, IEEE Transactions on 13.4 (1991): pp. 376 - 380.

[0268] 11. Esquivel, Sandro, Felix Woelk and Reinhard Koch. "Calibration of a multi - camera rig from non - overlapping views." Pattern Recognition.

[0269] Springer Berlin Heidelberg, 2007. pp. 82 - 91.

[0270] 12. Han, Hu et al. "Demographic Estimation from Face Images: Human vs. Machine Performance."

[0271] 13. Dodgson, Neil A. "Variation and extrema of human interpupillary distance." Proceedings of SPIE. Vol. 5291. 2004.

[0272] 14. Clipp, Brian et al. "Robust 6dof motion estimation for non -

[0273] overlapping, multi - camera systems." Applications of Computer Vision,

[0274] 2008. IEEE Workshop on WACV 2008. IEEE, 2008

[0275] 15. Ren, Shaoqing et al. “Face Alignment at 3000FPS via Regressing Local Binary Features.”

[0276] 16. Martins, Pedro and Jorge Batista. “Accurate single view model-based headpose estimation.” Automatic Face & Gesture Recognition, 2008.

[0277] FG'08. 8th IEEE International Conference on. IEEE, 2008.

[0278] 17. Asthana, Akshay et al. “Incremental Face Alignment in the Wild.”

[0279] Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014.

[0280] 18. Wang, Peng et al. “Automatic eye detection and its validation.”

[0281] Computer Vision and Pattern Recognition - Workshops, 2005. CVPR Workshop. IEEE Computer Society Conference on. IEEE, 2005.

[0282] 19. Turk, Matthew A. and Alex P. Pentland. “Face recognition using eigenfaces.” Computer Vision and Pattern Recognition,

[0283] 1991.Proceedings CVPR'91.,IEEE Computer Society Conference on.

[0284] IEEE,1991.

[0285] 20.Daniel Kurz,Selim Behimane“Method of providing a descriptor foratleast one feature of an image and method of matching features”US

[0286] 20120219188A1.

Claims

1. A method for determining spatial coordinates of a 3D re-construction of at least a portion of a first object, comprising: Acquire a first image including at least a portion of a first object from a first camera system; acquiring a second image from a second camera system, the second image including at least a portion of a second object associated with the known geometric property, wherein the first camera system and the second camera system have a known spatial relationship; determining a scale of at least a portion of the second object based on the second image and the known geometric property of the at least a portion of the second object; and One or more coordinates of a geometric model of the first object at an absolute scale are determined based on the known spatial relationship and the scale of the at least a portion of the second object.

2. The method of claim 1, wherein: The second object includes at least a portion of a human face.

3. The method of claim 2, wherein: The proportion of at least a portion of the human face is determined based on a generic human face model.

4. The method of claim 2, wherein: The face belongs to a specific user, and wherein the proportion of the at least a portion of the face is determined based on a user-specific face model.

5. The method according to any one of claims 1 to 4, wherein: The first camera system and the second camera system are included in the electronic device.

6. The method according to any one of claims 1 to 4, wherein: The viewing cone of the first camera system and the viewing cone of the second camera system do not overlap.

7. The method according to any one of claims 1 to 4, further comprising: The virtual object is presented in a proportion consistent with the first physical object according to one or more coordinates of the geometric model of the first physical object in absolute proportion.

8. A non-transitory computer readable medium comprising computer readable code for determining spatial coordinates of a 3D re-construction of at least a portion of a first physical object, the computer readable code being executable by a processor to: Acquire a first image including at least a portion of a first object from a first camera system; A second image is acquired from a second camera system, the second image including at least a portion of a second object associated with the known geometric property, wherein: The first camera system and the second camera system have a known spatial relationship; determining a scale of at least a portion of the second object based on the second image and the known geometric property of the at least a portion of the second object; and One or more coordinates of a geometric model of the first object at an absolute scale are determined based on the known spatial relationship and the scale of the at least a portion of the second object.

9. The non-transitory computer readable medium of claim 8, wherein: The second object includes at least a portion of a human face.

10. The non-transitory computer readable medium of claim 9, wherein: The proportion of at least a portion of the human face is determined based on a generic human face model.

11. The non-transitory computer readable medium of claim 9, wherein: The face belongs to a specific user, and wherein the proportion of the at least a portion of the face is determined based on a user-specific face model.

12. The non-transitory computer readable medium of any one of claims 8 to 11, wherein: The first camera system and the second camera system are included in the electronic device.

13. The non-transitory computer readable medium of any one of claims 8 to 11, wherein: The viewing cone of the first camera system and the viewing cone of the second camera system do not overlap.

14. The non-transitory computer readable medium of any one of claims 8 to 11, further comprising computer readable code for performing the following operations: The virtual object is presented in a scale consistent with the first object according to the one or more coordinates of the geometric model of the first object in absolute scale.

15. A system for determining spatial coordinates of a 3D re-construction of at least a portion of a first physical object, comprising: The first camera system; a second camera system having a known spatial relationship to the first camera system; one or more processors; and A computer readable medium comprising computer readable code executable by the one or more processors to: Acquire a first image including at least a portion of a first object from a first camera system; acquiring a second image from a second camera system, the second image including at least a portion of a second object associated with the known geometric property; determining a scale of at least a portion of the second object based on the second image and the known geometric property of the at least a portion of the second object; and One or more coordinates of a geometric model of the first object at an absolute scale are determined based on the known spatial relationship and the scale of the at least a portion of the second object.

16. The system of claim 15, wherein: The second object includes at least a portion of a human face.

17. The system of claim 16, wherein: The proportion of at least a portion of the human face is determined based on a generic human face model.

18. The system of claim 16, wherein: The face belongs to a specific user, and wherein the proportion of the at least a portion of the face is determined based on a user-specific face model.

19. The system of any one of claims 15 to 18, wherein: The viewing cone of the first camera system and the viewing cone of the second camera system do not overlap.

20. The system of any one of claims 15-18, further comprising computer readable code for: The virtual object is presented in a proportion consistent with the first object according to the one or more coordinates of the geometric model of the first object in absolute proportion.

Images