Boundary box transformation for object depth estimation in multi-camera device

Abstract

An apparatus includes a processor, an image sensor, and a memory storing instructions to obtain an image from the sensor and process the first image to identify coordinates of a bounding box around an object. The device processes the region within the first frame to determine a 2-D position of a landmark associated with the object, derives a first 3-D position of the landmark, and uses the 3-D landmark position to determine coordinates of a second frame defining a boundary of the object in the second image. The device processes the region within the second frame to determine a 2-D position of the landmark and derives a second 3-D position of the landmark using triangulation. In general, a device obtains an image, detects an object and a landmark, determines a 2-D position and a 3-D position of the landmark, and triangulates the 3-D position.

Description

[0001] Declaration of priority

[0002] This application claims the benefit of priority to Greek patent application serial number 20230100669, filed August 10, 2023, and U.S. patent application serial number 18 / 475,720, filed September 27, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to techniques in the field of computer vision systems. More precisely, this application describes techniques related to the subfield of processing depth estimation in computer vision systems, where depth estimation is the process of inferring the distances of objects in a real-world scene. Background Technology

[0004] Computer vision systems are systems that use algorithms and techniques to enable computers to interpret and analyze visual data from the world around us. These systems can be designed to analyze video to identify objects, people, and other visual patterns, as well as extract useful information from visual data across a wide range of contexts. Computer vision systems typically involve a combination of hardware components and software components. Hardware may include image sensors (e.g., camera devices) for capturing images and video, while software includes algorithms for processing and analyzing the captured images and video. Computer vision systems can be used in a variety of applications, such as augmented reality (AR), autonomous vehicles, robotics, security and surveillance, medical imaging, and more. In each of these applications, one of the more common tasks performed by computer vision systems is, for example, determining the position of an object in three dimensions relative to some origin or reference point by estimating the depth of an object detected in a real-world scene.

[0005] In the field of AR, the depth of objects can be detected using head-mounted AR devices (e.g., AR glasses). Obtaining an accurate representation of an object's depth and position is particularly important in AR applications. Accurate depth estimation allows for the tracking of real-world objects and the rendering of virtual objects, enabling the virtual objects to be properly positioned within the real-world scene and appear realistic. For example, when virtual objects are rendered with appropriate depth information, they appear to be in the correct position relative to real-world objects. This enhances the visual coherence and immersion of the AR experience, allowing virtual content to seamlessly integrate with the user's surrounding environment. Attached Figure Description

[0006] In accompanying drawings that are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. To facilitate identification of any discussion of a particular element or operation, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the accompanying drawings:

[0007] Figure 1 illustrates a conventional technique for determining the depth of real-world objects using a multi-camera augmented reality (AR) device.

[0008] Figure 2 This illustrates a technique consistent with some embodiments of the present invention for determining the depth of a real-world object using a multi-camera device, wherein a region of interest in one image is determined based on a region of interest in another image.

[0009] Figure 3 Examples are shown that, consistent with some embodiments of the present invention, a 3-D pose estimator can be used to derive 3-D positioning data of one or more landmarks associated with an object by taking 2-D positioning data of one or more landmarks as input.

[0010] Figure 4 This is a block diagram illustrating examples of functional components (e.g., hardware components) of an AR device (e.g., AR glasses) consistent with this embodiment, with which the methods and techniques described herein can be implemented.

[0011] Figure 5 It is a block diagram illustrating the software architecture that can be installed on any one or more of the devices described herein. Detailed Implementation

[0012] This document describes methods, systems, and computer program products for determining the location of an object (e.g., a human hand, body, or part thereof) by estimating the depth of an object observed using a multi-camera augmented reality (AR) device. In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding of various aspects of different embodiments of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without all of these specific details.

[0013] In AR applications—particularly for head-mounted AR devices such as AR glasses—estimating the depth of an object and thus determining its precise location is important for a variety of reasons. For example, estimating the depth of an object is crucial because it allows the AR device to accurately track real-world objects and appropriately render virtual content (e.g., virtual objects), thus realistically enhancing real-world scenes. If the dimensions of a particular real-world object are known in advance, for example, because the AR device accesses an accurate 3D model of that object (e.g., a particular user's hand), the depth of that particular object can be accurately determined by analyzing a single image (or video frame) depicting the object. However, when the dimensions of an object are unknown in advance, often as is the case with random human body parts (e.g., hands and heads), the AR device may only be able to calculate a rough estimate of the object's depth using an estimate of its actual dimensions. For example, if the object is a human hand, the AR device could use the average size of a human hand as an estimate of the actual size of the hand depicted in a single image to attempt to determine the hand's depth or distance.

[0014] Due to the challenges involved in accurately estimating the depth of an object from a single image, many AR devices use alternative methods that rely on triangulation in a multi-camera setup. Such a method is illustrated in Figure 1. Consistent with this method, as shown in Figure 1, AR device 100 includes two calibrated cameras 102-A and 102-B, each with known intrinsic parameters (e.g., focal length, principal point, lens distortion model) and extrinsic parameters (baseline, relative position, and orientation). When each camera captures an image, two corresponding images—a first image 104-A obtained with the right camera 102-A and a second image 104-B obtained with the left camera 102-B—are processed. One or more computer vision algorithms (e.g., object detection algorithms or bounding box algorithms) are used to analyze the two images 104-A and 104-B to first identify the object depicted in each image (e.g., a hand), and then to construct regions of interest 106-A and 106-B, or bounding boxes, that geometrically enclose the identified object in each image. Here, object detection algorithms (e.g., bounding box algorithms) mainly involve locating and delineating the extent or boundary of a region of interest within an image or video frame, where the region of interest surrounds an object of interest.

[0015] Once the coordinates of bounding boxes 106-A and 106-B for objects in each of the two images have been determined, one or more computer vision algorithms are used to process the portions of each image corresponding to the respective bounding boxes 106-A and 106-B in order to identify one or more features of the object, or, in the case of certain objects (e.g., hands), to identify one or more distinct two-dimensional (2-D) landmarks. As shown in Figure 1, during the second processing step, the fingertip of the index finger of the hand in each of the individual images 104-A and 104-B is identified as 2-D landmarks 108-A and 108-B. In this example, only a single landmark is shown. However, in practice, several 2-D landmarks can be identified.

[0016] Once the 2-D position data of corresponding (e.g., matching) 2-D landmarks 108-A and 108-B of the object have been determined using the geometry of the AR device and its calibrated camera setup, triangulation technique 110 is applied to calculate the three-dimensional (3-D) position data of the landmarks. For example, to estimate the depth of the landmarks, the AR device calculates the rays emanating from the center of the camera setup through the corresponding 2-D landmarks in each image. These rays represent the visual projection lines of each landmark. By applying triangulation, the system intersects the corresponding rays from the viewpoints of the two camera setups. The intersection point represents the 3-D position of the object in the real world. This triangulation point provides an accurate estimate of the landmark's position in 3-D space, to some extent making the accuracy of the landmark's position independent of the object's size estimation. By determining the 3-D position data of several landmarks, an accurate estimate of the object's position in 3-D space can be obtained.

[0017] The accuracy of the calculated 3D location data 112 of the landmarks is highly influenced by and primarily determined by the measurement accuracy of the associated 2D landmarks 108-A and 108-B. The location data of the 2D landmarks are typically determined by computer vision algorithms that process each image separately, specifically the portion of each image corresponding to the bounding box of the object in the image. By restricting the operation or processing of the landmark detection algorithm to the image region defined by the bounding box, the overall accuracy of landmark detection is significantly improved while simultaneously reducing the processing time and power required for landmark recognition. Therefore, determining the bounding boxes of objects in an image using bounding box detection algorithms is an important processing step in various computer vision tasks, such as object detection and tracking. By determining the location of the bounding boxes of objects in an image, the landmark search space is reduced, thereby improving the accuracy of landmark location determination while reducing runtime (e.g., processing time) and power requirements. For battery-powered head-mounted AR devices, using bounding boxes also helps maintain battery life.

[0018] While using bounding box detection algorithms has its advantages, performing these algorithms to predict the location of object bounding boxes incurs high costs in terms of both runtime (e.g., processing time) and power consumption. Furthermore, in many cases, each image may depict multiple objects. For example, in the context of AR applications for head-mounted AR devices, the AR application may track the hands of the end user wearing the AR device (as opposed to one hand as shown in Figure 1). When tracking multiple objects, each image may depict multiple objects (e.g., two hands), and therefore the bounding boxes in each image may overlap. When the bounding boxes of each object overlap within a single image, identifying the corresponding bounding boxes between individual images can become even more difficult.

[0019] This paper describes a technique for determining the location of objects in 3D space using a multi-camera device such as a head-mounted AR device, wherein a region of interest (ROI) in a second image is determined based on a ROI in a first image. In conventional depth estimation techniques, given a pair of corresponding image or video frames, a bounding box detection algorithm is applied twice—once for each of the two individual images or frames. Consistent with the embodiments of this invention, the need to perform the bounding box detection process twice (once for each image or frame) is eliminated, thereby reducing processing time and power. Instead, a bounding box algorithm is first applied to the first image in the corresponding image pair. Then, a landmark detection algorithm is used to analyze the region of the first image defined by the bounding box, which determines 2D location data of one or more landmarks associated with an object depicted within the region defined by the bounding box. The 2D location data of one or more landmarks is provided as input to a 2D-to-3D lifter—a deep neural network that generates 3D location information from the 2D location information. The bounding box is then projected into the second image using the obtained 3D location information of one or more landmarks and known parameters of the camera device of the AR device. Once a bounding box is projected onto the second image, the region in the second image corresponding to the bounding box is processed using a 2-D landmark detection algorithm to generate 2-D position data of the landmark as depicted in the second image. Finally, the 2-D position data of the landmark obtained from processing the first image and the 2-D position data of the landmark obtained from processing the second image are used as input to a triangulation process that utilizes device parameters to generate accurate 3-D position data of the landmark, and thus generate objects associated with the landmark.

[0020] Consistent with embodiments of the present invention, because the bounding box detection algorithm or bounding box detector is applied only once to one image in an image pair, processing time and power are reduced without sacrificing the increased accuracy resulting from limiting the area in the image where 2-D landmark search is performed. While the techniques described herein are presented in the context of head-mounted AR devices such as AR headsets or AR glasses, those skilled in the art will readily recognize that the innovative techniques described herein are applicable to a wide variety of other applications, use cases, and contexts. Further aspects and advantages of the invention will be conveyed through the following description of several figures.

[0021] Figure 2 An improved technique for determining the depth of real-world objects using a multi-camera device, consistent with some embodiments of the present invention, is illustrated. When a multi-camera device, such as a pair of head-mounted AR glasses 200, acquires a first image 202-A from a first camera or image sensor and a second image 202-B from a second camera or image sensor, Figure 2 The technique illustrated begins here. These two images are corresponding images because they were captured simultaneously and depict the same real-world scene—albeit from slightly different perspectives. For example, two camera devices are typically located at different positions on an AR device, resulting in slight variations in the viewpoint from which they capture the real-world scene. Although referred to as images herein, an image can be a separate frame from a sequence of frames captured as a video stream by each camera device.

[0022] Identify the object and determine the bounding box for the first image.

[0023] After obtaining two corresponding images, the next step 204 involves processing one of the image pairs (e.g., 202-A and 202-B) by identifying one or more objects of interest within image 202-A and determining the bounding box of each identified object of interest within the image. This is achieved by applying a bounding box detection algorithm or bounding box detector to image 202-A. Figure 2In the example presented, the object of interest is a hand. Therefore, in image 202-A captured by the left camera, the result of step 204 is a rectangular box 206, shown as defining or enclosing the boundary of the hand of the person wearing the AR device 200 as depicted in image 202-A. In this example, the bounding box detector is applied only to one of the two corresponding images (in this case, image 202-A captured by the left camera). However, it should be noted that the specific image—the left or right image—is not important. What is important is that the bounding box detector is applied only to the first image, and then the location of the bounding box in the first image is used to determine the location of the bounding boxes in one or more corresponding images, thereby reducing overall processing time and power consumption.

[0024] A bounding box detector or bounding box detection algorithm is a computer vision algorithm or technique designed to identify and locate an object of interest (ROI) within an image by enclosing it with a rectangular bounding box. The algorithm automatically determines the location and extent of an object based on its visual characteristics. Initially, the bounding box detection algorithm generates a set of potential object proposals or candidate regions within the image that may contain the object. Various methods can be used to generate these proposals, such as selective search, Region Proposal Networks (RPNs), or sliding window methods. The bounding box detection algorithm extracts relevant features from the proposed regions or the entire image. These features can be based on color, texture, shape, or other visual attributes. Common techniques include using convolutional neural networks (CNNs) to extract depth features from the image. The extracted features are then used to classify whether each proposed region contains an ROI (e.g., a hand, human body, head). This typically involves training a classifier such as a Support Vector Machine (SVM), Random Forest, or Deep Neural Network on a labeled dataset to learn discriminative properties of object categories. After classifying the presence of an object, the bounding box detection algorithm refines it by adjusting the position and size of the bounding box. This regression step helps improve the accuracy of bounding box localization by estimating the precise boundaries of the identified objects. The output of an object detection algorithm is a region of interest (ROI), which defines the (approximate) location of an object within an image, for example, through bounding boxes or pixel-by-pixel segmentation, and includes a corresponding class label and / or confidence score. Popular bounding box detection algorithms include Faster R-CNN, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and RetinaNet. Of course, other algorithms can be used.

[0025] Process bounding boxes to identify 2D location data of landmarks

[0026] After one or more bounding boxes have been identified for the first image 202-A, the next step 208 involves applying another computer vision algorithm (called a 2-D landmark detection algorithm or landmark detector) to identify the 2-D location data of various landmarks within the region of each identified bounding box. Figure 2 As shown, the hand depicted in image 202-A has been marked with landmarks to indicate the location where each landmark in the image was detected. These landmarks are visual indicators that highlight specific points of interest or key landmarks on the hand, such as fingertips, knuckles, or joints. For example, the landmark with reference numeral 210 indicates the location of the landmark at the tip of the thumb.

[0027] 2D landmark detection algorithms, or landmark detectors (sometimes called keypoint detection algorithms or keypoint localization algorithms), are computer vision algorithms designed to identify and locate specific points of interest (called landmarks or keypoints) within an image. These landmarks are unique and meaningful locations that can be used as references for further analysis or interaction. In the context of augmented reality (AR) applications used to identify hands or other body parts, 2D landmark detection algorithms are used to detect and locate points on the hand, such as fingertips, palms, joints, or other anatomical landmarks.

[0028] 2D landmark detection algorithms are typically trained on labeled datasets that provide manually labeled or ground truth landmarks for each hand or body part image. The dataset can include a variety of hand or body poses, viewpoints, and conditions to capture variations in real-world scenes. The algorithm extracts relevant features from the input image or image region surrounding the hand or body part. These features can be based on color, texture, gradient, or other visual properties that help to accurately distinguish and locate landmarks. Using the extracted features, the 2D landmark detection algorithm predicts the location, or 2D coordinates, of the landmark within the image. Depending on the algorithm architecture, this can involve regression, classification, or a combination of both techniques. Machine learning techniques such as deep neural networks, CNNs, or recurrent neural networks (RNNs) are typically used to learn the mapping between image features and landmark locations. After landmark localization, post-processing techniques can be applied to refine the results, remove outliers, or enforce geometric constraints. These techniques can include filtering, smoothing, or statistical methods to improve the accuracy and robustness of the detected landmarks. The output of the 2D landmark detection algorithm is a set of coordinates representing the 2D location data of the local landmarks representing the identified hand or body part within the image.

[0029] Estimated 3D location data is generated from 2D location data.

[0030] After the 2-D position data of the landmarks for each object have been detected in the first image, the next step 214 involves using a 3-D pose estimator to generate estimated 3-D position data of the landmarks using the 2-D position data. While several techniques exist that can be used, in some implementations, a 2-D to 3-D lifter network (a specific type of deep neural network) is used to generate the estimated 3-D position data of the landmarks, using at least the 2-D position data of the landmarks as input and, in some cases, an estimate of the object's size as input. Note that at this point, the size of the object is unknown and therefore can be estimated. Thus, the accuracy of the estimated 3-D position data of the landmarks initially derived by the monocular pose estimator is referred to herein as "estimated," as the result is generally less accurate than the final result obtained via triangulation processing (described in more detail below). Furthermore, once the triangulation process is completed and the object's dimensions are determined through it, subsequent estimations performed by the monocular pose estimator can include the object's dimensions as input, thereby making the estimation of the 3-D position data more accurate than the estimation in the initial stage, in which the actual dimensions of the object were unknown.

[0031] like Figure 3 As shown, consistent with some implementations, given 2-D position data 304 of a landmark (such as obtained from analyzing a single image 306), a 2-D to 3-D lifter network 300 (deep neural network) is used to generate 3-D position data 310 of a landmark associated with a hand. The input to the lifter network 300 includes image 306, which includes 2-D position data 304 of landmarks of objects detected in the image. In this example, the 2-D position data represents the detected landmarks on the hand in the first (left camera) image 202-A. Additionally, camera device parameters 308 (e.g., an intrinsic camera device matrix containing focal length and principal point) can be provided as input to the network 300. Optionally, in some implementations, an object scaling factor 302 associated with the detected object (e.g., a reference skeletal metric in the case of a hand) can be provided as input to the lifter network 300. Here, in Figure 3 In the example presented, the average length of the reference bone 302 in the hand (e.g., the length between two specific landmarks) can be included as input to help scale the resulting 3D position.

[0032] The lifter network 300 performs feature extraction on the input data to capture meaningful representations. This can involve convolutional layers to extract hierarchical visual features from 2-D landmark locations and camera device parameters. The lifter network 300 can also be combined with other layers, such as fully connected layers or recurrent layers, to capture more complex relationships and dependencies within the data.

[0033] The lifter network 300 uses the extracted features to estimate the depth, or 3D position data, of each landmark. It learns a mapping between 2D position data and corresponding depth values. By utilizing camera device parameters, the lifter network 300 can account for perspective distortion and project the 2D position data into 3D space. If a reference bone length is provided, it can help scale the estimated 3D position data to an appropriate size.

[0034] The output of the lifter network 300 is the generated 3D position data for each landmark. These 3D positions represent the estimated spatial position of the landmark within a 3D coordinate system defined by the camera device. The positions are typically expressed as 3D coordinates (X, Y, Z) relative to a selected reference point or origin.

[0035] The lifter network 300 is trained on a dataset containing labeled 2-D landmark position pairs and their corresponding ground truth 3-D positions. The lifter network 300 learns to predict accurate 3-D position data based on the input 2-D position data, camera device parameters, and (if available) a reference bone length. Training involves optimizing the network's parameters to minimize the difference between the predicted 3-D positions and the ground truth positions. Through this process, the 2-D to 3-D lifter network utilizes deep learning techniques to transform the 2-D position data of hand landmarks, along with camera device parameters and optionally a reference bone length, into 3-D position data.

[0036] Projecting the second bounding box

[0037] Refer again Figure 2 After determining the estimated 3D position data of the landmark, the next step 214 involves projecting a bounding box into the 2D space of the second image 202-B using the 3D position data of the landmark, where the goal is to ensure that the second bounding box 216 encloses the object (e.g., a hand) as depicted in the second image 202-B. Using the estimated 3D position data of the object (e.g., the hand) obtained from processing the first image 202-A, a transformation is applied using a so-called rigid transformation matrix to align the 3D landmark with the viewpoint of the left camera device. For example, this transformation involves using extrinsic parameters that define the relationship between the two camera devices, such as rotation and translation between the coordinate systems of the camera devices. This transformation can be expressed as follows...

[0038]

[0039] Here, X Second These are the distortion-corrected (homogeneous) pixel coordinates in the second image, where K is the intrinsic matrix. It is a rigid transformation (external parameter of the camera device), and X Main It is the (homogeneous) 3-D coordinate of the landmark in the main camera device reference frame.

[0040] In a multi-camera setup, a rigid transformation matrix "T" facilitates the transformation of points (e.g., landmarks) from the coordinate system of one camera device to the coordinate system of another, and vice versa. By applying this transformation, a correspondence can be established between images from two camera devices, and triangulation can be performed to estimate 3D position data of objects in the scene. The camera device matrix "K" represents the intrinsic parameters of the camera device, describing its internal properties and characteristics. These intrinsic parameters are used to correlate the 3D world coordinates of the scene with their corresponding 2D pixel coordinates. Using the camera device matrix "K" and the transformation matrix "T," 3D position data can be projected to determine the 2D position data of a second bounding box 216 associated with the image plane or coordinate system of the second camera device.

[0041] Consistent with some implementations, the precise coordinates of the second bounding box 216 can be determined by calculating the smallest rectangle enclosing all 3-D to 2-D projected landmarks. Alternatively, in some implementations, only a subset of the landmarks (or even a single landmark representing the centroid) can be projected into the coordinate space or image plane of the second image, and the size of the bounding box can be assumed to be equal to the size of the bounding box in the first image (e.g., the image from the left camera device 202-A). In some implementations, to compensate for potential inaccuracies in the estimated 3-D position data derived from the lifter network, the second bounding box 216 can be expanded or enlarged using a predefined scaling factor.

[0042] Apply the landmark detection algorithm to the second bounding box

[0043] After the second bounding box has been projected onto the image plane of the second image (e.g., an image obtained from the right image sensor 202-A), the next step 218 involves processing the region of the second image represented by the second bounding box with a landmark detection algorithm or landmark detector in the same manner as that performed for the first image (e.g., the step with reference numeral 208) to accurately identify the 2-D location data of the landmarks of the object as depicted in the second image.

[0044] Use triangulation to determine the accurate 3D position data of the boundary marker.

[0045] Finally, in final step 222, the 2-D position data of the landmark obtained from analyzing the first image 202-A and the second image 202-B are used in the triangulation algorithm or processing to derive the final 3-D position data of the object's landmark. With the accurate 3-D position data of the landmark known, the AR device can utilize this data as input to several different algorithms or processes, including object tracking, gesture detection, and displaying virtual content.

[0046] Despite the combination Figure 2 and Figure 3 The examples shown and described involve determining 3-D positional data of a single object (e.g., a hand), and those skilled in the art will readily understand that the techniques described herein are applicable to scenarios in which multiple objects (including objects other than the hand) are tracked.

[0047] Augmented Reality (AR) Device Examples

[0048] Figure 4 This is a block diagram illustrating examples of functional components (e.g., hardware components) of an AR device (e.g., AR glasses 200) consistent with this embodiment, with which the methods and techniques described herein can be implemented. Those skilled in the art will readily understand that... Figure 4 The AR glasses 200 depicted is merely one example of many different devices to which the subject matter of this invention can be applied. For example, embodiments of the invention are not limited to AR devices, but are also applicable to virtual reality devices and mixed reality devices.

[0049] AR glasses 200 include a data processor 402, a display 410, two or more camera devices 408, and additional input / output elements 416. Input / output elements 416 may include a microphone, audio speaker, biometric sensor, additional sensor, or additional display element integrated with the data processor 402. For example, input / output elements 416 may include any of the I / O components containing moving parts, etc.

[0050] Consistent with one example and as described herein, display 410 includes displays for a user's left and right eyes. Each display in AR glasses 200 may include: a forward-facing optical assembly (not shown) comprising a right projector and a right near-eye display; and a forward-facing optical assembly comprising a left projector and a left near-eye display. In some examples, the near-eye display is a waveguide. The waveguide includes reflective or diffractive structures (e.g., gratings and / or optical elements such as mirrors, lenses, or prisms). Light emitted by the right projector encounters the diffractive structure of the waveguide of the right near-eye display, which directs the light toward the user's right eye to provide an image on or in the right optical element, superimposed with a view of the real world seen by the user. Similarly, light emitted by the left projector encounters the diffractive structure of the waveguide of the left near-eye display, which directs the light toward the user's left eye to provide an image on or in the left optical element, superimposed with a view of the real world seen by the user.

[0051] The data processor 402 includes an image processor 406 (e.g., a video processor), a graphics processing unit (GPU) & display driver 448, a tracking processor 440, an interface 412, a low-power circuit system 404, and a high-speed circuit system 420. The components of the data processor 402 are interconnected via a bus 442.

[0052] Interface 412 refers to any source of user commands provided to data processor 402. In one or more examples, interface 412 is a physical button that, when pressed, sends a user input signal from interface 412 to low-power processor 414. Low-power processor 414 may process pressing such a button followed immediately by release as a request to capture a single image, or vice versa. Low-power processor 414 may process pressing such a button for a first time period as a request to capture video data while the button is pressed and to stop video capture when the button is released, wherein the video captured while the button is pressed is stored as a single video file. Alternatively, pressing the button for an extended time period may capture a still image. In some examples, interface 412 may be any mechanical switch or physical interface capable of accepting user input associated with a data request from camera device 408. In other examples, interface 412 may have software components or may be associated with commands received wirelessly from another source (e.g., from client device 428).

[0053] Image processor 406 includes circuitry for receiving signals from camera device 408 and processing those signals into a format suitable for storage in memory 424 or for transmission to client device 428. In one or more examples, image processor 406 (e.g., video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera device 408, and volatile memory used by the microprocessor in operation.

[0054] Low-power circuit system 404 includes a low-power processor 414 and a low-power wireless circuit system 418. These elements of low-power circuit system 404 can be implemented as separate components or as part of a single-chip system on a single IC. Low-power processor 414 includes logic for managing other components of AR glasses 200. As described above, for example, low-power processor 414 can accept user input signals from interface 412. Low-power processor 414 can also be configured to receive input signals or command communications from client device 428 via a low-power wireless connection. Low-power wireless circuit system 418 includes circuit elements for implementing a low-power wireless communication system. Bluetooth™ Smart, also known as Bluetooth™ Low Energy, is a standard implementation of the low-power wireless communication system that can be used to implement low-power wireless circuit system 418. Other low-power communication systems may be used in other examples.

[0055] The high-speed circuit system 420 includes a high-speed processor 422, a memory 424, and a high-speed wireless circuit system 426. The high-speed processor 422 can be any processor capable of managing high-speed communication and operation for any general-purpose computing system used by the data processor 402. The high-speed processor 422 includes processing resources for managing high-speed data transmission over the high-speed wireless connection 434 using the high-speed wireless circuit system 426. In some examples, the high-speed processor 422 executes an operating system such as LINUX or another such operating system. Among other duties, the high-speed processor 422, which executes the software architecture of the data processor 402, manages data transmission with the high-speed wireless circuit system 426. In some examples, the high-speed wireless circuit system 426 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented via the high-speed wireless circuit system 426.

[0056] Memory 424 includes any storage device capable of storing camera data generated by camera device 408 and image processor 406. While memory 424 is shown as integrated with high-speed circuitry 420, in other examples, memory 424 may be a separate, independent element of data processor 402. In some such examples, electrical wiring may provide a connection from image processor 406 or low-power processor 414 to memory 424 via a chip including high-speed processor 422. In other examples, high-speed processor 422 may manage addressing of memory 424, such that low-power processor 414 initiates high-speed processor 422 whenever a read or write operation involving memory 424 is required.

[0057] The tracking processor 440 estimates the pose of the AR glasses 200. For example, the tracking processor 440 uses image data and corresponding inertial data from the camera device 408 and the positioning component, as well as GPS data, to track the position and determine the pose of the AR glasses 200 relative to a reference frame (e.g., a real-world scene). The tracking module 440 continuously collects and uses updated sensor data describing the movement of the AR glasses 200 to determine an updated 3D pose of the AR glasses 200, indicating changes in the relative position and orientation of the AR glasses 200 with respect to physical objects in the real-world environment. The tracking processor 440 allows the AR glasses 200 to visually position virtual objects relative to physical objects within the user's field of view via the display 410.

[0058] When the AR glasses 200 are running in conventional AR mode, the GPU and display driver 438 can use the pose of the AR glasses 200 to generate frames of virtual content or other content to be displayed on the display 410. In this mode, the GPU and display driver 438 generate updated frames of virtual content based on the updated 3D pose of the AR glasses 400, which reflects changes in the position and orientation of the user relative to physical objects in the user's real-world environment.

[0059] One or more functions or operations described herein can also be performed on an application residing on AR glasses 200, client device 428, or remote server 430. Consistent with some examples, AR glasses 200 can operate in a networked system comprising AR glasses 200, client computing device 428, and server 430, which can be communicatively coupled via a network. Client device 428 can be a smartphone, tablet, phablet, laptop, access point, or any other such device capable of connecting to AR glasses 200 using low-power wireless and / or high-speed wireless connections. Client device 428 is connected to server system 430 via a network. The network can include any combination of wired and wireless connections. Server 430 can be one or more computing devices as part of a service or network computing system.

[0060] Software Architecture

[0061] Figure 5This is a block diagram 500 illustrating a software architecture 504 that can be installed on any or more of the devices described herein. The software architecture 504 is supported by hardware such as a machine 502 including a processor 520, memory 526, and I / O components 538. In this example, the software architecture 504 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 504 includes layers such as an operating system 512, libraries 508, frameworks 510, and applications 506. Operationally, application 506 invokes API calls 550 through the software stack and receives messages 552 in response to API calls 550.

[0062] Operating system 512 manages hardware resources and provides public services. Operating system 512 includes, for example, a kernel 514, services 516, and drivers 522. Kernel 514 acts as an abstraction layer between the hardware layer and other software layers. For example, kernel 514 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 516 can provide other public services to other software layers. Drivers 522 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 522 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® low-power drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0063] Library 508 provides low-level public infrastructure used by application 506. Library 508 may include system library 518 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 508 may include API library 524, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering two-dimensional (2D) and three-dimensional (3D) graphical content on a display, and GLMotif for implementing 3D user interfaces), image feature extraction libraries (e.g., OpenIMAJ), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functionality), etc. Library 508 may also include various other libraries 528 to provide many other APIs to application 506.

[0064] Framework 510 provides high-level common infrastructure for use by application 506. For example, framework 510 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Framework 510 can provide a wide range of other APIs that can be used by application 506, some of which may be specific to a particular operating system or platform.

[0065] In the example, application 506 may include home application 536, contact application 530, browser application 532, book reader application 534, location application 542, media application 544, messaging application 546, game application 548, and a variety of other applications such as third-party application 540. Application 506 is a program that performs the functions defined in the program. One or more applications 506 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 540 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or another mobile operating system. In this example, third-party application 540 may activate API calls 550 provided by operating system 512 to facilitate the functions described herein.

[0066] Example

[0067] Example 1 is a device comprising: a processor; a first image sensor; a second image sensor; and a memory storing instructions thereon, the instructions, when executed by the processor, causing the device to perform operations including: acquiring a first image from the first image sensor; acquiring a second image from the second image sensor; processing the first image with an object detector to identify the coordinates of a first region of interest, the first region of interest indicating the location of an object depicted in the first image; processing a region in the first image corresponding to the first region of interest with a landmark detector to determine two-dimensional position data of one or more landmarks associated with the object; and using at least the first image, the two-dimensional position data of the one or more landmarks, and parameters of the device as input to a monocular pose estimator to derive the first position of the one or more landmarks. The method includes: 1) three-dimensional position data; 2) using the three-dimensional position data of the one or more landmarks to determine the coordinates of a second region of interest, the second region of interest indicating the position of the object depicted in a second image; 3) processing the region in the second image corresponding to the second region of interest with the landmark detector to determine two-dimensional position data of one or more landmarks associated with the object; and 4) using triangulation calculations to derive second three-dimensional position data of the one or more landmarks as input to triangulation calculations: i) two-dimensional position data of the one or more landmarks determined from processing the region in the first image corresponding to the first region of interest, ii) the two-dimensional position data of the one or more landmarks determined from processing the region in the second image corresponding to the second region of interest, and iii) parameters of the device.

[0068] In Example 2, the subject of Example 1 includes: a display device; wherein the object is a hand, and the memory stores additional instructions thereon, which, when executed by a processor, cause the device to perform additional operations, including: tracking the position and orientation of the hand using second three-dimensional position data of the one or more landmarks.

[0069] In Example 3, the subject matter of Examples 1 to 2 includes, wherein obtaining first three-dimensional position data of the one or more landmarks using a monocular pose estimator further includes: using a reference metric as input to the monocular pose estimator, the reference metric representing an estimated length or distance between two specific landmarks; or using an estimated size of the object as input to the monocular pose estimator.

[0070] In Example 4, the subject of Example 3 includes the following: the estimated length or distance between two specific landmarks represents the estimated length of a bone having the two specific landmarks as endpoints, the estimated length being derived from second three-dimensional position data.

[0071] In Example 5, the subject matter of Examples 1 to 4 includes determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks by: using a rigid transformation matrix defined for the device to transform the three-dimensional position data of the one or more landmarks from a coordinate system associated with a first image and a first image sensor to a coordinate system associated with a second image and a second image sensor.

[0072] In Example 6, the subject matter of Examples 1 to 5 includes determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks by: calculating the smallest rectangle enclosing all of the one or more landmarks after projecting the landmarks from a coordinate system associated with a first image and a first image sensor to a coordinate system associated with a second image and a second image sensor.

[0073] In Example 7, the subject matter of Examples 1 to 6 includes determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks, which includes applying a scaling factor to the coordinates of the second region of interest to compensate for inaccuracies that may result from the first three-dimensional position data of the one or more landmarks obtained using a monocular pose estimator, the scaling factor enlarging the size of the second region of interest.

[0074] In Example 8, the subject matter of Examples 1 to 7 includes, wherein processing a region in a first image corresponding to a first region of interest with a landmark detector to determine two-dimensional location data of one or more landmarks associated with the object includes identifying a single representative landmark, via which the object can be transformed.

[0075] Example 9 is a computer-implemented method comprising: acquiring a first image from a first image sensor; acquiring a second image from a second image sensor; processing the first image with an object detector to identify the coordinates of a first region of interest, the first region of interest indicating the location of an object depicted in the first image; processing a region in the first image corresponding to the first region of interest with a landmark detector to determine two-dimensional position data of one or more landmarks associated with the object; using at least the first image, the two-dimensional position data of the one or more landmarks, and parameters associated with the first and second image sensors as input to a monocular pose estimator, deriving first three-dimensional position data of the one or more landmarks using the monocular pose estimator; and using the three-dimensional position data of the one or more landmarks... The coordinates of a second region of interest are determined, the second region of interest indicating the location of the object depicted in the second image; a landmark detector is used to process the region in the second image corresponding to the second region of interest to determine two-dimensional position data of one or more landmarks associated with the object; and a second three-dimensional position data of the one or more landmarks is obtained by triangulation calculation using the following as inputs: i) two-dimensional position data of the one or more landmarks determined from the region in the first image corresponding to the first region of interest, ii) two-dimensional position data of the one or more landmarks determined from the region in the second image corresponding to the second region of interest, and iii) parameters associated with the first image sensor and the second image sensor.

[0076] In Example 10, the subject of Example 9 includes: using second three-dimensional position data of the one or more landmarks to track the position and orientation of a hand, wherein the object is the hand.

[0077] In Example 11, the subject matter of Examples 9 and 10 includes, wherein obtaining first three-dimensional position data of the one or more landmarks using a monocular pose estimator further includes: using a reference metric as input to the monocular pose estimator, the reference metric representing an estimated length or distance between two specific landmarks; or using an estimated size of the object as input to the monocular pose estimator.

[0078] In Example 12, the subject of Example 11 includes the following: the estimated length or distance between two specific landmarks represents the estimated length of a bone having the two specific landmarks as endpoints, the estimated length being derived from second three-dimensional position data.

[0079] In Example 13, the subject matter of Examples 9 to 12 includes determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks by: using a rigid transformation matrix defined for a first image sensor and a second image sensor to transform the three-dimensional position data of the one or more landmarks from a coordinate system associated with the first image and the first image sensor to a coordinate system associated with the second image and the second image sensor.

[0080] In Example 14, the subject matter of Examples 9 to 13 includes, wherein determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks includes: calculating the smallest rectangle enclosing all of the one or more landmarks after projecting the landmarks from a coordinate system associated with a first image and a first image sensor to a coordinate system associated with a second image and a second image sensor.

[0081] In Example 15, the subject matter of Examples 9 to 14 includes determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks by applying a scaling factor to the coordinates of the second region of interest to compensate for inaccuracies that may result from the first three-dimensional position data of the one or more landmarks obtained using a monocular pose estimator. The scaling factor will enlarge the size of the second region of interest.

[0082] In Example 16, the subject matter of Examples 9 to 15 includes, wherein processing a region in a first image corresponding to a first region of interest with a landmark detector to determine two-dimensional location data of one or more landmarks associated with the object includes identifying a single representative landmark, via which the object can be transformed.

[0083] Example 17 is a system comprising: means for acquiring a first image; means for acquiring a second image; means for processing the first image to identify the coordinates of a first region of interest, the first region of interest indicating the location of an object depicted in the first image; means for processing a region in the first image corresponding to the first region of interest to determine two-dimensional position data of one or more landmarks associated with the object; means for deriving first three-dimensional position data of the one or more landmarks using at least the first image, the two-dimensional position data of the one or more landmarks, and parameters of the system as input; means for determining the coordinates of a second region of interest using the three-dimensional position data of the one or more landmarks, the second region of interest indicating the location of the object depicted in the second image; means for processing a region in the second image corresponding to the second region of interest to determine two-dimensional position data of one or more landmarks associated with the object; and means for deriving second three-dimensional position data of the one or more landmarks using the following as input: i) two-dimensional position data of the one or more landmarks from the first image, ii) two-dimensional position data of the one or more landmarks from the second image, and iii) parameters of the system.

[0084] In Example 18, the subject of Example 17 includes: an apparatus for tracking the position and orientation of a hand using second three-dimensional position data of the one or more landmarks, wherein the object is a hand.

[0085] In Example 19, the subject matter of Examples 17 and 18 includes, wherein the means for deriving first three-dimensional position data of the one or more landmarks further includes: means for using a reference metric as input, the reference metric representing an estimated length or distance between two particular landmarks; or means for using an estimated size of the object as input.

[0086] In Example 20, the subject of Example 19 includes the following: the reference metric represents the estimated length of a bone having the two specific landmarks as endpoints, the estimated length being derived from second three-dimensional position data.

[0087] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform operations to implement any one of Examples 1 to 20.

[0088] Example 22 is an apparatus that includes means for implementing any one of Examples 1 to 20.

[0089] Example 23 is a system for implementing any one of Examples 1 through 20.

[0090] Glossary

[0091] "Carrier signal" refers to any intangible medium, such as a medium capable of storing, encoding, or carrying instructions to be executed by a machine and including digital or analog communication signals, or other intangible medium that facilitates the transmission of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0092] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, multiple laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0093] "Communications network" refers to one or more parts of a network, such as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0094] A “component” refers to, for example, a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that implement specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and part of a program that typically performs a related function. A component can constitute a software component (e.g., code contained on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Hardware components may also include programmable logic or circuitry systems temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine) that is uniquely tailored to perform the configured functionality and is no longer a general-purpose processor. It will be appreciated that the decision to implement a hardware component mechanically in a dedicated and permanently configured circuitry system or in a temporarily configured (e.g., software-configured) circuitry system may be made for cost and time considerations. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider examples of hardware components being temporarily configured (e.g., programmed), without needing to configure or instantiate each of the hardware components at any given time. For example, in the case where a hardware component includes a general-purpose processor configured by software to be a dedicated processor, the general-purpose processor may be configured at different times as its own different dedicated processors (e.g., including different hardware components). For example, software accordingly configures one or more specific processors to constitute a particular hardware component at one time and different hardware components at different times. Hardware components can provide information to other hardware components and can receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components, also referred to as “computer-implemented.” Furthermore, one or more processors can operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not only residing within a single machine but also deployed across multiple machines. In some examples, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component may be distributed across several geographic locations.

[0095] "Computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Therefore, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and may be used interchangeably in this disclosure.

[0096] A "brief message" is a message that is accessible for a limited time, such as a short period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0097] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0098] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0099] "Signal medium" means, for example, any intangible medium capable of storing, encoding, or carrying machine-executable instructions and including digital or analog communication signals, or other intangible media that facilitates the transmission of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0100] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user and that the user interacts with to perform actions or interactions (including interactions with other users or computer systems).

Claims

1. An apparatus comprising: processor; First image sensor; Second image sensor; as well as A memory storing instructions that, when executed by the processor, cause the device to perform operations, including: A first image is obtained from a first image sensor; A second image is obtained from a second image sensor; The first image is processed using an object detector to identify the coordinates of a first region of interest, the first region of interest indicating the location of an object depicted in the first image; The region in the first image corresponding to the first region of interest is processed by a landmark detector to determine two-dimensional (2-D) location data of one or more landmarks associated with the object; Using at least a first image, 2-D position data of the one or more landmarks, and parameters of the device as input to a monocular pose estimator, the monocular pose estimator is used to derive first three-dimensional (3-D) position data of the one or more landmarks; The coordinates of a second region of interest are determined using the 3-D position data of one or more landmarks, the second region of interest indicating the position of the object depicted in the second image; The region in the second image corresponding to the second region of interest is processed using a landmark detector to determine 2D location data of one or more landmarks associated with the object; and The following are used as inputs to the triangulation calculation to obtain the second 3D position data of the one or more landmarks: i) 2D position data of the one or more landmarks determined from the region corresponding to the first region of interest in the first image, ii) 2D position data of the one or more landmarks determined from the region corresponding to the second region of interest in the second image, and iii) parameters of the device.

2. The device according to claim 1, further comprising: Display devices; The object is a hand, and the memory stores additional instructions thereon that, when executed by the processor, cause the device to perform additional operations, including: The second 3D position data of the one or more landmarks are used to track the position and orientation of the hand.

3. The device according to claim 1, wherein, The process of obtaining the first 3D position data of the one or more landmarks using a monocular pose estimator also includes: A reference metric is used as input to the monocular pose estimator, where the reference metric represents the estimated length or distance between two specific landmarks; or The estimated dimensions of the object are used as input to the monocular pose estimator.

4. The device according to claim 3, wherein, The estimated length or distance between two specific landmarks represents the estimated length of a bone with the two specific landmarks as endpoints, the estimated length being derived from second 3-D location data.

5. The device according to claim 1, wherein, Determining the coordinates of the second region of interest using the 3-D position data of one or more of the landmarks includes: The 3-D position data of the one or more landmarks are transformed from the coordinate system associated with the first image and the first image sensor to the coordinate system associated with the second image and the second image sensor using a rigid transformation matrix defined for the device.

6. The device according to claim 1, wherein, Determining the coordinates of the second region of interest using the 3-D position data of one or more of the landmarks includes: After projecting the landmarks from the coordinate system associated with the first image and the first image sensor to the coordinate system associated with the second image and the second image sensor, calculate the smallest rectangle that encloses all one or more of the landmarks.

7. The device according to claim 1, wherein, Determining the coordinates of the second region of interest using the 3-D position data of one or more of the landmarks includes: To compensate for inaccuracies that may result from the use of a monocular pose estimator to obtain the first 3D position data of the one or more landmarks, a scaling factor is applied to the coordinates of the second region of interest, which enlarges the size of the second region of interest.

8. The device according to claim 1, wherein, Processing a region in a first image corresponding to a first region of interest using a landmark detector to determine 2-D location data of one or more landmarks associated with the object includes: identifying a single representative landmark, via which the object can be transformed.

9. A computer-implemented method, comprising: A first image is obtained from a first image sensor; A second image is obtained from a second image sensor; The first image is processed using an object detector to identify the coordinates of a first region of interest, the first region of interest indicating the location of an object depicted in the first image; The region in the first image corresponding to the first region of interest is processed by a landmark detector to determine two-dimensional (2-D) location data of one or more landmarks associated with the object; Using at least a first image, 2-D position data of the one or more landmarks, and parameters associated with a first image sensor and a second image sensor as input to a monocular pose estimator, the monocular pose estimator is used to derive first 3-D position data of the one or more landmarks. The coordinates of a second region of interest are determined using the 3-D position data of one or more landmarks, the second region of interest indicating the position of the object depicted in the second image; The region in the second image corresponding to the second region of interest is processed using a landmark detector to determine 2-D location data of one or more landmarks associated with the object; as well as The following are used as inputs to the triangulation calculations to derive the second 3D position data of the one or more landmarks: i) 2D position data of the one or more landmarks determined from the region corresponding to the first region of interest in the first image, ii) 2D position data of the one or more landmarks determined from the region corresponding to the second region of interest in the second image, and iii) parameters associated with the first image sensor and the second image sensor.

10. The computer-implemented method according to claim 9, further comprising: The position and orientation of the hand are tracked using second 3D position data from one or more of the landmarks, wherein the object is the hand.

11. The computer-implemented method according to claim 9, wherein, The process of obtaining the first 3D position data of the one or more landmarks using a monocular pose estimator also includes: A reference metric is used as input to the monocular pose estimator, where the reference metric represents the estimated length or distance between two specific landmarks; or The estimated dimensions of the object are used as input to the monocular pose estimator.

12. The computer-implemented method according to claim 11, wherein, The estimated length or distance between two specific landmarks represents the estimated length of a bone with the two specific landmarks as endpoints, the estimated length being derived from second 3-D location data.

13. The computer-implemented method according to claim 9, wherein, Determining the coordinates of the second region of interest using the 3-D position data of one or more of the landmarks includes: The 3-D position data of the one or more landmarks are transformed from the coordinate system associated with the first image and the first image sensor to the coordinate system associated with the second image and the second image sensor using a rigid transformation matrix defined for the first image sensor and the second image sensor.

14. The computer-implemented method according to claim 9, wherein, Determining the coordinates of the second region of interest using the 3-D position data of one or more of the landmarks includes: After projecting the landmarks from the coordinate system associated with the first image and the first image sensor to the coordinate system associated with the second image and the second image sensor, calculate the smallest rectangle that encloses all one or more of the landmarks.

15. The computer-implemented method according to claim 9, wherein, Determining the coordinates of the second region of interest using the 3-D position data of one or more of the landmarks includes: To compensate for inaccuracies that may result from the use of a monocular pose estimator to obtain the first 3D position data of the one or more landmarks, a scaling factor is applied to the coordinates of the second region of interest, which enlarges the size of the second region of interest.

16. The computer-implemented method according to claim 9, wherein, Processing a region in a first image corresponding to a first region of interest using a landmark detector to determine 2-D location data of one or more landmarks associated with the object includes: identifying a single representative landmark, via which the object can be transformed.

17. A system comprising: A means for acquiring a first image; A device for obtaining a second image; A means for processing a first image to identify the coordinates of a first region of interest, the first region of interest indicating the location of an object depicted in the first image; A means for processing a region in a first image corresponding to a first region of interest to determine two-dimensional (2-D) location data of one or more landmarks associated with the object; A means for deriving first 3D position data of one or more landmarks using at least a first image, 2D position data of the one or more landmarks, and parameters of the system as input; A means for determining the coordinates of a second region of interest using 3-D position data of one or more landmarks, the second region of interest indicating the position of the object depicted in a second image; A means for processing a region in a second image corresponding to a second region of interest to determine 2-D location data of one or more landmarks associated with the object; as well as A means for obtaining second 3D position data of the one or more landmarks using the following as inputs: i) 2D position data of the one or more landmarks from a first image, ii) 2D position data of the one or more landmarks from a second image, and iii) parameters of the system.

18. The system of claim 17, further comprising: A means for tracking the position and orientation of a hand using second 3-D position data of one or more landmarks, wherein the object is the hand.

19. The system according to claim 17, wherein, The apparatus for obtaining the first 3D position data of the one or more landmarks further includes: A device for using a reference metric as input, the reference metric representing an estimated length or distance between two specific landmarks; or A means for using the estimated size of the object as input.

20. The system according to claim 19, wherein, The reference metric represents the estimated length of the bone with the two specific landmarks as endpoints, the estimated length being derived from second 3-D location data.

Images