3D space sculpture using hands for object capture

Abstract

A method for engraving a 3D space using hand tracking is described. In one aspect, a method includes acquiring a first frame from a camera of a display device, tracking hand pixels corresponding to one or more user hands depicted in the first frame using a hand tracking algorithm running at the display device, detecting a depth of the hand pixels using a sensor of the display device, identifying a 3D region based on the depth of the hand pixels, and displaying the 3D region on the display device. And applying a 3D reconstruction engine to the 3D region.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to Greek patent application serial number 20220100720 filed on September 1, 2022 and U.S. patent application serial number 17 / 973,167 filed on October 25, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein generally relates to 3D model systems. Specifically, the present disclosure proposes systems and methods for constraining 3D space using hand motions. Background Art

[0004] Acquiring the 3D geometry of real-world objects is generally known in the art. In computer vision, given a set of 2D images of a scene, image-based scene reconstruction techniques are used to create a 3D model of the scene. In one 3D reconstruction technique, the system uses multi-view stereo to analyze the images of the scene. These image-based methods can construct 3D models relatively simply and inexpensively using standard imaging hardware (such as consumer-grade digital cameras). Due to advances in image sensors, these image-based methods can provide color information of the scene and provide high-resolution scans. Most multi-view stereo methods filter, smooth or denoise the reconstructed depth map, and usually these steps are integrated into the depth estimation stage and converted into a (global) optimization problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0006] Figure 1 is a block diagram illustrating a network environment for operating an AR display device according to an example embodiment.

[0007] Figure 2 is a block diagram illustrating an AR display device according to one example embodiment.

[0008] Figure 3 is a block diagram illustrating a tracking system according to one example implementation.

[0009] Figure 4 is a block diagram illustrating a 3D model engine according to an example implementation.

[0010] Figure 5 is a flow chart illustrating a method for identifying a 3D region according to an example embodiment.

[0011] Figure 6 Routine 600 is shown according to one example implementation.

[0012] Figure 7 An example of a sequence of hand movements is shown according to an example embodiment.

[0013] Figure 8 A top view of hand detection according to an example implementation is shown.

[0014] Fig. 9 An example of a 3D zone according to one example implementation is shown.

[0015] Fig.10 A head wearable device 1000 is shown according to an example embodiment.

[0016] Fig.11 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to an example implementation.

[0017] Fig.12 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein according to an example embodiment. DETAILED DESCRIPTION

[0018] The following description describes the system, method, technology, instruction sequence and computing machine program product of the example implementation of this theme. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various implementations of this theme. However, it will be apparent to those skilled in the art that the implementation of this theme can be put into practice without some or other details in these specific details. Examples represent possible variations only. Unless otherwise clearly stated, structures (e.g., structural components, such as modules) are optional and can be combined or subdivided, and operations (e.g., operations in processes, algorithms or other functions) can be changed in order or combined or subdivided.

[0019] Wearable devices such as smart glasses can be used to estimate 3D models of physical objects based on point cloud data generated by sensors in the wearable device. The wearable device can adopt a multi-view stereo method to build a 3D model by first calculating the camera pose and then estimating the depth map of all views by finding corresponding pixels between views and triangulating the depth. Under this method, all pixels are then projected into 3D space to obtain a point cloud, from which a surface mesh can be extracted using point cloud meshing techniques. However, a disadvantage of the method described above is that processing all pixels in an image to generate a point cloud consumes limited processing resources, especially on mobile devices such as smartphones or mixed reality glasses.

[0020] The present application describes a method for tracking the hands of a user of a wearable device to carve out a 3D space for a 3D reconstruction engine to focus on. In other words, the 3D reconstruction engine does not consider areas outside the carved 3D space. In one example embodiment, a user of the wearable device walks to a nearby physical object and moves his / her hand in front of, behind, and to the side of the physical object. The wearable device runs a hand tracking algorithm on an image generated by the wearable device. The hand tracking algorithm marks / segments pixels in the image that belong to the hand. The wearable device determines the depth of these pixels based on: (1) a stereo camera or a depth camera of the wearable device, or (2) a match of the contours of the tracked hand in the two images. These pixels indicate unoccupied 3D space between the camera of the wearable device and the hand. The currently described method results in lower power consumption of the wearable device when generating / identifying a 3D envelope / shell of a physical object to be 3D reconstructed, because the 3D reconstruction engine will only need to resolve the occupancy of voxels in a smaller area rather than the entire 3D space depicted in the image. Furthermore, unlike background subtraction methods that are only applicable to single foreground objects, with the currently described method, users can sculpt the shell of any physical object depicted in a cluttered scene.

[0021] In one example implementation, a method for sculpting a 3D space using hand tracking for 3D capture of a physical object is described. In one aspect, the method includes: accessing a first frame from a camera of a display device; using a hand tracking algorithm running at the display device to track hand pixels corresponding to one or more user hands depicted in the first frame; using a sensor of the display device to detect the depth of the hand pixels; identifying a 3D region based on the depth of the hand pixels; and applying a 3D reconstruction engine to the 3D region.

[0022] Thus, one or more of the methods described herein help to solve the technical problem of limited computing resources on mobile devices. The currently described method provides an improvement in the operation of the functions of a computer by reducing the power consumption associated with 3D capture of physical objects using a camera device of a mobile device. Thus, one or more of the methods described herein can eliminate the need for certain workloads or computing resources. Examples of such computing resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0023] Figure 1 1 is a network diagram illustrating a network environment 100 suitable for operating a display device 108 according to some example embodiments. The network environment 100 includes a display device 108 and a server 110 communicatively coupled to each other via a network 104. The display device 108 and the server 110 may each be configured in whole or in part as described below with respect to FIG. Fig.12 The server 110 may be implemented in the computer system described. The server 110 may be part of a network-based system. For example, the network-based system may be or include a cloud-based server system that provides additional information, such as virtual content (e.g., a three-dimensional model of a virtual object), to the display device 108.

[0024] User 106 operates display device 108. User 106 may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with display device 108), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). User 106 is not part of network environment 100 but is associated with display device 108.

[0025] The display device 108 may include a computing device with a display, such as a smart phone, a tablet computer, or a wearable computing device (e.g., a watch or glasses). The computing device may be handheld or may be removably mounted to the head of the user 106. In one example, the display may be a screen that displays content captured using a camera of the display device 108. In another example, the display of the display device 108 may be transparent, such as in a lens of wearable computing glasses. In another example implementation, the display may be opaque and can be worn by the user 106 to cover the field of view of the user 106.

[0026] The display device 108 includes a tracking system (not shown). The tracking system tracks the posture (e.g., position and orientation) of the display device 108 relative to the real world environment 102 using optical sensors (e.g., depth-enabled 3D cameras, image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to determine the position of the display device 108 in the real world environment 102. In another example embodiment, the tracking system tracks the posture of the hand 114 in the video frame captured by the camera. For example, the tracking system identifies the hand 114 and tracks the movement of the hand 114. The user 106 can move his / her hand 114 in front of, behind, and to the side of the physical object 112.

[0027] The display device 108 includes a 3D reconstruction engine (not shown) configured to construct a 3D model of the physical object 112 based on the depth of the tracked hand 114. The display device 108 can use the 3D model to identify the physical object 112 and use the 3D model to operate the application. For example, the application may include an AR (augmented reality) application configured to provide the user 106 with an experience triggered by the physical object 112. For example, the user 106 can instruct the camera of the display device 108 to capture an image of the physical object 112. Then, the display device 108 tracks the physical object 112 and accesses the virtual content associated with the physical object 112. In one example, the AR application generates additional information corresponding to the 3D model of the physical object 112, and presents the additional information in the display of the display device 108. If the 3D model is not locally identified at the display device 108, the display device 108 downloads additional information (e.g., other 3D models) from the database of the server 110 via the network 104.

[0028] In one example implementation, the server 110 receives the depth data of the carved 3D space and applies a 3D reconstruction engine to the depth data of the carved 3D space to construct a 3D model of the physical object 112. The server 110 may also identify virtual content (e.g., virtual objects) based on the 3D model of the physical object 112. The server 110 transmits the virtual objects back to the display device 108. Object recognition, tracking, and AR rendering may be performed on the display device 108, the server 110, or a combination between the display device 108 and the server 110.

[0029] Figure 1 Any of the machines, databases, or devices shown in the can be implemented in a general-purpose computer that is modified (e.g., configured or programmed) by software to become a special-purpose computer to perform one or more of the functions described herein for the machine, database, or device. Figure 5 to Figure 6A computer system capable of implementing any one or more of the methods described herein is discussed. As used herein, a "database" is a data storage resource and may store data structured as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database), a triple store, a hierarchical data store, or any suitable combination thereof. In addition, Figure 1 Any two or more of the machines, databases, or devices shown in the may be combined into a single machine, and the functionality described herein for any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

[0030] The network 104 may be any network that enables communication between or among machines (e.g., server 110), databases, and devices (e.g., display device 108). Thus, the network 104 may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network 104 may include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0031] Figure 2 2 is a block diagram illustrating modules (e.g., components) of a display device 108 according to some example embodiments. The display device 108 includes a sensor 202, a display 204, a processor 208, a rendering system 224, and a storage device 206. Examples of the display device 108 include a head mounted device, a wearable computing device, a desktop computer, a vehicle mounted computer, a tablet computer, a navigation device, a portable media device, or a smart phone.

[0032] Sensors 202 include, for example, optical sensors 214 (e.g., stereo cameras, cameras such as color cameras, thermal cameras, depth sensors, and one or more grayscale global shutter tracking cameras) and inertial sensors 216 (e.g., gyroscopes, accelerometers). Other examples of sensors 202 include proximity sensors or position sensors (e.g., near field communication, GPS, Bluetooth, Wi-Fi), audio sensors (e.g., microphones), or any suitable combination thereof. Note that the sensors 202 described herein are for illustrative purposes, and thus the sensors 202 are not limited to the sensors described above.

[0033] The display 204 includes a screen or monitor configured to display images generated by the processor 208. In one example implementation, the display 204 may be transparent or translucent so that the user 106 can see through the display 204 (in an AR use case). In another example, the display 204 (e.g., an LCOS display) presents each frame of the virtual content in multiple renderings.

[0034] The processor 208 operates the AR application 210, the 3D model engine 226, and the tracking system 212. The tracking system 212 uses computer vision to detect and track the hand 114 and the physical object 112. The 3D model engine 226 constructs a 3D model of the physical object 112 and stores the 3D model data 228 in the storage device 206. The AR application 210 retrieves virtual content based on the 3D model of the physical object 112. The AR rendering system 224 renders the virtual object in the display 204. In the AR scene, the AR application 210 generates annotations / virtual content that are superimposed on the image of the physical object 112 captured by the optical sensor 214 (e.g., superimposed on it, or otherwise displayed simultaneously with it, and appearing to be anchored to it). The annotations / virtual content can be manipulated by changing the posture of the physical object 112 relative to the optical sensor 214 (e.g., its physical position, orientation, or both). Similarly, the visualization of the annotations / virtual content can be manipulated by adjusting the posture of the display device 108 relative to the physical object 112.

[0035] The tracking system 212 estimates the pose of the display device 108 and / or the pose of the physical objects 112. In one example, the tracking system 212 uses image data and corresponding inertial data from the optical sensor 214 and the inertial sensor 216 to track the position and pose of the display device 108 relative to a reference frame (e.g., the real-world environment 102). In one example, the tracking system 212 uses the sensor data to determine a three-dimensional pose of the display device 108. The three-dimensional pose is a determined orientation and position of the display device 108 relative to the user's real-world environment 102. For example, the display device 108 can use an image of the user's real-world environment 102 and other sensor data to identify the relative position and orientation of the display device 108 and the physical objects in the real-world environment 102 surrounding the display device 108. The tracking system 212 continuously collects and uses updated sensor data describing the movement of the display device 108 to determine an updated three-dimensional pose of the display device 108, the updated three-dimensional pose indicating a change in the relative position and orientation of the display device 108 and the physical objects in the real-world environment 102. Tracking system 212 provides the three-dimensional pose of display device 108 to rendering system 224 .

[0036] The rendering system 224 includes a graphics processing unit 218 and a display controller 220. The graphics processing unit 218 includes a rendering engine (not shown) configured to render frames of a 3D model of a virtual object based on the virtual content provided by the AR application 210 and the gesture of the display device 108. In other words, the graphics processing unit 218 uses the three-dimensional gesture of the display device 108 to generate a frame of virtual content to be presented on the display 204. For example, the graphics processing unit 218 uses the three-dimensional gesture to render the frame of the virtual content so that the virtual content is presented in the display 204 with an appropriate orientation and position to appropriately enhance the realism of the user. As an example, the graphics processing unit 218 can use the three-dimensional gesture data to render the frame of the virtual content so that when presented on the display 204, the virtual content appears to be anchored to the physical object 112 in the user's real world environment 102. The graphics processing unit 218 generates updated frames of virtual content based on the updated three-dimensional pose of the display device 108 that reflect the changes in the position and orientation of the user 106 relative to the physical objects 112 in the user's real-world environment 102 .

[0037] The graphics processing unit 218 transmits the rendered frame to the display controller 220. The display controller 220 is positioned as an intermediary between the graphics processing unit 218 and the display 204, receiving image data (e.g., annotated rendered frames) from the graphics processing unit 218 and providing the annotated rendered frames to the display 204.

[0038] Storage device 206 stores virtual object content 222 and 3D model data 228. Virtual object content 222 includes, for example, a database of visual references (e.g., images, QR codes) and corresponding virtual content (e.g., three-dimensional models of virtual objects). 3D model data 228 is generated by 3D model engine 226.

[0039] Any one or more modules in the modules described herein can be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein can configure a processor to perform the operation described herein for the module. In addition, any two or more modules in these modules can be combined into a single module, and the function described herein for a single module can be subdivided among multiple modules. In addition, according to various example embodiments, the modules described herein as implemented in a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0040] Figure 3 A tracking system 212 is shown according to an example implementation. The tracking system 212 includes a visual tracking system 308 and a hand tracking system 310, for example.

[0041] The visual tracking system 308 includes an inertial sensor module 302, an optical sensor module 304, and a pose estimation module 306. The inertial sensor module 302 accesses inertial sensor data from the inertial sensor 216. The optical sensor module 304 accesses optical sensor data from the optical sensor 214.

[0042] The pose estimation module 306 determines the pose (e.g., location, position, orientation) of the display device 108 relative to a reference frame (e.g., the real-world environment 102). In one example implementation, the pose estimation module 306 estimates the pose of the display device 108 based on images captured by the optical sensor 214 (via the optical sensor module 304) and a 3D map of feature points from inertial sensor data captured by the inertial sensor 216 (via the inertial sensor module 302).

[0043] In one example, the pose estimation module 306 includes an algorithm that combines inertial information from the inertial sensors 216 and image information from the optical sensors 214, which are coupled to a rigid platform (e.g., the display device 108) or a rig. The rig may consist of multiple cameras (with non-overlapping (distributed aperture) or overlapping (stereo or more) fields of view) mounted on a rigid platform with an inertial measurement unit (also referred to as an IMU) (e.g., the rig may thus have at least one IMU and at least one camera).

[0044] The hand tracking system 310 runs a computer vision algorithm (e.g., a hand tracking algorithm) to detect and track the location of a hand depicted in a frame captured by the optical sensor 214. In one example, the hand tracking system 310 detects and identifies pixels corresponding to the hand 114 of the user 106 in an image captured using the optical sensor 214. The hand tracking system 310 labels and segments the pixels in the image that belong to the hand 114.

[0045] Figure 4 is a block diagram illustrating the 3D model engine 226 according to one example implementation. The 3D model engine 226 includes a hand tracking interface 402, a pixel depth module 404, a 3D region carving module 406, and a 3D reconstruction engine 408.

[0046] The hand tracking interface 402 communicates with the hand tracking system 310 and receives data identifying pixels corresponding to the hand 114 in the image generated by the optical sensor 214. In one example, the hand tracking interface 402 identifies pixels that are marked for the hand 114. In another example, the hand tracking interface 402 accesses segmented pixels corresponding to the hand 114.

[0047] The pixel depth module 404 determines the depth of the labeled / segmented pixels identified from the hand tracking interface 402. In one example, the pixel depth module 404 determines the depth of the pixel by using techniques such as (1) a stereo or depth camera and (2) matching of the outline of the tracked hand 114 in both images.

[0048] The 3D zone carving module 406 identifies a 3D space based on the depth data generated by the pixel depth module 404. For example, the 3D space corresponds to an unoccupied 3D space between the optical sensor 214 and the hand 114. In another example, the 3D zone carving module 406 carves out the 3D space including the 3D envelope / shell of the physical object 112 based on the movement of the hand 114. For example, the user 106 moves his / her hand 114 in front of, behind, and beside the physical object 112. As the user 106 moves his / her hand around (in front of, behind, and beside) the physical object 112, the 3D zone carving module 406 detects the 3D space based on the depth of the hand 114.

[0049] The 3D reconstruction engine 408 may be configured to construct or reconstruct a 3D model using point cloud data from the 3D space identified by the 3D region carving module 406. The 3D reconstruction performed by the 3D reconstruction engine 408 may employ any image-based technique that reconstructs scene geometry in the form of a depth map and any surface reconstruction technique that takes a point set as input.

[0050] Figure 5 is a flow chart illustrating a method for identifying a 3D region according to an example embodiment. The operations in routine 500 may be performed by tracking system 212 and 3D model engine 226 using the method described above with respect to Figure 2 , Figure 3 and Figure 4 Therefore, routine 500 is described by way of example with reference to tracking system 212 and 3D model engine 226. However, it should be understood that at least some of the operations of routine 500 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0051] In block 502, the hand tracking system 310 runs a hand tracking algorithm to track the hand 114 depicted in the image generated by the optical sensor 214. In block 504, the hand tracking system 310 identifies pixels corresponding to the hand 114. In block 506, the pixel depth module 404 identifies the depth of the pixels corresponding to the hand 114. In block 508, the 3D zone carving module 406 identifies a 3D zone between the optical sensor 214 and the hand 114. In block 510, the 3D reconstruction engine 408 performs a 3D volume reconstruction of the physical object 112 located in the 3D zone.

[0052] It should be noted that other embodiments can use different sorting, additional or less operations and different nomenclature or terminology to complete similar functions. In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein are selected to illustrate some operating principles in a simplified form.

[0053] Figure 6 A routine 600 according to an example implementation is shown. In block 602, the routine 600 accesses a first frame from a camera of a display device. In block 604, the routine 600 uses a hand tracking algorithm running at the display device to track hand pixels corresponding to one or more user hands depicted in the first frame. In block 606, the routine 600 uses a sensor of the display device to detect the depth of the hand pixels. In block 608, the routine 600 identifies a 3D region based on the depth of the hand pixels. In block 610, the routine 600 applies a 3D reconstruction engine to the 3D region.

[0054] Figure 7 An example of a sequence 702 of hand movements according to an example implementation is shown. Image 704 depicts a left hand 710 and a right hand 712 (of the user 106) near a physical object 112. The physical object 112 is located between the left hand 710 and the right hand 712. Image 706 depicts the right hand 712 in front of the physical object 112. Image 708 depicts the left hand 710 behind / beside the physical object 112.

[0055] Figure 8 A top view of hand detection according to an example implementation is shown. The acquired image 802 depicts the physical object 112 between the left hand 710 and the right hand 712. The top view 804 depicts the physical object 112 between the left hand 710 and the right hand 712. The top view 804 also shows the free, unoccupied 3D space 808 between the camera 806 and the left / right hand 710 / 712.

[0056] Fig. 9 An example of a 3D area (carved 3D space 902) is shown according to an example implementation. The carved 3D space 902 is constructed / identified based on depth data of the motion of the hand 114. The physical object 112 is located within the carved 3D space 902.

[0057] Fig.10 A head wearable device 1000 is shown according to an example embodiment. Fig.10 A perspective view of a head wearable device 1000 is shown according to an example implementation. In some examples, the display device 108 can be the head wearable device 1000.

[0058] exist Fig.10, the head wearable device 1000 is a pair of glasses. In some embodiments, the head wearable device 1000 can be sunglasses or goggles. Some embodiments may include one or more wearable devices, such as a pendant with an integrated camera device that is integrated, communicates with, or is coupled to the head wearable device 1000 or the display device 108. Any desired wearable device may be used in conjunction with embodiments of the present disclosure, such as a watch, headphones, wristbands, earbuds, clothing (e.g., a hat or jacket with integrated electronics), a clip-on electronic device, or any other wearable device. It should be understood that, although not shown, one or more portions of the system included in the head wearable device 1000 may be included in the display device 108 that can be used in conjunction with the head wearable device 1000.

[0059] exist Fig.10 In the embodiment, the head wearable device 1000 is a pair of glasses including a frame 1010, which includes a frame (or frame), and the frame (or frame) is respectively coupled to two handles (or temples) via hinges and / or end pieces. The frame of the frame 1010 carries or accommodates a pair of lenses (e.g., lens 1012 and lens 1014). The frame 1010 includes a first (e.g., right) side coupled to the first handle and a second (e.g., left) side coupled to the second handle. The first side is opposite to the second side of the frame 1010.

[0060] The head wearable device 1000 also includes a camera lens (e.g., camera lens 1006, camera lens 1008) and one or more proximity sensors (proximity sensor 1016, proximity sensor 1018). Camera lens 1006 and camera lens 1008 may be perspective camera lenses or non-perspective camera lenses. The non-perspective camera lens may be, for example, a fisheye lens, a wide-angle lens, an omnidirectional lens, etc. The image sensor captures digital video through camera lens 1006 and camera lens 1008. The image may also be a still image frame or a video including multiple still image frames. The camera module may be coupled to frame 1010. As shown in FIG. Fig.10 As shown in , frame 1010 is coupled to camera lens 1006 and camera lens 1008 so that the camera lenses (e.g., camera lens 1006, camera lens 1008) face forward. Camera lens 1006 and camera lens 1008 may be perpendicular to lens 1012 and lens 1014. The camera module may include dual front cameras separated by the width of frame 1010 or the width of the head of the user of head wearable device 1000.

[0061] exist Fig.10, two handles (or temples) are coupled to microphone housing 1002 and microphone housing 1004, respectively. The first handle and the second handle are coupled to opposite sides of the frame 1010 of the head wearable device 1000. The first handle is coupled to the first microphone housing 1002, and the second handle is coupled to the second microphone housing 1004. The microphone housing 1002 and the microphone housing 1004 can be coupled to the handles between the position of the frame 1010 and the position of the temple tip. When the user wears the head wearable device 1000, the microphone housing 1002 and the microphone housing 1004 can be located on either side of the temple of the user.

[0062] like Fig.10 As shown in , microphone housing 1002 and microphone housing 1004 enclose multiple microphones (not shown). A microphone is an air interface sound pickup device that converts sound into an electrical signal. More specifically, a microphone is a transducer that converts sound pressure into an electrical signal (e.g., an acoustic signal). The microphone may be a digital or analog micro-electromechanical system (MEMS) microphone. The acoustic signal generated by the microphone may be a pulse density modulation (PDM) signal.

[0063] Fig.11 1100 is a block diagram illustrating a software architecture 1104 that may be installed on any one or more of the devices described herein. The software architecture 1104 is supported by hardware such as a machine 1102, which includes a processor 1120, a memory 1126, and an I / O component 1138. In this example, the software architecture 1104 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 1104 includes layers such as an operating system 1112, a library 1110, a framework 1108, and an application 1106. In operation, the application 1106 invokes an API call 1150 through the software stack and receives a message 1152 in response to the API call 1150.

[0064] The operating system 1112 manages hardware resources and provides public services. The operating system 1112 includes, for example, a kernel 1114, services 1116, and drivers 1122. The kernel 1114 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1114 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1116 can provide other public services to other software layers. Drivers 1122 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1122 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0065] The libraries 1110 provide a low-level common infrastructure used by the applications 1106. The libraries 1110 may include a system library 1118 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1110 may include an API library 1124, such as a media library (e.g., a library for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presentation in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1110 may also include various other libraries 1128 to provide many other APIs to the application 1106 .

[0066] The framework 1108 provides a high-level common infrastructure used by the applications 1106. For example, the framework 1108 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. The framework 1108 can provide a wide range of other APIs that can be used by the applications 1106, some of which can be specific to a particular operating system or platform.

[0067] In an example embodiment, the applications 1106 may include a home application 1136, a contacts application 1130, a browser application 1132, a book reader application 1134, a location application 1142, a media application 1144, a messaging application 1146, a game application 1148, and a variety of other applications such as third-party applications 1140. Applications 1106 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1106 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 1140 (e.g., those created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system of a 1112 mobile phone or Linux OS or other mobile operating system. In this example, a third-party application 1140 can activate an API call 1150 provided by the operating system 1112 to facilitate the functions described herein.

[0068] Fig.12 1200 within which instructions 1208 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1200 to perform any one or more of the methods discussed herein. For example, the instructions 1208 may cause the machine 1200 to perform any one or more of the methods described herein. The instructions 1208 convert a general purpose, unprogrammed machine 1200 into a specific machine 1200 that is programmed to perform the functions described and illustrated in the manner described. The machine 1200 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a network deployment, the machine 1200 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1200 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1208, sequentially or otherwise, that specify actions to be taken by the machine 1200. Furthermore, while only a single machine 1200 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1208 to perform any one or more of the methodologies discussed herein.

[0069] The machine 1200 may include a processor 1202, a memory 1204, and an I / O component 1242 that may be configured to communicate with each other via a bus 1244. In an example embodiment, the processor 1202 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1206 that executes instructions 1208 and a processor 1210. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.12 Multiple processors 1202 are shown, but the machine 1200 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0070] The memory 1204 includes a main memory 1212, a static memory 1214, and a storage unit 1216, which are all accessible by the processor 1202 via a bus 1244. The main memory 1204, the static memory 1214, and the storage unit 1216 store instructions 1208 that embody any one or more of the methodologies or functions described herein. The instructions 1208 may also reside, completely or partially, within the main memory 1212, within the static memory 1214, within the machine-readable medium 1218 within the storage unit 1216, within at least one of the processors 1202 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1200.

[0071] I / O components 1242 may include a variety of components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 1242 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be appreciated that I / O components 1242 may include Fig.1212 and 13. In various example embodiments, the I / O components 1242 may include output components 1228 and input components 1230. The output components 1228 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. The input components 1230 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and / or force of touch or touch gestures), audio input components (e.g., microphones), and the like.

[0072] In another example embodiment, the I / O component 1242 may include a biometric component 1232, a motion component 1234, an environment component 1236, or a positioning component 1238, as well as various other components. For example, the biometric component 1232 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1234 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1236 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor used to detect concentrations of hazardous gases to ensure safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1238 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0073] A variety of technologies may be used to achieve communication. I / O components 1242 also include communication components 1240, which are operable to couple machine 1200 to network 1220 or device 1222 via coupling 1224 and coupling 1226, respectively. For example, communication components 1240 may include a network interface component or another suitable device to interface with network 1220. In other examples, communication components 1240 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low energy consumption), Device 1222 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0074] In addition, the communication component 1240 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1240 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1240, such as the location of a geographic location via an Internet Protocol (IP), the location of a location via a wireless network, and the location of a location via a wireless network. The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0075] Various memories (e.g., memory 1204, main memory 1212, static memory 1214, and / or memory of processor 1202) and / or storage unit 1216 may store one or more sets of instructions and data structures (e.g., software) that implement or are used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1208) when executed by processor 1202 cause various operations to implement the disclosed embodiments.

[0076] The instructions 1208 may be sent or received over the network 1220 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 1240) and using any one of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1208 may be sent or received via a coupling 1226 (e.g., a peer-to-peer coupling) with the device 1222 using a transmission medium.

[0077] Although embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes may be made to these embodiments without departing from the broader scope of the present disclosure. Therefore, the description and the accompanying drawings should be considered in an illustrative sense, rather than a restrictive sense. The accompanying drawings, which form a part of the present invention, illustrate specific embodiments in which the present subject matter can be practiced by way of illustration and not limitation. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be derived from these teachings so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, this detailed description should not be considered in a restrictive sense, and the scope of the various embodiments is limited only by the full scope of the equivalents given by the appended claims and such claims.

[0078] An abstract of the disclosure is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted with the understanding that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it can be seen that various features are combined together in a single embodiment for the purpose of simplifying the disclosure. This method of the disclosure should not be interpreted as reflecting the intention that the claimed embodiment requires more features than those explicitly recited in each claim. Rather, as reflected in the attached claims, the subject matter of the present invention lies in less than all the features of a single disclosed embodiment. Therefore, the claims are thereby incorporated into the specific description, with each claim existing as a separate embodiment.

[0079] Example

[0080] Example 1 is a method comprising: accessing a first frame from a camera of a display device; using a hand tracking algorithm running at the display device to track hand pixels corresponding to one or more user hands depicted in the first frame; detecting the depth of the hand pixels using a sensor of the display device; identifying a 3D region based on the depth of the hand pixels; and applying a 3D reconstruction engine to the 3D region.

[0081] Example 2 includes the method of Example 1, wherein the 3D zone comprises an unoccupied 3D space between the camera and the one or more user hands.

[0082] Example 3 includes the method of Example 1, wherein the sensor comprises a depth sensor or a stereo camera.

[0083] Example 4 includes the method of Example 1, wherein detecting the depth is based on matching contours of the one or more user hands in the two images.

[0084] Example 5 includes the method of Example 1, wherein identifying the 3D region comprises: tracking the movement of the one or more user hands; and identifying a 3D envelope including the physical object based on the movement of the one or more user hands.

[0085] Example 6 includes the method of Example 5, wherein applying the 3D reconstruction engine to the 3D region comprises generating a 3D model of the physical object included in the 3D envelope based on point cloud data from the 3D envelope.

[0086] Example 7 includes the method of Example 6, further comprising: identifying the physical object based on a 3D model of the physical object.

[0087] Example 8 includes the method of Example 7, further comprising: identifying virtual content corresponding to the physical object or the 3D model of the physical object; and displaying the virtual content as an overlay of the physical object in a display of the display device.

[0088] Example 9 includes the method of Example 1, wherein identifying the 3D zone based on the movement of the one or more user hands includes: filtering a first portion of the first frame based on the position of the one or more user hands in the first frame to identify a first region of interest; filtering a second portion of the second frame based on the position of the one or more user hands in the second frame to identify a second region of interest; identifying a first hand pixel depth of the one or more user hands in the first frame; identifying a second hand pixel depth of the one or more user hands in the second frame; and identifying the 3D zone based on the first region of interest, the second region of interest, the first hand pixel depth, and the second hand pixel depth.

[0089] Example 10 includes the method of Example 1, wherein applying the 3D reconstruction engine to the 3D region comprises excluding 3D space outside of the 3D region.

[0090] Example 11 is a computing device comprising a processor and a memory storing instructions that, when executed by the processor, configure the device to: access a first frame from a camera of a display device; track hand pixels corresponding to one or more user hands depicted in the first frame using a hand tracking algorithm running at the display device; detect the depth of the hand pixels using a sensor of the display device; identify a 3D region based on the depth of the hand pixels; and apply a 3D reconstruction engine to the 3D region.

[0091] Example 12 includes the computing device of Example 11, wherein the 3D region comprises an unoccupied 3D space between the camera and the one or more user hands.

[0092] Example 13 includes the computing device of Example 11, wherein the sensor comprises a depth sensor or a stereo camera.

[0093] Example 14 includes the computing device of Example 11, wherein detecting the depth is based on matching contours of the one or more user hands in the two images.

[0094] Example 15 includes the computing device of Example 11, wherein identifying the 3D region comprises: tracking the movement of the one or more user hands; and identifying a 3D envelope including the physical object based on the movement of the one or more user hands.

[0095] Example 16 includes the computing device of Example 15, wherein applying the 3D reconstruction engine to the 3D region comprises generating a 3D model of the physical object included in the 3D envelope based on point cloud data from the 3D envelope.

[0096] Example 17 includes the computing device of Example 16, wherein the instructions further configure the device to: identify the physical object based on a 3D model of the physical object.

[0097] Example 18 includes the computing device of Example 17, wherein the instructions further configure the device to: identify virtual content corresponding to the physical object or the 3D model of the physical object; and display the virtual content as an overlay of the physical object in a display of the display device.

[0098] Example 19 includes the computing device of Example 11, wherein identifying the 3D zone based on the movement of the one or more user hands comprises: filtering a first portion of the first frame based on the position of the one or more user hands in the first frame to identify a first region of interest; filtering a second portion of the second frame based on the position of the one or more user hands in the second frame to identify a second region of interest; identifying a first hand pixel depth of the one or more user hands in the first frame; identifying a second hand pixel depth of the one or more user hands in the second frame; and identifying the 3D zone based on the first region of interest, the second region of interest, the first hand pixel depth, and the second hand pixel depth.

[0099] Example 20 is a non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: access a first frame from a camera of a display device; track hand pixels corresponding to one or more user hands depicted in the first frame using a hand tracking algorithm running at the display device; detect the depth of the hand pixels using a sensor of the display device; identify a 3D region based on the depth of the hand pixels; and apply a 3D reconstruction engine to the 3D region.

Claims

1. A method comprising: Accessing a first frame from a camera device of a display device; tracking hand pixels corresponding to one or more user hands depicted in the first frame using a hand tracking algorithm executed at the display device; detecting the depth of the hand pixel using a sensor of the display device; identifying a 3D region based on the depth of the hand pixels; as well as A 3D reconstruction engine is applied to the 3D region.

2. The method according to claim 1, wherein: The 3D zone includes an unoccupied 3D space between the camera and the one or more user hands.

3. The method according to claim 1, wherein: The sensor includes a depth sensor or a stereo camera device.

4. The method according to claim 1, wherein: Detecting the depth is based on matching contours of the one or more user hands in the two images.

5. The method according to claim 1, wherein: Identifying the 3D region comprises: tracking the movement of the one or more user hands; and A 3D envelope including the physical object is identified based on the movement of the one or more user hands.

6. The method according to claim 5, wherein: Applying the 3D reconstruction engine to the 3D region comprises: A 3D model of the physical object included in the 3D envelope is generated based on point cloud data from the 3D envelope.

7. The method according to claim 6, further comprising: The physical object is identified based on the 3D model of the physical object.

8. The method according to claim 7, further comprising: identifying virtual content corresponding to the physical object or the 3D model of the physical object; as well as The virtual content is displayed in a display of the display device as an overlay of the physical object.

9. The method according to claim 1, wherein: Identifying the 3D zone based on the movement of the one or more user hands includes: filtering a first portion of the first frame to identify a first region of interest based on the location of the one or more user hands in the first frame; filtering a second portion of the second frame based on the location of the one or more user hands in the second frame to identify a second region of interest; identifying a first hand pixel depth of the one or more user hands in the first frame; identifying a second hand pixel depth of the one or more user hands in the second frame; and The 3D region is identified based on the first region of interest, the second region of interest, the first hand pixel depth, and the second hand pixel depth.

10. The method according to claim 1, wherein: Applying the 3D reconstruction engine to the 3D region comprises: The 3D space outside the 3D region is excluded.

11. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to: Accessing a first frame from a camera device of a display device; tracking hand pixels corresponding to one or more user hands depicted in the first frame using a hand tracking algorithm executed at the display device; detecting the depth of the hand pixel using a sensor of the display device; identifying a 3D region based on the depth of the hand pixels; as well as A 3D reconstruction engine is applied to the 3D region.

12. The computing device according to claim 11, wherein: The 3D zone includes an unoccupied 3D space between the camera and the one or more user hands.

13. The computing device of claim 11, wherein: The sensor includes a depth sensor or a stereo camera device.

14. The computing device of claim 11, wherein: Detecting the depth is based on matching contours of the one or more user hands in the two images.

15. The computing device of claim 11, wherein: Identifying the 3D region comprises: tracking the movement of the one or more user hands; and A 3D envelope including the physical object is identified based on the movement of the one or more user hands.

16. The computing device of claim 15, wherein: Applying the 3D reconstruction engine to the 3D region comprises: A 3D model of the physical object included in the 3D envelope is generated based on point cloud data from the 3D envelope.

17. The computing device of claim 16, wherein: The instructions further configure the apparatus to: The physical object is identified based on the 3D model of the physical object.

18. The computing device of claim 17, wherein: The instructions further configure the apparatus to: identifying virtual content corresponding to the physical object or the 3D model of the physical object; and The virtual content is displayed in a display of the display device as an overlay of the physical object.

19. The computing device of claim 11, wherein: Identifying the 3D zone based on the movement of the one or more user hands includes: filtering a first portion of the first frame to identify a first region of interest based on the location of the one or more user hands in the first frame; filtering a second portion of the second frame based on the location of the one or more user hands in the second frame to identify a second region of interest; identifying a first hand pixel depth of the one or more user hands in the first frame; identifying a second hand pixel depth of the one or more user hands in the second frame; and The 3D region is identified based on the first region of interest, the second region of interest, the first hand pixel depth, and the second hand pixel depth.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: Accessing a first frame from a camera device of a display device; tracking hand pixels corresponding to one or more user hands depicted in the first frame using a hand tracking algorithm executed at the display device; detecting the depth of the hand pixel using a sensor of the display device; identifying a 3D region based on the depth of the hand pixels; as well as A 3D reconstruction engine is applied to the 3D region.

Images