Controller creation using object scanning and IR retroreflectors

Abstract

A method for enabling controller tracking in an augmented reality environment, and a system-non-temporary computer-readable medium are provided. [Solution] A method for creating a control from a 3D object includes registering object 135 as a controller, placing marker 115 on the object in association with a constellation, capturing the object along with the marker in image data, determining the 3D geometry from the image data, determining the location of the marker relative to the geometry, and registering the 3D object as a controller by associating the geometry with the marker location. The constellation is tracked in additional image data, and the motion characteristics of the controller are determined based on the orientation determined by the constellation, determining the orientation of the controller and using it for user input actions.

Description

Technical Field

[0001] Some devices can generate and present an extended reality (XR) environment. The XR environment can include a fully or partially simulated environment that people perceive and / or interact with via an electronic system. In XR, a subset of a person's body movements or their representation is tracked, and in response, one or more characteristics of one or more virtual objects simulated within the XR environment are adjusted to behave in realistic ways.

[0002] Handheld controllers can be used in an XR environment to enhance user input. The handheld controller can be used as an input system for interacting with a virtual environment. This can improve the immersive experience and provide a more intuitive and natural way to interact with virtual content. These controllers can be tracked by the system to provide input, for example, based on illuminators on the controller. For example, image data of the controller can be captured to determine characteristics corresponding to the input.

Brief Description of the Drawings

[0003] [Figure 1A] An exemplary diagram showing a user registering an object as a controller according to some embodiments is shown.

[0004] [Figure 1B] An exemplary diagram showing a user performing a user input action using a registered controller according to some embodiments is shown.

[0005] [Figure 2] A flowchart of a technique for registering an object as a controller according to one or more embodiments is shown.

[0006] [Figure 3]A flowchart of a technique for associating markers with the geometry of an object is shown, according to several embodiments.

[0007] [Figure 4] A flowchart shows a technique for using a registration controller to perform user input actions, according to several embodiments.

[0008] [Figure 5] A system diagram of an electronic device that can be used for controller tracking, according to one or more embodiments, is shown.

[0009] [Figure 6] This document presents an exemplary system for use in various augmented reality technologies. [Modes for carrying out the invention]

[0010] This disclosure relates to a system, method, and computer-readable medium that enables controller tracking in an augmented reality environment. In particular, the techniques described herein concern creating a controller using a three-dimensional object and then tracking it in an augmented reality environment.

[0011] In some augmented reality contexts, handheld controllers can be used to generate user input. These handheld controllers may be tracked to determine their motion or orientation characteristics, which can then be converted into user input. For example, a handheld controller may include one or more markers, such as light-emitting diodes (LEDs) and / or retroreflective markers, and these markers can be tracked to determine the controller's position and / or orientation, from which user input can be generated. Similarly, other features of the controller can be tracked in image data to determine the controller's motion characteristics. However, controllers are often provided for specific applications or systems. The techniques described herein provide flexible techniques for using controllers for user input by enabling the transformation of three-dimensional objects into controllers.

[0012] Embodiments described herein provide techniques for registering a three-dimensional object as a controller. Markers can be applied to a three-dimensional object in a default or user-defined arrangement. According to some embodiments, markers may be active (e.g., emitting light or other signals) or passive (e.g., reflecting light or other signals). The controller may be scanned or captured by an image capture device to generate image data, which can be used to determine the object's geometry. For example, the image data may represent the object captured from multiple angles so that the device can use this image data to construct a three-dimensional model of the object. In addition, the location of markers on the object relative to the object's 3D structure can be identified. The arrangement of markers can be identified as a constellation. The constellation may be associated with the object's geometry, and the constellation and geometry may be stored when the object is registered as a controller, so that the markers can be tracked to determine the position and / or orientation of the controller's 3D structure.

[0013] Embodiments described herein further provide techniques for tracking objects based on markers placed on 3D objects. In particular, an image including a controller within the field of view may be captured. Markers on the controller may be identified in the image data, and the controller's orientation may be determined based on the configuration of the markers in the image data and registration data for the controller. Then, user input movements may be determined based on the controller's orientation, and user input actions may be triggered based on user input movements.

[0014] The technology described herein provides a technical improvement to input controllers by providing a technology that converts 3D objects into controllers. Next, an improvement for controller tracking is provided by using registration on a client device to map user-generated constellations to the controller geometry and detect user movement.

[0015] In the following disclosure, "physical environment" refers to the physical world that people can perceive and / or interact with without the aid of electronic devices. A physical environment may include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park, including physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through their senses such as sight, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people perceive and / or interact with through electronic devices. For example, an XR environment may include Augmented Reality (AR) content, Mixed Reality (MR) content, Virtual Reality (VR) content, etc. In an XR system, a subset or representation of a person's bodily movements is tracked, and accordingly, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. As an example, an XR system can detect a person's head movements and adjust the graphic content and sound field presented to that person accordingly, in the same way that such views and sounds would change in the physical environment. As another example, an XR system can detect the movement of an electronic device presenting an XR environment (e.g., a mobile phone, tablet, laptop) and adjust the graphic content and sound field presented to that person accordingly, in the same way that such views and sounds would change in the physical environment. In some situations (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphic content(s) within the XR environment in response to a representation of bodily movement (e.g., a voice command).

[0016] The existence of a wide variety of electronic systems enables people to perceive and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speakers and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mountable system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0017] The following description provides numerous specific details for illustrative purposes to enhance understanding of the disclosed concepts. As part of this description, some of the drawings in this disclosure represent structures and devices in block diagram form to avoid obscuring novel aspects of the disclosed concepts. Also, for clarity, not all features of actual implementations are described herein. Furthermore, as part of this description, some of the drawings in this disclosure may be provided in flowchart form. Any particular boxes in a flowchart may be presented in a particular order. However, it should be understood that any particular sequence in any flowchart is used only to illustrate one embodiment. In other embodiments, any of the various components shown in the flowchart may be omitted, or the illustrated sequence of operations may be performed in a different order or simultaneously. In addition, other embodiments may include additional steps not shown as part of the flowchart. Furthermore, the language used in this disclosure has been chosen primarily for readability and explanatory purposes, and not to limit or restrict the subject matter of the invention, and it is necessary to rely on the claims to determine such subject matter of the invention. In this disclosure, any reference to “one embodiment” or “one embodiment” means that a particular feature, structure, or characteristic described in relation to the embodiment is included in at least one embodiment of the disclosed subject matter, and any multiple references to “one embodiment” or “one embodiment” should not be understood as all referring to the same embodiment.

[0018] It should be understood that in the development of actual implementations (such as software and / or hardware development projects), numerous decisions must be made to achieve the developer's specific objectives (e.g., compliance with system and business-related constraints), and these objectives may vary depending on the implementation. It should also be understood that while such development efforts can be complex and time-consuming, they are nevertheless routine work for those skilled in the art who are engaged in the design and implementation of graphic modeling systems that are of interest to this disclosure.

[0019] Figure 1A is an illustrative diagram showing how user 105 registers object 135 as a controller in several embodiments. Figure 100A includes the possibility that user 105 may register object 135 as a controller using an electronic device 120. Object 135 may or may not be an object known to the electronic device 120. For example, object 135 may be any object selected or created by the user, which is registered as a controller by scanning and placing markers on the object for the purpose of treating the object as a controller in a mixed reality environment. The electronic device 120 is shown as a head-mounted device (HMD), but other illustrative devices include smartphones, tablets, laptops, desktops, game consoles, or any other suitable device.

[0020] Multiple markers 115 may be attached to or embedded in a 3D object 135. The object 135 may be a handheld object such as a wand, game controller, glove, ring, baton, or any other three-dimensional object that conforms to default controller parameters. In some embodiments, the markers 115 may be attached to the object 135 by the user. The placement of the markers 115 may not be initially known to the device 105, such as during the creation of the object 135. According to some embodiments, the markers may be placed on the object before or after the object is scanned. Thus, the geometry may be determined before the object is scanned. Alternatively, the object can be scanned to determine the geometry and marker placement simultaneously. In some embodiments, the placement of the markers 115 may be guided or directed by an application or service via a computing device such as an electronic device 120. Alternatively, the user 105 may be prompted by the electronic device 120 to attach the markers 115 to the object 135 according to one or more default marker placement parameters. The multiple markers 105 may be active (e.g., emitting light or other signals) or passive (e.g., reflecting light or other signals). For example, the multiple markers 105 may be infrared (IR) illuminators or IR reflectors that can be detected by an IR camera. The multiple markers 115 may have different shapes, sizes, colors, or patterns to facilitate their identification and distinction.

[0021] According to some embodiments, the marker can be placed on a movable button on an object or on other components, which can move the marker on the button relative to other markers when pressed or otherwise operated. That is, the configuration of the marker is not always consistent. According to one or more embodiments, button presses can be detected based on the changed relationship between the markers. Thus, registering the object as a controller can further include determining a coordinate system for movement, e.g., in the controller space, which can represent the movement of the button as a whole separately from the movement of the controller.

[0022] The electronic device 120 can include one or more cameras that can be used to capture image data that can capture the object 135. For example, the user 105 can place an object within the field of view 125 of the camera of the electronic device 120. In some embodiments, the electronic device 120 can prompt the user to capture images of the object 135 from multiple angles so that the geometry of the object can be reconstructed. Additionally, the image data can be used to determine the location information of the marker relative to the geometry of the object. Exemplary cameras can include depth cameras, infrared cameras, visible light cameras, or any other suitable camera.

[0023] The object 135 can be registered as a controller by storing an association between the geometry of the object 135 and the constellation of the markers 115. In some embodiments, controller registration can further include determining an input portion of the object and associating the input portion with the geometry and constellation.

[0024] Referring to FIG. 1B, an exemplary diagram is presented in which object 135 is used as a handheld controller for performing user input actions. In particular, marker 115 can be detected by the image data captured by electronic device 120. Using the movement of marker 115, the corresponding movement of object 135 can be determined based on the controller registration data of object 135. Object 135 can be used to provide user input to one or more applications. Exemplary related applications include game applications, virtual reality applications, augmented reality applications, or other applications that utilize the handheld controller as an input device. In the example shown, the movement of controller 135 is used to drag icon 145 on virtual display 120 within an augmented reality environment, for example, using augmented reality, virtual reality, enhanced reality, etc.

[0025] According to one or more embodiments, if the geometry of the controller is known, a virtual representation of the controller can be generated. Additionally, the virtual representation can be accurately positioned based on the tracking of the marker location. As an example, if the physical object is a paintbrush that the user is using to create digital art, a virtual representation of the paintbrush performing the stroke can be generated and accurately presented within the mixed reality environment to either the local user or another user within a common mixed reality communication session.

[0026] Figure 2 shows a flowchart of a technique for registering an object as a controller according to one or more embodiments. In particular, Figure 2 shows a technique for mapping a constellation of markers to the geometry of an object for controller registration during the registration or enrollment phase, according to one or more embodiments. While the flowchart describes various components as performing a particular process, it should be understood that the flow in the figure may differ depending on the embodiment, and the function of the components may also differ depending on the embodiment. Furthermore, the various processes may be performed in different orders.

[0027] Flowchart 200 begins at block 205, where the system optionally prompts the user to place markers on a physical object according to a pattern. The physical object can be any suitable object the user wishes to use as a controller, such as a pen, toy, or tool. The user may attach the markers to the physical object using adhesive, magnets, or any other suitable means. In some embodiments, the system may provide a set of parameters for marker placement. For example, at least three markers may need to be placed on the object, or the markers should not all be placed in a single line. Thus, the user can select the placement of the markers within the marker parameters. In some embodiments, the marker parameters may be predefined. In some embodiments, the marker parameters may be specific to the device type or the application in which the object is registered as a controller. A marker may be an IR retroreflector, which is a device that reflects IR light back to its light source.

[0028] Optionally, as shown in block 210, the system may provide a pattern of markers to be placed on a physical object. For example, a predefined pattern may be provided by the system for the placement of objects on a physical object. The pattern may specify the number, shape, size, and placement of markers on the physical object. The pattern may be designed to facilitate detection and tracking of the markers by camera 505. The pattern may be displayed on display 580 or transmitted to another device such as a printer, projector, or mobile device.

[0029] Flowchart 200 proceeds to block 215, where sensor data of a physical object is captured. The sensor data may include image data captured by the device's camera. In some embodiments, additional sensor data, such as accelerometer data, gyroscope data, and magnetometer data, may be captured by the sensors of the local device and / or controller. The sensor data can facilitate the localization of the object registered as the controller. That is, the position and / or orientation of the local device capturing the image of the physical object can be used to determine the relative characteristics and positional information of the object.

[0030] Optionally, as shown in block 220, the system may prompt the user to position a physical object in front of a camera. The camera may be a stereo camera, a depth camera, an IR camera, or any other suitable camera capable of capturing image data of the physical object and marker. The camera may be integrated into the device performing registration, or it may be communicatively coupled to the device, for example, via a wired or wireless connection. In addition, in optional block 225, the system may prompt the user to rotate the physical object to capture images or other sensor data of the physical object from multiple angles. The system may use the sensor data to determine the rotation angle of the physical object.

[0031] Flowchart 200 proceeds to block 230, where the system determines the geometry of a physical object from sensor data. The geometry may include a three-dimensional (3D) model, mesh, point cloud, contour, or other representation of the physical object. The object's geometry can be reconstructed from sensor data using various techniques, such as 3D scanning, depth determination and combination from multiple frames, and trained networks.

[0032] In block 235, flowchart 200 includes determining a pattern of markers on a physical object, such as a constellation. The pattern may include the 3D position, orientation, and shape characteristics of the markers on the physical object. In some embodiments, the markers can be detected by analyzing an image of the object. The marker pattern may be determined based on image data and / or additional data such as depth data, brightness, etc. In some embodiments, the markers can be identified in multiple captured images of the physical object, for example, as described in block 225, and the 3D position of the markers can be triangulated.

[0033] Optionally, as shown in block 240, the system may generate and present feedback regarding marker placement to the user. For example, the placement of markers on an object may cross-reference with a set of parameters, such as specifications or rules, that define the minimum requirements for a device to track an object. As an example, the set of parameters may include a default number of markers that are always visible. In this case, if, after scanning, it is determined that the marker placement does not meet the placement parameters, the user may be prompted to place additional markers on the device so that the placement parameters are met.

[0034] Flowchart 200 ends at block 245, where the system registers a physical object as a controller by associating the marker pattern with the geometry of the physical object. In particular, the system may store controller information so that the object can be identified as a controller. Furthermore, the system may store controller information used to track the geometry of the object registered as a controller, as well as a marker constellation, so that the constellation can be used to track the object being used as a controller during runtime.

[0035] According to one or more embodiments, registering a physical object as a controller may involve determining the spatial relationship between the structural features of the physical object and markers. For example, a device may determine that a structure has a particular orientation when a particular orientation of a constellation is visible. In some embodiments, registration may track a particular part of the structure related to determining the orientation of the object. For example, an object may have a deformable part. Returning to the example of the paintbrush, if the paintbrush has a tip with markers, the tip may deform when in contact with a surface. In this case, the deformation can be tracked based on the movement of the tip marker relative to other markers on the object. Thus, a virtual representation of the paintbrush may be modified accordingly by tracking the movement of the markers.

[0036] Figure 3 shows flowcharts of techniques for associating markers with object geometry according to several embodiments. In particular, Figure 3 shows exemplary techniques for registering a physical object as a controller, as described above with respect to block 240 in Figure 2. While flowcharts describe various components as performing a particular process, it should be understood that the flow in the figure may differ depending on the embodiment, and the function of the components may also differ depending on the embodiment. Furthermore, the various processes may be performed in different orders.

[0037] In block 305, the system may prompt the user to select an input portion of a physical object as the input portion of the controller. The input portion may be a specific part of the object's geometry that is converted into user input movement. The input portion may be any part of a physical object that the user wishes to use to convert into user input, such as an end, joint, or articulation point.

[0038] Optionally, as shown in block 310, the system can detect a selection of an input portion of an object in the captured image data by a user making a selection. For example, the selection may be made by the user touching the input portion with a finger, stylus, pointer, etc. Based on the captured image data, the system may recognize the touch using a vision-based touch detection process. For example, the gap distance between the touch object and the physical object may be determined and analyzed to detect the touch event.

[0039] In block 315, the system can identify a selected input portion within the object geometry. That is, the touch location may be identified in the 3D model of the physical object or in other geometric information. For example, the system may determine the 3D position and orientation of the selected input portion in order to position the input portion of the physical object selected by the user. In some embodiments, the system may additionally or alternatively use the geometry of the physical object and / or a marker constellation to refine the identification of the selected input portion.

[0040] The flowchart ends at block 320, and the system registers the selected input portion of the object with the controller. For example, the portion of the physical object's geometry associated with the selected input portion may be stored along with the controller registration for use during runtime.

[0041] Figure 4 shows flowcharts of techniques for executing user input actions using objects registered as controllers, according to several embodiments. While the flowcharts describe various components as performing a specific process, it should be understood that the flow in the figure may differ depending on the embodiment, and the functionality of the components may also differ depending on the embodiment. Furthermore, the various processes may be executed in different orders.

[0042] Flowchart 400 begins in block 405, where the system captures a user image frame using a controller. The system may capture images during the runtime phase. For example, an image frame may be captured when the user is using an application that utilizes the controller for user input. In block 410, the system may detect markers within the image frame. The system may use various techniques such as computer vision, machine learning, and image processing to detect markers within the image frame and determine their position and / or orientation.

[0043] Flowchart 400 proceeds to block 415, where the system determines controller orientation information based on the detected marker. The controller orientation information may include the 3D position and / or orientation of the marker, from which the position and / or orientation of the physical object can be determined based on the known spatial relationship between the marker and the 3D structure of the controller. In some embodiments, the system can use controller registration to associate the detected marker with the geometry of the physical object.

[0044] The system can use various techniques to convert the movement of physical objects into user input movements. For example, as shown in optional block 420, the system can reconstruct the geometry of the controller based on detected markers and registrations. That is, the system can use controller registrations to extract the geometry of physical objects and convert marker locations into geometry according to controller pose information. In addition, in optional block 425, the system can determine the pose information of the controller's input portion. As described above, registrations can maintain a mapping between the controller and its input portion. The system can use controller registrations to determine the 3D position and orientation of the input portion based on the controller pose information.

[0045] Flowchart 400 proceeds to block 430, where the system may determine the movement of the user input corresponding to the controller posture information. The movement of the user input may be the movement or posture of a physical object or selected input part that represents the user's intention for the user input.

[0046] The flowchart ends at block 435, where the system triggers user input actions according to user input movements. The system can use user input movements to trigger user input actions for the current application or other processes based on user input movements. For example, returning to Figure 1, a user movement of swiping the controller may result in a movement of dragging an icon on the virtual display.

[0047] Referring to Figure 5, a simplified system diagram is shown. In particular, the system includes an electronic device 500 used as a controller to register and track physical objects. The electronic device 500 may be part of a multifunction device such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system described herein. The electronic device 500 may include one or more additional devices such as a server device, base station, or accessory device, in which various functions may be contained within or distributed throughout. Exemplary networks include, but are not limited to, local networks such as a Universal Serial Bus (USB) network, an organization's local area network, and a wide area network such as the Internet. According to one or more embodiments, the electronic device 500 is used to interact with the user interface of an application. It should be understood that the various components and functions within the electronic device 500 may be distributed differently across modules or components, or even across additional devices.

[0048] The electronic device 500 may include one or more processors 520, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 500 may also include memory 530. Memory 530 may include one or more different types of memory that can be used to perform device functions in conjunction with the processor(s) 520. For example, memory 530 may include cache, ROM, RAM, or any type of temporary or non-temporary computer-readable storage medium capable of storing computer-readable code. Memory 530 may store various programming modules for execution by the processor(s) 520, including a controller registration module 535, a controller tracking module 545, and one or more applications 555.

[0049] The controller registration module 535 may be used to register physical objects for use as controllers. The controller registration module may determine the geometry of the physical object and associate the geometry with a constellation of markers on the physical object. In some embodiments, the controller registration module 535 additionally registers the input portion of the physical object. The controller tracking module 545 may determine the position and / or orientation information of the physical controller or markers on the physical controller by analyzing image data captured, for example, by a camera(s) 505. The position and / or orientation information may then be used, for example, for an application(s) 555, for user input.

[0050] The electronic device 500 may also include a storage device 540. The storage device 540 may include one or more non-temporary computer-readable media, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). The storage device 640 may be used to store various data and structures that can be used to store data related to controller tracking. In addition, the storage device 540 may be configured to store controller registration data 525, such as mappings between geometry and markers. Similarly, the controller registration 525 may also map input portions of objects to the geometry and / or markers of objects. The storage device 540 may also include tracking data 530 that can be used to perform tracking, such as trained networks, controller data, and user preferences.

[0051] In one or more embodiments, each of the one or more cameras 505 may be a conventional RGB camera or a depth camera. Furthermore, the cameras 505 may include a stereo camera or other multi-camera system. In addition, the electronic device 500 may include other sensors that can collect sensor data for tracking user movement, such as a depth camera, an infrared sensor, or one or more gyroscopes, accelerometers, or other orientation sensors.

[0052] The electronic device 500 may also include a display 580 that can present a user interface (UI) for user interaction. The display 580 may be an opaque display, or it may be semi-transparent or transparent. The display 580 may incorporate LEDs, OLEDs, digital light projectors, liquid crystal on silicon, and the like.

[0053] Although the electronic device 500 is shown as including the numerous components described above, in one or more embodiments, various components may be distributed across multiple devices. Therefore, while specific calls and transmissions are described in this specification with respect to a particular illustrated system, in one or more embodiments, various calls and transmissions may be directed in different directions based on differently distributed functions. Furthermore, additional components may be used, and combinations of some of the functionalities of any of the components may be linked.

[0054] Referring next to Figure 6, a simplified functional block diagram of an exemplary multifunctional electronic device 600 according to one embodiment is shown. Each of the electronic devices may be a multifunctional electronic device, or may have some or all of the described components of the multifunctional electronic device described herein. The multifunctional electronic device 600 may include a processor 605, a display 610, a user interface 615, graphics hardware 620, device sensors 625 (e.g., proximity sensor / ambient light sensor, accelerometer, and / or gyroscope), a microphone 630, one or more audio codecs 635, one or more speakers 640, a communication circuit 645, a digital image capture circuit 650 (e.g., including a camera system), one or more video codecs 655 (e.g., supporting a digital image capture unit), a memory 660, a storage device 665, and a communication bus 670. The multifunctional electronic device 600 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0055] The processor 605 can execute instructions necessary to perform or control the operation of numerous functions performed by the device 600 (e.g., image generation and / or processing as disclosed herein). The processor 605 can, for example, drive the display 610 and receive user input from the user interface 615. The user interface 615 may enable the user to interact with the device 600. For example, the user interface 615 can take various forms such as buttons, keypads, dials, click wheels, keyboards, display screens and / or touchscreens, gaze and / or gestures. The processor 605 may also be a system-on-a-chip, such as those found in mobile devices, and may include a dedicated GPU. The processor 605 may be based on a reduced instruction set computer (RISC) or composite instruction set computer (CISC) architecture or any other preferred architecture, and may include one or more processing cores. The graphics hardware 620 may be dedicated computing hardware for processing graphics and / or assisting the processor 605 in processing graphics information. In one embodiment, the graphics hardware 620 may include a programmable GPU.

[0056] The image capture circuit 650 may include two (or more) lens assemblies 680A and 680B, each having a distinct focal length. For example, lens assembly 680A may have a shorter focal length than lens assembly 680B. Each lens assembly may have separate associated sensor elements 690A and 690B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 650 can capture still images and / or video images. The output from the image capture circuit 650 may be processed by a video codec(single or multiple) 655 and / or a processor 605 and / or graphics hardware 620, and / or a dedicated image processing unit or pipeline incorporated within the circuit 650. The images thus captured may be stored in memory 660 and / or storage device 665.

[0057] The image capture circuit 650 can capture still images and video images, which, in accordance with this disclosure, may be processed, at least in part, by a video codec(single or multiple) 655 and / or a processor 605 and / or graphics hardware 620, and / or a dedicated image processing unit incorporated within the circuit 650. Images thus captured may be stored in memory 660 and / or storage device 665. Memory 660 may include one or more different forms of media used by the processor 605 and graphics hardware 620 to perform the functions of the device. For example, memory 660 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 665 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 665 may include one or more non-temporary computer-readable storage media, such as magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and DVDs, and semiconductor memory devices such as EPROMs and EEPROMs. The memory 660 and storage device 665 can be organized into one or more modules and used to tangibly hold computer program instructions or code written in any desired computer programming language. For example, when executed by the processor 605, such computer program code can perform one or more of the methods described herein.

[0058] The various processes defined herein consider options for obtaining and using user identification information. For example, such personal information may be used to track movements performed by the user using a controller. However, insofar as such personal information is collected, such information should be obtained with the user's informed consent, and the user should have knowledge and control over the use of that personal information.

[0059] Personal information will be used only for legitimate and reasonable purposes by the appropriate parties. Those who use such information will adhere to privacy policies and practices that comply with at least the applicable laws and regulations. Furthermore, such policies should be well-established and meet or exceed government / industry standards. In addition, these parties will not distribute, sell, or share such information for any purpose other than a reasonable and legitimate purpose.

[0060] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable, including in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization can be facilitated, where appropriate, by removing certain identifiers (e.g., date of birth), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means.

[0061] It should be understood that the above description is illustrative and not limiting. The material is presented in the content of specific embodiments so that a person skilled in the art can manufacture and use the disclosed subject matter as claimed, and variations of those embodiments will be readily apparent to a person skilled in the art (for example, some of the disclosed embodiments may be used in combination with one another). Accordingly, the specific arrangement of steps or actions shown in Figures 2-4, or the arrangement of elements shown in Figures 1 and 5-6, should not be interpreted as limiting the scope of the disclosed subject matter. Accordingly, the scope of the invention should be determined by referring to the appended claims and the entire scope of equivalents given to such claims. In the appended claims, the words “including” and “in which” are used as plain English equivalents of the terms “comprising” and “wherein,” respectively.

Claims

1. It is a method, Capturing image data of a 3D object, Identifying the constellation of markers on the three-dimensional object in the image data, To define the spatial relationship between the constellation and the geometry of the three-dimensional object, To capture image data of the aforementioned constellation, Determining the orientation of the constellation from the image data of the constellation, A method comprising determining the orientation of an object based on the orientation of the constellation and the spatial relationship of the constellation.

2. To generate a three-dimensional digital reconstruction of the three-dimensional object based on the orientation of the constellation, The method according to claim 1, further comprising:

3. Register the input portion of the three-dimensional object according to the aforementioned constellation. The method according to claim 1, further comprising:

4. Identifying the aforementioned multiple markers The user is prompted to touch the aforementioned marker, The method according to claim 3, comprising detecting a touch between a finger and the three-dimensional object in response to the prompt.

5. Based on the aforementioned additional inputs and registrations, the location characteristics of the input portion of the three-dimensional object are determined. The user input action is performed based on the location characteristics of the input portion of the three-dimensional object. The method according to claim 3, further comprising:

6. The method according to claim 1, wherein the plurality of markers include at least one selected from the group consisting of IR illuminators and IR reflectors.

7. The method according to claim 1, wherein the image data includes a plurality of images capturing the three-dimensional object from different views.

8. Defining the spatial relationship between the constellation and the geometry of the three-dimensional object is, The method according to claim 1, comprising registering the three-dimensional object as a controller by associating the constellation with the geometry of the three-dimensional object.

9. A non-temporary computer-readable medium containing computer-readable code, wherein the computer-readable code is Capturing image data of a 3D object, Identifying the constellation of markers on the three-dimensional object in the image data, To define the spatial relationship between the constellation and the geometry of the three-dimensional object, To capture image data of the aforementioned constellation, Determining the orientation of the constellation from the image data of the constellation, A non-temporary computer-readable medium in which determining the orientation of an object based on the orientation of the constellation and the spatial relationships is performable by one or more processors.

10. Further including a computer-readable code, the computer-readable code is A non-temporary computer-readable medium according to claim 9, which generates a three-dimensional digital reconstruction of the three-dimensional object based on the orientation of the constellation.

11. Further including a computer-readable code, the computer-readable code is A non-temporary computer-readable medium according to claim 10, which registers the input portion of the three-dimensional object according to the constellation.

12. The computer-readable code for identifying the plurality of markers is, The user is prompted to touch the aforementioned marker, A non-temporary computer-readable medium according to claim 11, comprising a computer-readable code that detects a touch between a finger and the three-dimensional object in response to the prompt.

13. Further including a computer-readable code, the computer-readable code is Based on the additional inputs and the registration, the location characteristics of the input portion of the three-dimensional object are determined. A non-temporary computer-readable medium according to claim 9, which performs a user input action based on the location characteristics of the input portion of the three-dimensional object.

14. The non-temporary computer-readable medium according to claim 9, wherein the plurality of markers include at least one selected from the group consisting of IR illuminators and IR reflectors.

15. The non-temporary computer-readable medium according to claim 9, wherein the image data includes a plurality of images capturing the three-dimensional object from different views.

16. The computer-readable code for defining the spatial relationship between the constellation and the geometry of the three-dimensional object is: A non-temporary computer-readable medium according to claim 9, comprising computer-readable code for registering the three-dimensional object as a controller by associating the constellation with the geometry of the three-dimensional object.

17. It is a system, One or more processors, The system comprises one or more computer-readable media including the computer-readable code, wherein the computer-readable code is Capturing image data of a 3D object, Identifying the constellation of markers on the three-dimensional object in the image data, To define the spatial relationship between the constellation and the geometry of the three-dimensional object, To capture image data of the aforementioned constellation, Determining the orientation of the constellation from the image data of the constellation, A system in which determining the orientation of an object based on the orientation of the constellation and the spatial relationships is performed by one or more processors.

18. Further including a computer-readable code, the computer-readable code is The system according to claim 17, which generates a three-dimensional digital reconstruction of the three-dimensional object based on the orientation of the constellation.

19. Further including a computer-readable code, the computer-readable code is The system according to claim 17, wherein the input portion of the three-dimensional object is registered according to the constellation.

20. The computer-readable code for identifying the plurality of markers is, The user is prompted to touch the aforementioned marker, The system according to claim 18, comprising a computer-readable code for detecting a touch between a finger and the three-dimensional object in response to the prompt.

21. Further including a computer-readable code, the computer-readable code is Based on the additional inputs and the registration, the location characteristics of the input portion of the three-dimensional object are determined. The system according to claim 17, which performs a user input action based on the location characteristics of the input portion of the three-dimensional object.

22. The system according to claim 17, wherein the plurality of markers include at least one selected from the group consisting of IR illuminators and IR reflectors.

23. The system according to claim 17, wherein the image data includes a plurality of images capturing the three-dimensional object from different views.

24. The computer-readable code for defining the spatial relationship between the constellation and the geometry of the three-dimensional object is: The system according to claim 17, comprising computer-readable code for registering the three-dimensional object as a controller by associating the constellation with the geometry of the three-dimensional object.

Images