Using a Wearable Device to Identify the Location of a Controllable Device

Abstract

According to one aspect, a method for identifying the location of a controllable device includes: receiving visual data from an image sensor on a wearable device; generating identification data by an object identification module based on the visual data; and using the visual data to identify a first 3D map from a map database storing a plurality of 3D maps including a first three-dimensional (3D) map and a second 3D map, wherein the first 3D map is associated with a first controllable device and the second 3D map is associated with a second controllable device. The method includes obtaining the location of the first controllable device in physical space based on visual positioning data of the first 3D map, and rendering a user interface (UI) object at a location within a threshold distance of the location of the first controllable device on a display.

Description

Technical Field

[0001] This description generally relates to obtaining the location of a controllable device using a wearable device. Background Art

[0002] Wearable devices can be used to control one or more controllable devices, such as media streaming devices, home speakers, smart light bulbs, etc. In some examples, the wearable device includes a camera that captures visual data and image recognition software that processes the image data to detect a device in the field of view of the camera of the wearable device, which can obtain the two-dimensional (2D) location of the device. However, the 2D location may not be sufficient to render user interface (UI) controls on the display of the wearable device at a location that approximates the location of the device in 3D space, such that the user can control the device with an interactive UI or be presented with additional information about the device. Additionally, according to some conventional techniques, if the user has multiple devices that look similar (e.g., two identical-looking smart light bulbs), the wearable device may not be able to distinguish between the multiple devices to control the device that the user intends. Summary of the Invention

[0003] According to one aspect, a method of using a wearable device to identify the location of a controllable device includes: receiving visual data from an image sensor on the wearable device; generating identification data by an object recognition module based on the visual data; and using the identification data to identify the first 3D map from a map database that stores a plurality of 3D maps including a first three-dimensional (3D) map and a second 3D map, wherein the first 3D map is associated with a first controllable device and the second 3D map is associated with a second controllable device. The method includes obtaining the location of the first controllable device in physical space based on the visual positioning data of the first 3D map, and rendering a user interface (UI) object at a location within a threshold distance of the location of the first controllable device on the display of the wearable device. In some examples, the UI object is rendered at a location on the display corresponding to the location of the first controllable device.

[0004] According to some aspects, the method may include one or more (or any combination) of the following features. The location includes the six-degree-of-freedom location of the controllable device. The identification data may include at least one of the device name or device type of the first controllable device. The identification data may include the spatial type of the physical space associated with the first controllable device. The identification data is stored in the map database in association with the first 3D map. The method may include comparing the visual data with the first 3D map. The method may include generating the first 3D map during a setup process associated with the first controllable device. The first 3D map may include a feature point map corresponding to a virtual cloud anchor, and the virtual cloud anchor is configured to be shared with one or more other users. The UI object may include one or more controls that allow a user to control the first controllable device.

[0005] According to one aspect, a non-transitory computer-readable medium stores executable instructions that, when executed by at least one processor, are configured to cause the at least one processor to: receive visual data from an image sensor on a wearable device; generate identification data based on the visual data by an object recognition module; use the identification data to identify a first 3D map from a map database storing a plurality of 3D maps including a first three-dimensional (3D) map and a second 3D map, wherein the first 3D map is associated with a first controllable device and the second 3D map is associated with a second controllable device; obtain the location of the first controllable device in a physical space at least partially represented by the visual data based on the visual positioning data of the first 3D map; and render a user interface (UI) object at a location within a threshold distance of the location of the first controllable device on a display of the wearable device, wherein the UI object includes one or more interactive controls to control the first controllable device. In some examples, the UI object is rendered at a location on the display corresponding to the location of the first controllable device (e.g., the six DoF location).

[0006] According to some aspects, a non-transitory computer-readable medium may include one or more of the above / below features (or any combination thereof). A map database may be stored at a server computer. Executable instructions, when executed by at least one processor, may be configured to cause the at least one processor to communicate with the server computer to access the map database. The map database may be stored at a wearable device or a computing device communicatively coupled to the wearable device. Executable instructions, when executed by at least one processor, may be configured to cause the at least one processor to communicate with the wearable device or the computing device, respectively, to access the map database. The identification data may include at least one of a device name, a device type, or a spatial type associated with a first controllable device. The executable instructions may include instructions that cause the at least one processor to generate a first 3D map during a setup process associated with the first controllable device and a second 3D map during a setup process associated with a second controllable device. The first 3D map may include a feature point map corresponding to a virtual cloud anchor, where the virtual cloud anchor is configured to be shared with one or more other users.

[0007] According to one aspect, a computing system for identifying the location of a controllable device may include: an image sensor configured to receive visual data; an object identification module configured to generate identification data based on the visual data; a location identifier configured to use the identification data to identify a first 3D map from a map database storing a plurality of 3D maps including a first three-dimensional (3D) map and a second 3D map, the first 3D map being associated with a first controllable device and the second 3D map being associated with a second controllable device; and a location identifier configured to obtain the location of the first controllable device in physical space based on the visual positioning data of the first 3D map; and a user interface (UI) object renderer configured to render a UI object at a location within a threshold location of the first controllable device on a display of the wearable device. In some examples, the UI object is rendered at a location corresponding to the location of the first controllable device on the display.

[0008] According to some aspects, the computing system may include one or more of the above / below features (or any combination thereof). The computing system may include a wearable device and a computing device, where the computing device is communicatively coupled to the wearable device via a wireless connection. The computing system may include a wearable device, and the wearable device may include smart glasses. The map database may be stored in a memory device of the computing system. The map database may be stored in a memory device associated with a server computer. The computing system may include an antenna configured to transmit the identification data to the server computer for identifying the first 3D map at the server computer, where the antenna is configured to receive visual positioning data from the server computer.

[0009] Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A A computing system for identifying the location of a controllable device using visual data captured by an image sensor and a three-dimensional (3D) map is depicted, according to one aspect.

[0011] Figure 1B A user interface (UI) object rendered on a display of a computing device at a location corresponding to the location of a controllable device is depicted, according to one aspect.

[0012] Figure 1C A map database storing multiple 3D maps is shown, according to one aspect.

[0013] Figure 1D An example of identification data linked to a 3D map in the map database is shown, according to one aspect.

[0014] Figure 1E An example of a device detector is shown that can identify a 3D map in the map database based on the identification data, according to one aspect.

[0015] Figure 1F A system for generating and storing a 3D map at a server computer is shown, according to one aspect.

[0016] Figure 1G A computing system for generating and storing a 3D map is shown, according to one aspect.

[0017] Figure 1H A computing system for generating and storing a 3D map is shown, according to one aspect.

[0018] Figure 2 A computing system having a head-mounted display device and a computing device is shown, according to one aspect.

[0019] Figure 3 An example of smart glasses is shown, according to one aspect.

[0020] Figures 4A to 4D An example of rendering a UI object on a display of a wearable device is shown, according to one aspect.

[0021] Figures 5 to 8 An exemplary display having a UI object rendered at a location corresponding to the location of a controllable device is shown.

[0022] Figure 9 A flowchart depicting an exemplary operation of a computing system is shown, according to one aspect.

[0023] Figure 10 An exemplary computing device showing a computing system discussed herein according to one aspect. Detailed Description

[0024] The present disclosure relates to a wearable device that captures visual data and identifies a pre-stored three-dimensional (3D) map (e.g., 3D mesh, 3D feature map, virtual cloud anchor, etc.) from a map database based on the visual data, where the 3D map marks the orientation of a controllable device in a physical space at least partially represented by the visual data. The map database may store multiple 3D maps, where each 3D map includes visual positioning data regarding a separate controllable device. The map database may be stored locally or remotely on a server computer. Information from the 3D map is used to detect the six degrees of freedom (DoF) position of the controllable device. The wearable device may then render a user interface (UI) object on a display of the wearable device at an orientation within a threshold distance of the position of the controllable device in 3D space. In some examples, the UI object is rendered at a position on the display corresponding to the position of the controllable device (e.g., six DoF position). The UI object may provide the user with one or more controls to control the controllable device and / or additional information regarding the controllable device.

[0025] For example, during the setup of a first controllable device, a computing device (e.g., a wearable device or another type of device such as a smartphone) may be used to mark the orientation of the first controllable device. The user may point an image sensor of the computing device at the first controllable device and move the device around to map the physical space from different perspectives and positions, thereby calculating feature points around the first controllable device.

[0026] In some examples, the feature points are transmitted to a server computer to generate a first 3D map, and the first 3D map is stored in the map database of the server computer. In some examples, the first 3D map is a feature point map. In some examples, the first 3D map is a virtual anchor that can be used to localize the captured scene for another user. The first 3D map is stored in association with identification data including information about the first controllable device. In some examples, the identification data includes the device type and / or device name of the first controllable device. In some examples, the identification data includes the type of space (e.g., living room, bedroom, kitchen) identifying the space in which the first controllable device is located. For example, the server computer may analyze the feature points to determine which type of physical space is associated with the first controllable device (e.g., a bed in the room of the first controllable device may indicate that the first controllable device is in a bedroom), and store the first 3D map in association with the space type. In some examples, the computing device may locally generate and store the first 3D map using the feature points or generally any type of 3D scanning technique.

[0027] The same operations can be applied during the setup of another controllable device. For example, during the setup of a second controllable device, a second 3D map identifying the orientation of the second controllable device is generated. The second 3D map is stored in association with identification data that can identify the second controllable device.

[0028] After the setup process, the user can enter the physical space and use the image sensor of the wearable device to capture visual data. In some examples, the wearable device includes an object identification module that can determine the type of objects included in the visual data. For example, if the first controllable device enters the field of view of the image sensor of the wearable device, the object identification module can analyze the visual data to determine the type of the device. In some examples, the object identification can analyze the visual data to detect the type of space where the user is located. For example, detecting a bed can indicate that the user is in a bedroom. The object identification module can generate identification data (such as device type, device name, space type, etc.) identifying the first controllable device and / or the physical space. The identification data generated by the object identification module can be used to identify an appropriate 3D map from a map database. For example, if the first controllable device is a home speaker, the map database stores a first 3D map associated with the type of the controllable device (such as a home speaker). Then, when the user enters a room with a home speaker and the object identification module of the wearable device detects the home speaker, the first 3D map is identified because the first 3D map has been annotated with the same identification data. In some examples, the wearable device transmits the identification data (derived by the object identifier) to a server computer to identify which 3D maps are stored in the map database.

[0029] Then, to determine whether the identified first 3D map at least partially corresponds to the physical space represented by the visual data captured by the image sensor, the visual data is compared with the first 3D map to determine whether there is a match (e.g., relating to the same physical space). In some examples, the comparison is performed at the server computer, and if the comparison result matches, the server computer returns the pose (e.g., position and orientation) of the first controllable device, and the wearable device uses this pose to determine its position and orientation relative to the first controllable device. In some examples, the comparison is performed locally. In some examples, the pose can be referred to as the six-DoF position of the first controllable device. In some examples, the operation of comparing the visual data with the 3D map can be referred to as parsing the 3D map.

[0030] In some examples, instead of using an object identification module to detect the type of the device captured by the visual data (and then using the type of the device to quickly identify the 3D map), the visual data can be continuously or periodically compared with the 3D maps stored in the map database to determine which 3D map corresponds to the visual data captured by the image sensor. When there is a match, the wearable device can obtain the six-DoF position of the controllable device from the matched 3D map. However, in some examples, parsing the 3D map is computationally expensive, and if the map database includes a relatively large number of 3D maps, the processing power and / or time may be relatively large. Additionally, the system will have to repeatedly (and / or continuously) compare the incoming video data with the 3D maps, thereby reducing the efficiency of the system. However, by leveraging the information obtained by the object identification module, the efficiency can be improved (and the processing power and / or time can be reduced) by parsing fewer 3D maps (e.g., the 3D maps associated with the identified device) for a relatively small number of frames (as compared to continuously comparing the visual database with the 3D maps).

[0031] Figures 1A to 1H FIG. shows a computing system 100 configured to detect the position 134 of one or more controllable devices 152 based on visual data 116 captured by a sensor system 112 on the computing system 100 and 3D maps 124 stored in a map database 105. The position 134 can be the 3D position of the controllable device 152, where the 3D position is the orientation (and optionally, the orientation) of the controllable device 152 in the 3D physical space in which the user (and / or the computing system 100) is located. In some examples, the position 134 includes the six-DoF position 134a of the controllable device 152. Refer to Figure 1B, the computing system 100 uses the location 134 to render a user interface (UI) object 140 on the display 108 in an orientation corresponding to the location of the controllable device 152 in 3D space and / or within a threshold distance of the location of the controllable device 152 in 3D space. In some examples, the computing system 100 uses the location 134 to render the UI object 140 on the display 108 in an orientation that is close to (e.g., from the user's perspective) the location of the controllable device 152 in 3D space. In some examples, the UI object 140 is a virtual object rendered in the physical space captured by the image sensor 114 of the computing system 100. In some examples, the UI object 140 is a visual indicator highlighting the location of the controllable device 152 in the room. In some examples, the UI object 140 provides one or more interactive controls to control the controllable device 152. In some examples, the user can move his / her hand in an orientation close to the UI object 140 (e.g., from the user's perspective) such that the user can interact with the UI object 140. In some examples, the UI object 140 provides additional information about the controllable device 152.

[0032] The computing system 100 can include one or more devices, where at least one device is a display device capable of being worn on or near a person's skin. In some examples, the computing system 100 is a wearable device or includes a wearable device. The wearable device can include a head-mounted display (HMD) device, such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, an augmented reality (AR) device, or other devices, such as goggles or head-mounted headphones having sensors, a display, and computing capabilities. In some examples, the wearable device includes smart glasses. Smart glasses are optical head-mounted display devices designed in the shape of a pair of glasses. For example, smart glasses are glasses that add information (e.g., the projection display 108) next to what the wearer sees through the glasses. In some examples, when the user wears the smart glasses and enters a bedroom where a smart speaker is on a dressing table, the wearable device can render the UI object 140 in a location corresponding to the smart speaker and / or within a threshold distance of the smart speaker (e.g., the location of the display) such that the user can control the smart speaker and / or view additional information about the smart speaker.

[0033] In some examples, computing system 100 includes a wearable device (e.g., smart glasses) and a computing device (e.g., a mobile computing device such as a smartphone, a tablet computer, a laptop computer, or another wearable device such as a smartwatch). The wearable device may be connected to the computing device via a wireless connection such as a short-range connection (e.g., a Bluetooth connection or a Near Field Communication (NFC) connection) or an Internet connection (e.g., Wi-Fi or a mobile network). In some examples, some components of computing system 100 are included in the wearable device, and some components of computing system 100 are included in the computing device. In some examples, all components of computing system 100 are included in the wearable device.

[0034] Computing system 100 includes one or more processors 104, which may be formed in a substrate configured to execute one or more machine-executable instructions or software segments, firmware, or a combination thereof. Processor 104 may be semiconductor-based, that is, the processor may include semiconductor materials capable of performing digital logic. Computing system 100 may also include one or more memory devices 106. Memory device 106 may include any type of storage device that stores information in a format readable and / or executable by processor 104. Memory device 106 may store applications and modules (e.g., device detector 126, location identifier 132, UI object renderer 136, object identification module 118, virtual anchor application 170, etc.), which, when executed by processor 104, perform certain operations. In some examples, the applications and modules may be stored in an external storage device and loaded into memory device 106.

[0035] Computing system 100 includes a display 108 that projects into the user's field of view. Display 108 may include a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting display (OLED), an electrophoretic display (EPD), or a micro-projection display employing an LED light source. In some examples, in the case of smart glasses, display 108 may provide a transparent or semi-transparent display such that a user wearing the glasses can see the image provided by display 108 and also see information in the field of view of the smart glasses that is located behind the projected image. In some examples, UI objects 140 are superimposed onto the user's optical field of view of the physical space. Computing system 100 may include one or more antennas 110 to enable communication with other computing devices. Although not shown in Figures 1A to 1H it, computing system 100 may include a touchpad that allows the user to control computing system 100 (e.g., it may allow swiping across an interface displayed on display 108).

[0036] The computing system 100 includes a sensor system 112. The sensor system 112 includes an image sensor 114. In some examples, the sensor system 112 includes multiple image sensors 114. In some examples, when a user wears the computing system 100 (or a portion of the computing system 100), the image sensor 114 captures visual data 116, which may include image data as well as depth information. In some examples, the image sensor 114 is a red-green-blue (RGB) camera. In some examples, the image sensor 114 includes a pulsed laser sensor (e.g., a LiDAR sensor) or a depth camera. For example, the image sensor 114 can be a camera configured to detect and convey information for creating an image represented by the visual data 116. The image sensor 114 can take pictures and record videos. The sensor system 112 can include an inertial motion unit (IMU) 154. The IMU 115 can detect the motion, movement, and / or acceleration of the computing system 100. The IMU 115 can include various different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The sensor system 112 can include other types of sensors, such as light sensors, audio sensors, distance and / or proximity sensors, contact sensors (such as capacitive sensors), timers, and / or other sensors and / or different combinations of sensors.

[0037] In some examples, the computing system 100 is configured to communicate with a server computer 160 via a network 150. The server computer 160 can be a computing device in the form of multiple different devices, such as a standard server, a group of such servers, or a rack-mounted server system. In some examples, the server computer 160 is a single system that shares components such as processors and memory. The network 150 can include the Internet and / or other types of data networks, such as local area networks (LANs), wide area networks (WANs), cellular networks, satellite networks, or other types of data networks. The network 150 can also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within the network 150.

[0038] The server computer 160 includes one or more processors 162, which can be formed in a substrate configured to execute one or more machine-executable instructions or software segments, firmware, or a combination thereof. The processors 162 can be semiconductor-based, that is, the processors can include semiconductor materials capable of performing digital logic. The server computer 160 includes one or more memory devices 164. The memory devices 164 can include a main memory that stores information in a format readable and / or executable by the processors 162.

[0039] In some examples, the server computer 160 is configured to execute a visual positioning data service 161. The visual positioning data service 161 can be an augmented reality (AR) collaboration service that allows users to create cloud anchors (e.g., 3D map 124) to create a multi-player or collaborative AR experience that users can share. For example, a user can add virtual objects to an AR scene, and then multiple users can view and interact with these virtual objects simultaneously from different locations in the shared physical space. For example, a user can create a local anchor (e.g., 3D map 124) in their environment. During hosting, the computing system 100 can upload data to generate a 3D map 124 at the visual positioning data service 161, the visual positioning data service 161 returns a unique identifier for the 3D map 124, and this unique identifier can be distributed to other users to join the same AR environment. When another user in the same environment points their device camera at the area hosting the 3D map 124 (e.g., virtual cloud anchor), the parsing request causes the visual positioning data service 161 to periodically compare visual features from the scene with the created 3D map 124, and the computing system 100 uses this comparison to accurately determine the user's position and orientation relative to the 3D map 124.

[0040] The controllable devices 152 can include a variety of devices that can be controlled by the computing system 100. As Figure 1A shown, the controllable devices 152 can include a first controllable device 152-1 and a second controllable device 152-2. In some examples, the controllable devices 152 are connected to the same network as the computing system 100 (e.g., the controllable devices 152 are connected to the same Wi-Fi network as the computing system 100). Although two controllable devices 152 are depicted in Figure 1A , the computing system 100 can be capable of connecting to any number of controllable devices 152. The controllable devices 152 can include lights, switches, sockets, thermostats, tag readers, fire or other environmental alarms, blinds, entertainment devices such as televisions, stereos, media players, and computing equipment such as wireless network access points, printers, scanners, and copiers. In some cases, a building may have multiple controllable devices of the same type. For example, a building may have multiple home speakers or multiple video streaming devices. Additionally, a building can have multiple identical or nearly identical controllable devices (e.g., light bulbs, home speakers, etc.).

[0041] A user may wish to use computing system 100 to access, control, or otherwise communicate with controllable device 152. For example, computing system 100 may provide access, control, and / or communication to one or more controllable devices 152 (e.g., first controllable device 152-1 and second controllable device 152-2). To enable computing system 100 to obtain the accurate location of controllable device 152 such that UI object 140 can be rendered on display 108 to access, control, or otherwise communicate with controllable device 152, computing system 100 may cause the generation and storage of 3D map 124 in map database 105, where each 3D map 124 provides visual positioning data for a corresponding controllable device 152. In some examples, map database 105 is stored in memory device 164 at server computer 160. In some examples, map database 105 is stored in memory device 106 at computing system 100. In some examples, map database 105 is stored on a wearable device. In some examples, map database 105 is stored on a computing device (locally) and wirelessly connected to a wearable device.

[0042] As Figure 1C shown, map database 105 may store multiple 3D maps 124, where each 3D map 124 corresponds to a separate controllable device 152. For example, first 3D map 124-1 corresponds to first controllable device 152-1, and second 3D map 124-2 corresponds to second controllable device 152-2. In some examples, 3D map 124 is a 3D mesh. In some examples, 3D map 124 is a 3D feature map. In some examples, 3D map 124 is a virtual anchor or virtual cloud anchor. In some examples, 3D map 124 includes a coordinate space in which visual information from physical space and controllable device 152 is located.

[0043] As Figure 1C shown, 3D map 124 may be stored in association with identification data 120. Identification data 120 may include one or more characteristics about the corresponding controllable device 152 and / or the space in which controllable device 152 is located. In some examples, as Figure 1DAs shown, the identification data 120 includes the device name 121 of the controllable device 152. In some examples, the device name 121 is a name associated with the controllable device 152 (e.g., nest thermostat, Google home mini, etc.). In some examples, the identification data 120 includes the device type 123 of the controllable device 152. In some examples, the device type 123 identifies the type of the device, such as, smart speaker, streaming device, smart thermostat, etc. In some examples, the identification data 120 includes the space type 125 that identifies the type of the space associated with the 3D map 124. For example, the space type 125 can specify bedroom, study, living room, kitchen, etc. As Figure 1C shown, the first 3D map 124-1 is associated with the identification data 120-1 that identifies the first controllable device 152-1, and the second 3D map 124-2 is associated with the identification data 120-2 that identifies the second controllable device 152-2.

[0044] In some examples, the 3D map 124 and the corresponding identification data 120 (or a portion thereof) are generated during the setup process of a particular controllable device 152. In some examples, the 3D map 124 and the corresponding identification data 120 (or a portion thereof) can be generated at some point after the setup process of a particular controllable device 152. In some examples, the 3D map 124 and the corresponding identification data 120 are generated simultaneously (or approximately simultaneously) with each other. In some examples, after the 3D map 124 is generated, the identification data 120 is generated and linked to the corresponding 3D map 124.

[0045] In some examples, during the setup of the first controllable device 152-1, the computing system 100 (e.g., which can be a wearable device or a mobile device such as a smart phone) is used to mark the orientation of the first controllable device 152-2. The user can point the image sensor 114 on the computing system 100 at the center of interest (e.g., the first controllable device 152-2), and move the computing system 100 around to draw the environment from different perspectives and positions, thereby calculating the feature points around the first controllable device 152-2. The data collected by the computing system 100 is used to generate the feature points. The feature points can be points of interest representing the user's environment. In some examples, each feature point approximates a fixed orientation and direction in physical space, and this set of visual feature points can be updated over time as the user moves the device around to draw the physical space.

[0046] In some examples, the feature points are then transmitted to the visual localization data service 161 at the server computer 160 to generate a first 3D map 124-1, and the first 3D map 124-1 is stored in the map database 105 at the server computer 160. As described above, the first 3D map 124-1 is stored in association with the identification data 120-1 that identifies the first controllable device 152-1 (e.g., device name 121, device type 123, space type 125, etc.). If the first controllable device 152-1 is a smart speaker, the identification data 120-1 may indicate that the first 3D map 124-1 is associated with the smart speaker, the name of the smart speaker, and / or the type of the smart speaker. In some examples, the visual localization data service 161 at the server computer 160 may analyze the feature points to determine which type of physical space is associated with the first controllable device 152-1 (e.g., a bed in the room of the first controllable device 152-1 may indicate that the first controllable device 152-1 is in a bedroom), and the first 3D map 124-1 is stored in association with the space type 125. In some examples, instead of generating the first 3D map 124-1 at the server computer 160, the computing system 100 may generate and store the first 3D map 124-1 (and the corresponding identification 120-1) in the map database 105 at the memory device 106 using the feature points or generally any type of 3D scanning technology.

[0047] The same operations can be applied during the setup of another controllable device 152. For example, during the setup of the second controllable device 152-2, a second 3D map 124-2 that identifies the orientation of the second controllable device 152-2 is generated. The second 3D map 124-2 is stored in association with the identification data 120-2 associated with the second controllable device 152-2 (e.g., device name 121, device type 123, space type 125, etc.). If the second controllable device 152-2 is a smart thermostat, the identification data 120-2 may indicate that the second 3D map 124-2 is associated with the smart thermostat, the name of the smart thermostat, and / or the type of the smart thermostat. In some examples, the visual localization data service 161 at the server computer 160 may analyze the feature points to determine which type of physical space is associated with the second controllable device 152-2 (e.g., a sofa in the room of the second controllable device 152-2 may indicate that the second controllable device 152-2 is in a living room), and the second 3D map 124-2 is stored in association with the space type 125. In some examples, the first controllable device 152-1 is located in the same space as the second controllable device 152-2. In some examples, the first controllable device 152-1 is located in a different space from the second controllable device 152-2.

[0048] After a setup process (e.g., during normal use of computing system 100), a user may enter a physical space and use image sensor 114 to capture visual data 116. Computing system 100 includes a device detector 126 configured to use visual data 116 and map database 105 to detect controllable device 152 and the location 134 of controllable device 152. Location 134 may include the six-DoF location 134a of controllable device 152. In some examples, the six-DoF location 134 includes information describing translation and rotation, such as moving up and down (e.g., lifting / heaving), moving left and right (e.g., panning / rocking), moving forward and backward (e.g., walking / surging), rotating left and right (e.g., yawing), tilting forward and backward (e.g., pitching), and / or pivoting left and right (e.g., rolling).

[0049] Device detector 126 includes an object identification module 118. Object identification module 118 may include one or more image identification algorithms, which may include one or more neural networks. In some examples, when image sensor 114 is pointed at an object, object identification module 118 is configured to identify the object by reading barcodes, QR codes, tags, and / or text. Object identification module 118 is configured to identify objects in visual data 116 and generate identification data 120 regarding the detected objects. The identification data 120 generated by object identification module 118 is used to select or identify which 3D maps 124 in map database 105 to parse against visual data 116. As indicated above, identification data 120 may include device name 121, device type 123, space type 125, and / or other characteristics regarding controllable device 152 or the space including controllable device 152.

[0050] For example, if a first controllable device 152-1 is captured by visual data 116, object identification module 118 may generate identification data 120 to include the device name 121 and / or device type 123 of the first controllable device 152-1. Similarly, if a second controllable device 152-2 is captured by visual data 116, object identification module 118 may generate identification data 120 to include the device name 121 and / or device type 123 of the second controllable device 152-2. In some examples, if a bed is captured by visual data 116, object identification module 118 may generate identification data 120 to include the space type 125 (e.g., bedroom). In some examples, if an oven is captured by visual data 116, object identification module 118 may generate identification data 120 to include the space type 125 (e.g., kitchen).

[0051] Device detector 126 includes a location identifier 132 that uses identification data 120 to identify 3D maps 124 stored in map database 105. AsFigure 1E As shown, if the identification data 120 is the identification data 120-1, the first 3D map 124-1 can be identified among the multiple 3D maps 124 because the first 3D map 124-1 is stored in the map database 105 in association with the identification data 120-1. In some examples, the identification data 120-1 can indicate a home speaker, and the second identification data 120-2 can indicate a smart bulb. If the identification data 120 received at the location identifier 132 indicates a home speaker, the first 3D map 124-1 can be identified.

[0052] Then, the visual data 116 is compared with the identified 3D map 124 to determine if there is a match. For example, to determine that the identified 3D map 124 at least partially corresponds to the physical space represented by the visual data 116 captured by the image sensor 114, the visual data 116 is compared with the 3D map 124 to determine if there is a match (e.g., relating to the same physical space). For example, the visual feature points in the 3D map 124 can be used to compare and match with other visual feature points (e.g., detected from the visual data 116) to determine if the physical space is the same as the physical space of the stored feature points, and to calculate the orientation of the controllable device 152 within the physical space.

[0053] If there is a match, the location identifier 132 obtains the visual positioning data 141 of the 3D map 124. In some examples, the visual positioning data 141 includes the pose (e.g., position and orientation) of the controllable device 152. The location identifier 132 uses the visual positioning data 141 to calculate the position 134 of the controllable device 152-1 relative to the computing system 100. In some examples, the location identifier 132 uses the visual positioning data 141 and the information from the IMU 115 to calculate the position 134 of the controllable device 152-1 relative to the computing system 100.

[0054] If the map database 105 is stored in the memory device 164 at the server computer 160, the device detector 126 can operate in combination with the antenna 110 to communicate with the visual positioning data service 161 at the server computer 160. For example, the device detector 126 can transmit the identification data 120 and the visual data 116 to the visual positioning data service 161. The visual positioning data service 161 can identify the appropriate 3D map 124 from the map database 105 stored in the memory device 164 at the server computer 160, compare the visual data 116 with the identified 3D map 124, and if the comparison result is a match, the visual positioning data service 161 can return the visual positioning data 141 of the 3D map 124 to the device detector 126.

[0055] If the map database 105 is stored in the memory device 106 at the computing system 100, in some examples, the device detector 126 may identify an appropriate 3D map 124 from the map database 105 stored in the memory device 106 at the computing system 100, compare the visual data 116 with the identified 3D map 124, and if the comparison result matches, the position identifier 132 may obtain visual positioning data 141 from the appropriate 3D map 124.

[0056] In some examples, the device detector 126 uses signals other than the visual data 116 to assist in detecting the controllable device 152. For example, the device detector 126 may receive one or more wireless signals 130 and use the wireless signals 130 to distinguish between multiple controllable devices 152 with similar appearances. In some examples, the wireless signals 130 include short-range wireless signals such as Bluetooth signals and / or NFC signals (or ultrasonic signals). For example, if two controllable devices 152 with similar appearances are in the user's area, the device detector 126 may receive Bluetooth signals from the controllable devices 152, and the strength of the Bluetooth signals may provide an indication of which controllable device 152 the user wishes to control.

[0057] The computing system 100 includes a UI object renderer 136 that renders a UI object 140 on the display 108 at an orientation proximate to the position 134 of the controllable device 152 in the 3D space. In some examples, the UI object 140 is a virtual object located in the physical space that the user sees through the smart glasses. In some examples, the UI object 140 may be located at an orientation proximate to the position 134 of the controllable device 152 in the real world. In some examples, the UI object 140 is located on the display 108 at an orientation within a threshold distance of the position 134 of the controllable device 152 in the 3D space. In other words, the distance between the position of the UI object 140 and the position 134 of the controllable device 152 may be less than a threshold level. In some examples, the UI object 140 is rendered on the display 108 at a position where the user will recognize that the information provided by the UI object 140 corresponds to the position of the controllable device 152. In some examples, the UI object 140 is rendered on the display 108 at an orientation corresponding to the position 134 of the controllable device 152. In some examples, the position of the UI object 140 is determined or calculated using the position 134 of the controllable device 152 as a reference. In some examples, the UI object 140 provides one or more controls to control the controllable device 152. For example, the UI object 140 may provide one or more controls to start an application on a streaming media player, change the volume of a smart speaker, play a song on a smart speaker, etc. In some examples, the UI object 140 provides additional information about the controllable device 152. For example, the UI object 140 may specify historical data about a smart thermostat.

[0058] Figures 1F to 1H Illustrates various examples of generating and storing a 3D map 124. Figure 1F Illustrates generating and storing, on a server computer according to one aspect, a first 3D map 124 for a first controllable device 152-1. Figure 1G Illustrates generating and storing, on a computing system 100 according to one aspect, a first 3D map 124-1 for a first controllable device 152-1. Figure 1H Illustrates generating and storing, on a computing system 100 according to another aspect, a first 3D map 124-1 for a first controllable device 152-1.

[0059] Reference Figure 1F , during the setup of the first controllable device 152-1, the computing system 100 can be used to mark the orientation of the first controllable device 152-1. The user can point the image sensor 114 of the sensor system 112 at the first controllable device 152-1 and move the computing system 100 around to map the environment from different perspectives and positions. In some examples, the computing system 100 includes a virtual anchor application 170 that receives data from the sensor system 112 and generates virtual anchor data 172. The virtual anchor data 172 can include visual data 174, device pose 176, and anchor pose 178. In some examples, the visual data 174 is an example of the visual data 116. The virtual anchor application 170 is configured to communicate (via one or more application programming interfaces (APIs)) with a visual positioning data service 161 at the server computer 160. The virtual anchor application 170 is configured to detect and track its position relative to the physical space, detect the size and orientation of different types of surfaces (e.g., horizontal, vertical, angled), and estimate the current lighting conditions of the environment. The virtual anchor application 170 is configured to transmit the virtual anchor data 172 (when the computing system 100 maps the environment) to the visual positioning data service 161. The visual positioning data service 161 includes a 3D map generator 165 configured to generate a first 3D map 124-1 based on the virtual anchor data 172. The 3D map generator 165 is configured to store the first 3D map 124-1 in the map database 105.

[0060] In some examples, the virtual anchor application 170 is configured to detect a set of visual feature points from the visual data 174 and track the movement of the set of visual feature points over time. The set of visual feature points is a plurality of points (e.g., points of interest) representing the user's environment, and the set of visual feature points can be updated over time. In some examples, the set of visual feature points can refer to a set of anchors or persistent visual features representing physical objects in the physical world, and the set of visual feature points is stored in the map database 105, which can be used to localize the environment in a subsequent session or for another user.

[0061] Reference Figure 1G During the setup of the first controllable device 152-1, similar to Figure 1F , the computing system 100 is used to mark the orientation of the first controllable device 152-1. The user can point the image sensor 114 of the sensor system 112 at the first controllable device 152-1 and move the computing system 100 around to map the environment from different perspectives and positions. The virtual anchor application 170 receives data from the sensor system 112 and generates virtual anchor data 172, which includes visual data 174, device pose 176, and anchor pose 178. In Figure 1G example, the computing system 100 includes a 3D map generator 165 configured to generate a first 3D map 124-1 based on the virtual anchor data 172. The 3D map generator 165 is configured to store the first 3D map 124-1 in the map database 105 at the computing system 100.

[0062] Reference Figure 1H During the setup of the first controllable device 152-1, similar to Figure 1F and Figure 1G , the computing system 100 is used to mark the orientation of the first controllable device 152-1. In some examples, the computing system 100 includes a 3D scanning module 111 configured to scan the environment of the first controllable device 152-1 and create a first 3D map 124-1, which is stored in the map database 105 at the computing system 100.

[0063] Figure 2 Shows a computing system 200 according to another aspect. The computing system 200 can include any of the features described for the computing system 100 in Figures 1A to 1H . In some examples, the computing system 200 includes a head-mounted display device 202. In some examples, the head-mounted display device 202 includes Figures 1A to 1H all components of the computing system 100 of Figures 1A to 1H . For example, the head-mounted display device 202 can include Figures 1F to 1H the processor 104, memory device 106, display 108, sensor system 112 (including image sensor 114 and IMU 115), antenna, device detector 126 (and sub-components of the device detector 126), and UI object renderer 136 of the computing system 100 of Figures 1A to 1Ha server computer 160). In some examples, Figure 1F and Figure 1G the virtual anchor application 170 is included in the head-mounted display device 202. In some examples, the 3D scanning module 111 is included in the head-mounted display device 202.

[0064] In some examples, the computing system 200 includes a head-mounted display device 202 and a computing device 201. The computing device 201 may be connected to the head-mounted display device 202 via a wireless connection 275. In some examples, the computing device 201 includes a mobile computing device, such as a smart phone, a tablet computer, a laptop computer, or other wearable device. In some examples, the wireless connection 275 is a short-range communication link, such as a near field communication (NFC) connection or a Bluetooth connection. In some examples, the wireless connection 275 is a network connection, such as a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. In some examples, the wireless connection 275 may include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within the network.

[0065] In some examples, Figures 1A to 1H some components of the computing system 100 are included in the head-mounted display device 202, and Figures 1A to 1H some components of the computing system 100 are included in the computing device 201. In some examples, Figures 1A to 1H the sensor system 112, the processor 104, the memory device 106, the antenna 110, and the display 108 are included in the head-mounted display device 202. In some examples, Figures 1A to 1H the UI object renderer 136 is included in the head-mounted display device 202. In some examples, Figures 1A to 1H the UI object renderer 136 is included in the computing device 201. In some examples, Figures 1A to 1H the map database 105 is stored at the computing device 201. In some examples, Figures 1A to 1H the map database 105 is stored at a server computer (e.g., Figures 1A to 1H the server computer 160), where the computing device 201 and / or the head-mounted display device 202 are configured to communicate with Figures 1A to 1H the server computer 160.

[0066] In some examples, Figures 1A to 1HThe device detector 126 is included in the computing device 201. In some examples, the visual data 116 captured by the sensor system 112 on the head-mounted display device 202 can be transmitted to the computing device 201 via the wireless connection 275. The device detector 126 can detect the position 134 of the controllable device 152 and transmit the position 134 to the head-mounted display device 202 via the wireless connection 275 for use by the UI object renderer 136 on the head-mounted display device 202. In some examples, the UI object renderer 136 is included on the computing device 201, and the computing device 201 transmits instructions via the wireless connection 275 to render the UI object 140 on the display 108 on the head-mounted display device 202. In some examples, Figures 1A to 1H the object identification module 118 is included in the head-mounted display device 202, and the position recognizer 132 is included in the computing device 201, where the head-mounted display device 202 transmits the identification data 120 to the position recognizer 132 on the computing device 201.

[0067] In some examples, the computing device 201 is configured to generate and store the 3D map 124, as explained in reference Figures 1F to 1H (e.g., during the setup process), where the 3D map 124 can be stored locally on the computing device 201 or on a server computer (e.g., Figures 1A to 1H the server computer 160). In some examples, Figure 1F and Figure 1G the virtual anchor application 170 is included in the computing device 201. In some examples, Figure 1F and Figure 1G the virtual anchor application 170 is included in both the computing device 201 and the head-mounted display device 202. In some examples, the 3D scanning module 111 is included in the computing device 201.

[0068] Figure 3 Shows an example of a head-mounted display device 302 according to one aspect. The head-mounted display device 302 can be Figures 1A to 1H the computing system 100 and / or Figure 2An example of the head-mounted display device 202. The head-mounted display device 302 includes smart glasses 396. The smart glasses 396 are glasses that add information (e.g., a projection display 308) beside the content that the wearer sees through the glasses. In some examples, instead of projecting information, the display 308 is a microdisplay within the lens. In some examples, the smart glasses 396 (e.g., glasses or spectacles) are visual assistive devices, including lenses 372 (e.g., glass or hard plastic lenses) mounted in a frame 371. The frame 371 generally holds the lenses in front of a person's eyes using a nose bridge 373 on the nose and legs 374 (e.g., temple arms or temple pieces) that rest on the ears. The smart glasses 396 include electronic components 370, and the electronic components 370 include the circuitry of the smart glasses 396. In some examples, the electronic components 370 include a housing that encapsulates Figures 1A to 1H components of the computing system 100 and / or Figure 2 components of the head-mounted display device 202. In some examples, the electronic components 370 are included or integrated into one (or both) of the legs 391 of the smart glasses 396.

[0069] Figures 4A to 4D An example of a display 408 depicting a UI object 440 located in the orientation (e.g., in the resulting visual impression) close to the detected controllable device 452 is shown. In some examples, the controllable device 452 includes a smart speaker. However, the controllable device 452 can include any type of controllable device discussed herein. In some examples, the information depicted in the display 408 is visual information shown through the lenses of the smart glasses. The UI object 440 can be considered a virtual object located in physical space, as shown through the smart glasses.

[0070] Referring to Figure 4A , the UI object 440 can be rendered as a visual indicator around the controllable device 452, which depicts the orientation of the controllable device 452 in 3D space. Referring to Figure 4B , the UI object 440 can be rendered in a position close to (e.g., near) the controllable device 452. In Figure 4B , the UI object 440 includes a visual indicator around the controllable device 452 and UI controls that allow the user to interact with the controllable device 452. In some examples, the user can interact with the UI controls to control the controllable device, such as controls for playing music. Referring to Figure 4C , the UI object 440 can include UI controls that allow the user to interact with the controllable device 452, and a visual indicator indicating the area where the user's hand (or finger) is positioned with respect to other visual information shown in the display 408. For example, the user can move his / her hand (or finger) in the area of the UI object 440 to interact with the controllable device 452. Figure 4CThe visual indicator in can assist the user in determining the position / direction of the user's body part with respect to the UI object 440. Referring to 4D, the UI object 440 can include a visual indicator and a plurality of UI controls that allow the user to control the controllable device 452. The visual indicator indicates the area where the user's hand (or finger) is located with reference to other visual information shown in the display 408. In some examples, the UI controls can include actions such as playing music or searching the web.

[0071] Figure 5 An example of a display 508 is shown, which depicts a UI object 540 located in the vicinity of the detected controllable device 552. In some examples, the controllable device 552 includes a smart TV or a streaming device. However, the controllable device 552 can include any type of controllable device discussed herein. In some examples, the information depicted in the display 508 is visual information shown through the lenses of smart glasses. The UI object 540 can be considered a virtual object located in physical space, as shown through the smart glasses. Referring Figure 5 , the UI object 540 includes UI controls that allow the user to interact with the controllable device 552, such as opening an application such as a streaming application.

[0072] Figure 6 An example of a display 608 is shown, which depicts a UI object 640 located in the vicinity (e.g., near) of the detected controllable device 652. In some examples, the controllable device 652 includes a smart speaker. However, the controllable device 652 can include any type of controllable device discussed herein. In some examples, the information depicted in the display 608 is visual information shown through the lenses of smart glasses. The UI object 640 can be considered a virtual object located in physical space, as shown through the smart glasses. Referring Figure 6 , the UI object 640 includes UI controls that allow the user to interact with the controllable device 652, such as controls for playing music.

[0073] Figure 7 An example of a display 708 is shown, which depicts a UI object 740 located in the vicinity (e.g., near) of the detected controllable device 752. In some examples, the controllable device 752 includes a smart thermostat. However, the controllable device 752 can include any type of controllable device discussed herein. In some examples, the information depicted in the display 708 is visual information shown through the lenses of smart glasses. The UI object 740 can be considered a virtual object located in physical space, as shown through the smart glasses. Referring Figure 7 , the UI object 740 provides additional information about the controllable device 752, such as historical data or other information about the controllable device 752.

[0074] Figure 8 Shows an example of a display 808 that depicts a UI object 840 located in an orientation proximate (e.g., near) a detected controllable device 852. In some examples, the controllable device 852 includes a smart speaker. However, the controllable device 852 can include any type of controllable device discussed herein. In some examples, the information depicted in the display 808 is visual information shown through the lenses of smart glasses. The UI object 840 can be considered a virtual object located in physical space, as shown through the smart glasses. Refer to Figure 8 , the UI object 840 includes UI controls for controlling the smart speaker and information about the content being played on the smart speaker.

[0075] Figure 9 Shows a depiction Figures 1A to 1H of a flowchart 900 of exemplary operations of a computing system 100. Although the flowchart 900 is described with reference to Figures 1A to 1H the computing system 100, the flowchart 900 can be applicable to any embodiment herein.

[0076] Operation 902 includes receiving visual data 116 from an image sensor 114 on a wearable device (e.g., the computing system 100). Operation 904 includes using the visual data 116 to identify a first 3D map 124-1 from a map database 105 that stores a plurality of 3D maps 124 including a first three-dimensional (3D) map 124-1 and a second 3D map 124-2, where the first 3D map 124-1 is associated with a first controllable device 152-1 and the second 3D map 124-2 is associated with a second controllable device 152-2. Operation 906 includes obtaining a position 134 (e.g., a six degrees of freedom (DoF) position 134a) of the first controllable device 152-1 in space relative to the wearable device based on the first 3D map 124-1. Operation 908 includes rendering a user interface (UI) object 140 at a position on a display 108 of the wearable device that is proximate to the position 134 of the first controllable device 152-1.

[0077] Figure 10An example of an exemplary computer device 1000 and an exemplary mobile computer device 1050 that can be used with the techniques described herein is shown. The computing device 1000 includes a processor 1002, a memory 1004, a storage device 1006, a high-speed interface 1008 connected to the memory 1004 and the high-speed expansion port 1010, and a low-speed interface 1012 connected to the low-speed expansion port 1014 and the storage device 1006. Each of the components 1002, 1004, 1006, 1008, 1010, and 1012 is interconnected using various buses and can be mounted on a common motherboard or in other suitable manners. The processor 1002 can process instructions for execution within the computing device 1000, including instructions stored in the memory 1004 or on the storage device 1006, to display graphical information for a GUI on an external input / output device, such as a display 1016 coupled to the high-speed interface 1008. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories, as appropriate. Additionally, multiple computing devices 1000 can be connected, where each device provides a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multi-processor system).

[0078] The memory 1004 stores information within the computing device 1000. In one implementation, the memory 1004 is one or more volatile memory units. In another implementation, the memory 1004 is one or more non-volatile memory units. The memory 1004 can also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0079] The storage device 1006 is capable of providing large-capacity storage for the computing device 1000. In one implementation, the storage device 1006 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory devices, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 1004, the storage device 1006, or the memory on the processor 1002.

[0080] The high-speed interface 1008 manages bandwidth-intensive operations for the computing device 1000, while the low-speed interface 1012 manages less bandwidth-intensive operations. Such function allocation is merely illustrative. In one implementation, the high-speed interface 1008 is coupled to the memory 1004, the display 1016 (e.g., via a graphics processor or accelerator), and is coupled to a high-speed expansion port 1010 that can accept various expansion cards (not shown). In the implementation, the low-speed interface 1012 is coupled to the storage device 1006 and the low-speed expansion port 1014. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network connection device (such as a switch or a router), for example, via a network adapter.

[0081] The computing device 1000 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a standard server 1020, or implemented multiple times in a group of such servers. It can also be implemented as part of a rack server system 1024. Additionally, it can be implemented in a personal computer such as a laptop computer 1022. Alternatively, components from the computing device 1000 can be combined with other components (such as the device 1050) in a mobile device (not shown). Each of such devices can include one or more of the computing devices 1000, 1050, and the overall system can be composed of multiple computing devices 1000, 1050 that communicate with each other.

[0082] The computing device 1050 includes a processor 1052, a memory 1064, input / output devices such as a display 1054, a communication interface 1066 and a transceiver 1068, and other components. The device 1050 can also include a storage device (such as a microdrive or other device) to provide additional storage. Each of the components 1050, 1052, 1064, 1054, 1066, and 1068 is interconnected using various buses, and several components can be mounted on a common motherboard or in other appropriate ways.

[0083] The processor 1052 can execute instructions within the computing device 1050, including instructions stored in the memory 1064. The processor can be implemented as a chipset that includes separate and multiple analog and digital processors. For example, the processor can provide coordination of other components of the device 1050, such as the control of the user interface, the applications running on the device 1050, and the wireless communication performed by the device 1050.

[0084] The processor 1052 can communicate with a user through a control interface 1058 and a display interface 1056 coupled to a display 1054. The display 1054 can be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 1056 can include appropriate circuitry for driving the display 1054 to present graphics and other information to the user. The control interface 1058 can receive commands from the user and convert them for submission to the processor 1052. Additionally, an external interface 1062 can communicate with the processor 1052 to enable near-field communication of the device 1050 with other devices. For example, in some implementations, the external interface 1062 can provide wired communication, or in other implementations it can provide wireless communication, and multiple interfaces can also be used.

[0085] The memory 1064 stores information within the computing device 1050. The memory 1064 can be implemented as one or more of a computer-readable medium, one or more volatile memory units, or one or more non-volatile memory units. Extended memory 1074 can also be provided and connected to the device 1050 through an expansion interface 1072, which can include, for example, a SIMM (Single In-line Memory Module) card interface. Such extended memory 1074 can provide additional storage space for the device 1050, or can also store applications or other information for the device 1050. Specifically, the extended memory 1074 can include instructions for performing or supplementing the processes described above and can also include security information. Thus, for example, the extended memory 1074 can be provided as a security module for the device 1050 and can be programmed with instructions that allow for the secure use of the device 1050. Additionally, a security application can be provided along with additional information via the SIMM card, such as placing identification information on the SIMM card in an unbreakable manner.

[0086] The memory can include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier is, for example, a computer-readable medium or a machine-readable medium that can be received through a transceiver 1068 or an external interface 1062, such as the memory 1064, the extended memory 1074, or the memory on the processor 1052.

[0087] Device 1050 may communicate wirelessly via a communication interface 1066, which may include digital signal processing circuitry when necessary. The communication interface 1066 may provide communication in various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, etc. Such communication may occur, for example, via a radio frequency transceiver 1068. Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 1070 may provide additional navigation-related and location-related wireless data to device 1050, which data may be appropriately used by applications running on device 1050.

[0088] Device 1050 may also communicate audibly using an audio codec 1060, which may receive oral information from a user and convert it into usable digital information. The audio codec 1060 may similarly generate audible sounds for the user, such as via a speaker in, for example, the handset of device 1050. Such sounds may include sounds from a voice telephone call, may include recordings (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 1050.

[0089] Computing device 1050 may be implemented in many different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 1080. It may also be implemented as part of a smart phone 1082, a personal digital assistant, or another similar mobile device.

[0090] Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. Additionally, the term "module" may include software and / or hardware.

[0091] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) that provides machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that provides machine instructions and / or data to a programmable processor.

[0092] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0093] The systems and techniques described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a frontend component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0094] A computing system can include clients and servers. The clients and servers are typically remote from each other and typically interact through a communication network. The relationship between a client and a server arises from computer programs running on respective computers and has a client-server relationship with each other.

[0095] In some implementations, Figure 10 the computing device depicted in Figure 10One or more sensors on the computing device 1050 depicted or other computing devices may provide input to the VR headset 1090 or generally provide input to the VR space. The sensors may include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biosensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 1050 may use the sensors to determine the absolute position and / or detected rotation of the computing device in the VR space, which may then be used as input to the VR space. For example, the computing device 1050 may be incorporated into the VR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. When the user incorporates the computing device / virtual object into the VR space, the positioning of the computing device / virtual object may allow the user to position the computing device to view virtual objects in certain ways in the VR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. The user may move the computing device left and right, up and down, in a circle, etc., and use the device in a similar manner to use the laser pointer.

[0096] In some implementations, one or more input devices included on or connected to the computing device 1050 may be used as input to the VR space. The input devices may include, but are not limited to, touchscreens, keyboards, one or more buttons, trackpads, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, headsets or earbuds with input functionality, game controllers, or other connectable input devices. When the computing device is incorporated into the VR space, a user interacting with the input devices included on the computing device 1050 may cause specific actions to occur in the VR space.

[0097] In some implementations, the touchscreen of the computing device 1050 may be rendered as a touchpad in the VR space. The user may interact with the touchscreen of the computing device 1050. For example, in the VR headset 1090, the interaction is rendered as movement on a touchpad rendered in the VR space. The rendered movement may control objects in the VR space.

[0098] In some implementations, one or more output devices included on the computing device 1050 may provide output and / or feedback to the user of the VR headset 1090 in the VR space. The output and feedback may be visual, tactile, or audio. The output and / or feedback may include, but are not limited to, vibration, turning on and off or flashing and / or blinking one or more lights or strobe lights, emitting an alarm, playing a ringtone, playing a song, and playing an audio file. The output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobe lights, and speakers.

[0099] In some implementations, computing device 1050 can appear as another object in a computer-generated 3D environment. User interactions with computing device 1050 (e.g., rotating, shaking, touching the touchscreen, sliding a finger across the touchscreen) can be interpreted as interactions with objects in the VR space. In the example of a laser pointer in the VR space, computing device 1050 appears as a virtual laser pointer in the computer-generated 3D environment. When the user manipulates computing device 1050, the user in the VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 1050 in the VR space on computing device 1050 or on VR headset 1090.

[0100] In some implementations, one or more input devices other than the computing device (e.g., mouse, keyboard) can be rendered in the computer-generated 3D environment. The rendered input devices (e.g., rendered mouse, rendered keyboard) can be used as being rendered in the VR space to control objects in the VR space.

[0101] Computing device 1000 is intended to represent various forms of digital computers such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1050 is intended to represent various forms of mobile devices such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementations of the invention described and / or claimed in this document.

[0102] Many embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the specification.

[0103] Additionally, the logical flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired result. Additionally, other steps may be provided, or steps may be eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the appended claims.

Claims

1. A method for identifying the position of a controllable device, the method comprising: Receiving image data from an image sensor on a user device; Generating identification data regarding a first controllable device included in the image data by analyzing the image data; Identifying the first 3D map from a map database storing a plurality of 3D maps including a first 3D map and a second 3D map using the identification data, the first 3D map being associated with the first controllable device, the first 3D map being generated during the configuration process of the first controllable device, and the second 3D map being associated with a second controllable device; Comparing the image data with the first 3D map to determine that the first 3D map at least partially corresponds to the physical space represented by the image data; Obtaining the 3D position of the first controllable device in the physical space from the first 3D map; And Rendering a user interface object at a position based on the 3D position of the first controllable device on a display of the user device, the user interface object including controls for interacting with the first controllable device.

2. The method according to claim 1, wherein The 3D position includes the six-degree-of-freedom position of the first controllable device.

3. The method according to claim 1, wherein, The identification data includes information about at least one of the device name and device type of the first controllable device.

4. The method according to claim 1, wherein, The identification data includes information about the classification of the physical space including the first controllable device.

5. The method according to claim 1, wherein, The identification data is first identification data, and the first 3D map is labeled with second identification data regarding one or more controllable devices included in the first 3D map. The method further comprises: Determining whether the first identification data corresponds to the second identification data; and In response to the first identification data being determined to correspond to the second identification data, selecting the first 3D map from the map database.

6. The method according to claim 1, wherein, The identification data is first identification data, and the method further comprises: Generating a plurality of feature points regarding the physical space including the first controllable device; Generating the first 3D map based on the plurality of feature points, the first 3D map identifying the 3D position of the first controllable device in the physical space; and Storing the first 3D map in the map database.

7. The method according to claim 1, further comprising: Receiving a selection of the control on the user interface object; And Initiating the execution of a function on the first controllable device by communicating with the first controllable device.

8. A method for identifying the position of a controllable device, the method comprising: Receiving image data from an image sensor on a first user device; Generating identification data regarding a first controllable device included in the image data by analyzing the image data; Identifying the first 3D map from a map database storing a plurality of 3D maps including a first 3D map and a second 3D map using the identification data, the first 3D map being associated with the first controllable device, and the second 3D map being associated with a second controllable device; Compare the image data with the first three-dimensional map to determine that the first three-dimensional map at least partially corresponds to the physical space represented by the image data; Obtain the three-dimensional position of the first controllable device in the physical space from the first three-dimensional map; Render a user interface object on a display of the first user device at a position based on the three-dimensional position of the first controllable device, the user interface object including controls for interacting with the first controllable device; And Transmit an identifier of the first three-dimensional map to a second user device.

9. A non-transitory computer-readable medium storing executable instructions that, when executed by at least one processor, are configured to cause the at least one processor to perform operations including: Receive image data from an image sensor on a user device; Generate identification data regarding a first controllable device included in the image data by analyzing the image data; Use the identification data to identify the first three-dimensional map from a map database storing a plurality of three-dimensional maps including a first three-dimensional map and a second three-dimensional map, the first three-dimensional map being associated with the first controllable device, the second three-dimensional map being associated with a second controllable device, the first three-dimensional map being generated during a configuration process of the first controllable device, the first three-dimensional map and the second three-dimensional map representing different physical spaces; Compare the image data with the first three-dimensional map to determine that the first three-dimensional map at least partially corresponds to the physical space represented by the image data; Obtain the three-dimensional position of the first controllable device in the physical space from the first three-dimensional map; And Render a user interface object on a display of the user device at a position based on the three-dimensional position of the first controllable device, the user interface object including controls for interacting with the first controllable device.

10. The non-transitory computer-readable medium according to claim 9, wherein, The user interface object displays information about an action of the first controllable device.

11. The non-transitory computer-readable medium according to claim 9, wherein, The identification data includes information about at least one of a device name, a device type, and a space type of the physical space including the first controllable device.

12. The non-transitory computer-readable medium according to claim 9, wherein, The identification data is first identification data, and the first three-dimensional map is labeled with second identification data regarding one or more controllable devices included in the first three-dimensional map, wherein the operations further include: Determine whether the first identification data corresponds to the second identification data; and In response to the first identification data being determined to correspond to the second identification data, select the first three-dimensional map from the map database and obtain the three-dimensional position from the first three-dimensional map.

13. The non-transitory computer-readable medium according to any one of claims 9 to 12, wherein, The operations further include: Transmit an identifier of the first three-dimensional map to a second user device, the second user device being configured to use the identifier to join an augmented reality (AR) session including at least a portion of the physical space.

14. A computing system for identifying the position of a controllable device, the computing system including: At least one processor; A non-transitory computer-readable medium storing executable instructions that cause the at least one processor to: Receive image data from an image sensor; Generate identification data regarding a first controllable device included in the image data by analyzing the image data; Use the identification data to identify, from a map database storing a plurality of three-dimensional maps including a first three-dimensional map and a second three-dimensional map, the first three-dimensional map, the first three-dimensional map being associated with the first controllable device, the second three-dimensional map being associated with a second controllable device, the first three-dimensional map being generated during a configuration process of the first controllable device, and the first three-dimensional map and the second three-dimensional map representing different physical spaces; Compare the image data with the first three-dimensional map to determine that the first three-dimensional map at least partially corresponds to the physical space represented by the image data; Obtain a three-dimensional position of the first controllable device in the physical space from the first three-dimensional map; And Render a user interface object on a display at a position based on the three-dimensional position of the first controllable device, the user interface object including controls for interacting with the first controllable device.

15. The computing system according to claim 14, wherein, The identification data includes information regarding a device name and a device type of the first controllable device.

16. The computing system according to claim 14, wherein, The identification data is first identification data, and the first three-dimensional map is annotated with second identification data regarding one or more controllable devices included in the first three-dimensional map, wherein the executable instructions include instructions that cause the at least one processor to: Determine whether the first identification data corresponds to the second identification data; and In response to the first identification data being determined to correspond to the second identification data, select the first three-dimensional map from the map database.

17. The computing system according to claim 14, wherein The identification data is first identification data, wherein the executable instructions include instructions that cause the at least one processor to: Generate a plurality of feature points regarding the physical space including the first controllable device during a configuration process of the first controllable device; Generate the first three-dimensional map based on the plurality of feature points, the first three-dimensional map identifying the three-dimensional position of the first controllable device in the physical space; and Store the first three-dimensional map in the map database.

18. The computing system according to claim 14, wherein, The executable instructions include instructions that cause the at least one processor to: Transmit an identifier of the first three-dimensional map to a user device configured to use the identifier to join an augmented reality (AR) session including at least a portion of the physical space.

19. The computing system according to claim 14, wherein, The computing system includes a head-mounted display device.

Images