Mapping networked devices

Abstract

Systems, methods, and non-transitory media for locating and mapping a smart device are provided. An exemplary method can include receiving, by an extended reality (XR) device, an identification output from a connected device directly or indirectly coupled to the XR device, detecting an identification output from the connected device, where the identification output includes an audio pattern, a display pattern, and / or a light pattern, and mapping the connected device in a coordinate system of the XR device based on the identification output from the connected device.

Description

[Technical field]

[0001]

[0001] The present disclosure relates generally to mapping networked devices for controlling and interacting with the networked devices. For example, aspects of the present disclosure relate to using an extended reality system to locate and map networked devices for controlling and interacting with the networked devices. [Background technology]

[0002]

[0002] Extended reality (e.g., augmented reality, virtual reality, etc.) devices, such as smart glasses and head-mounted displays (HMDs), generally implement cameras and sensors to track the position of the extended reality (XR) device and other objects in a physical environment. The XR reality device can use the tracking information to provide a realistic XR experience to a user of the XR device. For example, the XR device can enable a user to experience or interact with an immersive virtual environment or content. To provide a realistic XR experience, the XR technology can integrate virtual content with the physical world, which can involve matching the relative pose and movement of objects and devices. The XR technology can use the tracking information to calculate the relative pose of the device, object, and / or map of the real-world environment to match the relative positions and movements of the device, object, and / or real-world environment and anchor the content to the real-world environment in a convincing manner. The relative pose information can be used to match the virtual content with the user's perceived movement and spatiotemporal state of the device, object, and real-world environment.

[0003]

[0003] Some electronic devices, such as XR devices and other electronic devices (e.g., mobile phones, tablet computers, etc.), may be paired with other devices to interact with the other devices. For example, an electronic device may be paired with a networked device, such as an Internet of Things (IoT) device, connected device, or smart home device (e.g., smart camera, speaker, light bulb, lock, plug, thermostat, display, television, security system, appliance, etc.), to control the networked device. In general, the process for configuring an electronic device to control a networked device or other electronic device can be difficult and generally involves time-consuming manual work. Summary of the Invention

[0004]

[0004] Systems and techniques for locating and mapping one or more devices, such as networked devices (e.g., Internet of Things (IoT) devices, connected devices, or smart devices, etc.), are described herein. According to at least one example, a method is provided for mapping a networked device to control the networked device using an electronic device, such as an extended reality (XR) device. The method can include receiving an identification output from a connected device coupled to the electronic device, and mapping the connected device in a coordinate system of the electronic device based on the identification output from the connected device.

[0005] According to at least one example, a non-transitory computer-readable medium for mapping a networked device is provided for controlling the networked device using an electronic device, such as an XR device. In some examples, the non-transitory computer-readable medium can include instructions that, when executed by one or more processors, cause the one or more processors to receive an identification output from a connected device coupled to the electronic device and map the connected device in a coordinate system of the electronic device based on the identification output from the connected device.

[0006] According to at least one example, an apparatus for mapping a networked device is provided for controlling the networked device using an electronic device, such as an XR device. In some examples, the apparatus can include a memory and one or more processors coupled to the memory, where the one or more processors are configured to receive an identification output from a connected device coupled to the electronic device and map the connected device in a coordinate system of the electronic device based on the identification output from the connected device.

[0007] According to at least one example, another apparatus is provided for mapping a networked device to control the networked device using an electronic device, such as an XR device. In some examples, the apparatus may include means for receiving an identification output from a connected device coupled to the electronic device, and mapping the connected device in a coordinate system of the electronic device based on the identification output from the connected device.

[0008]

[0008] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include determining a location of the connected device based on the identification information output, and mapping the connected device in a coordinate system of the electronic device based on the location of the connected device.

[0009] In some examples, the identification information output may include at least one of an audio pattern, a display pattern, a light pattern, and a wireless signal.

[0010]

[0010] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include sending a signal to the connected device configured to trigger an additional identification output from the connected device, sending one or more additional signals to the connected device at one or more different times based on failure to receive the additional identification output, receiving an additional identification output from the connected device, the one or more additional signals configured to trigger the additional identification output from the connected device, and determining a location of the connected device based on the additional identification output. In some examples, mapping the connected device in a coordinate system of the electronic device may include updating a device map to associate the connected device with the location of the connected device.

[0011]

[0011] In some examples, sending one or more additional signals at one or more different times may include determining a failure to receive the additional identification information output while at least one of the electronic device is within threshold proximity to the mapped location of the connected device and the mapped location of the connected device is within a field-of-view of an image sensor of the electronic device, and sending one or more additional signals to the connected device based on the failure to receive the additional identification information output.

[0012]

[0012] In some examples, sending one or more additional signals at one or more different times may include periodically sending one or more additional signals until at least one of: a predetermined amount of time has elapsed without receiving an additional identification information output from the connected device, and a number of additional signals are sent without receiving an additional identification information output from the connected device.

[0013]

[0013] In some examples, receiving an identification information output from the connected device may include detecting one or more patterns in the identification information output based on data from one or more sensors of the electronic device, and recognizing the one or more patterns as the identification information output from the connected device.

[0014] In some cases, the one or more patterns can include an audio pattern and the one or more sensors can include an audio sensor. In some cases, the one or more patterns can include a display pattern and the one or more sensors can include an image sensor. In some cases, the one or more patterns can include a light pattern and the one or more sensors can include a light sensor.

[0015]

[0015] In some aspects, recognizing one or more patterns as identification information output from the connected device may include recognizing the one or more patterns as identification information output using at least one of a neural network and feature matching.

[0016]

[0016] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include receiving a request to trigger an action by a connected device, identifying a location of the connected device based on a mapping of the connected device, and sending a command to the connected device configured to trigger the action by the connected device.

[0017] In some cases, the request to trigger an action by the connected device can include a request to send a video feed () captured by the connected device to the electronic device. In some examples, the method, the non-transitory computer-readable medium, and the apparatus can include sending a command to the connected device, receiving a video feed from the connected device, where the command is configured to trigger the connected device to send the video feed to the electronic device, and displaying the video feed at the electronic device.

[0018] In some examples, the request to trigger an action by the connected device can include a request to trigger the connected device to display content from the electronic device, and sending the command to the connected device can include sending data to the connected device that includes the command and the content from the electronic device. In some examples, the command is configured to trigger the connected device to display the content from the electronic device.

[0019]

[0019] In some examples, the request to trigger an action by the connected device may include a request to adjust a power mode of the connected device, and the command is configured to adjust the power mode of the connected device.

[0020]

[0020] In some cases, the connected device may include a light bulb, and the request to trigger an action by the connected device may include a request to adjust at least one of the brightness and color of light emitted by the light bulb, and the command is configured to trigger the light bulb to adjust at least one of the brightness and color of the light emitted by the light bulb.

[0021] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include detecting a gesture by a user associated with an electronic device, triggering one or more identification outputs from one or more connected devices in a physical environment based on the gesture, the gesture indicating a location of the connected device, receiving a particular identification output from the one or more identification outputs, and determining that the connected device is located at the location based on the particular identification output, the particular identification output emanating from the location. In some examples, mapping the connected device in a coordinate system of the electronic device may include updating a device map to associate the connected device with the location of the connected device.

[0022]

[0022] In some aspects, the method, the non-transitory computer readable medium, and the apparatus may include receiving a voice command by a user associated with an electronic device, triggering one or more identification information outputs from one or more connected devices in the physical environment based on the voice command, where the voice command indicates a location of the connected device, receiving a particular identification information output from the one or more identification information outputs, and determining that the connected device is located at the location based on the particular identification information output, where the particular identification information output emanates from the location.

[0023]

[0023] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include triggering one or more connected devices to enter a mapping mode, sending a device identification information output to the one or more connected devices, receiving localization information from the one or more connected devices, and mapping the one or more connected devices in a coordinate system of the electronic device based on the localization information from the one or more connected devices, where the localization information indicates a location of the one or more connected devices relative to the electronic device.

[0024]

[0024] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include triggering one or more connected devices to enter a mapping mode; receiving location information from the one or more connected devices; determining a second location of the one or more connected devices based on the location information, where the location information indicates a first location of a landmark relative to the one or more connected devices, and a third location of the landmark relative to the electronic device; and mapping the one or more connected devices in a coordinate system of the electronic device based on the second locations of the one or more connected devices.

[0025]

[0025] In some examples, the identification information output may include a light pattern, which may include at least one of a pattern of light dimming and brightening levels emitted by the electronic device, a pattern of color temperature changes of the light emitted by the electronic device, and a sequence of blinking light emitted by the electronic device.

[0026]

[0026] In some examples, the identification information output may include a display pattern, and the display pattern may include at least one of a code displayed on a display device of the electronic device, an image displayed on a display device of the electronic device, a visual pattern displayed on a display device of the electronic device, and a digital object displayed on a display device of the electronic device.

[0027]

[0027] In some examples, the identification information output may include an audio pattern, and the audio pattern may include at least one of an audio code output by the electronic device and a sequence of sounds output by the electronic device.

[0028]

[0028] In some aspects, the method, the non-transitory computer-readable medium, and the apparatus may include sending a signal to the connected device configured to trigger an identification information output from the connected device, and receiving the identification information output from the connected device.

[0029]

[0029] In some cases, receiving the identification information output may include receiving one or more identification information outputs from a plurality of identification information outputs broadcast by the connected devices, the one or more identification information outputs including the identification information output.

[0030]

[0030] In some cases, the connected device is at least one of a light bulb, a speaker, a television, a microphone, one or more sensors, a camera, a thermostat, and a wearable device.

[0031]

[0031] In some examples, the identification information output may include a received radio signal.

[0032] In some aspects, each of the devices described above may be part of, or may include, a networked device and / or an XR device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some examples, the device may include or be part of a vehicle, a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotics device or system, an aviation system, or other device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors. In some cases, one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or other state), and / or for other purposes.

[0033]

[0033] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0034]

[0034] The foregoing, together with other features and embodiments, will become more apparent from the following specification, claims, and accompanying drawings.

[0035]

[0035] Illustrative examples of the present application are described in detail below with reference to the following figures: [Brief description of the drawings]

[0036] [Figure 1]

[0036] A diagram illustrating an example of an extended reality system used to locate, map, and / or control a computing device, in accordance with some examples of the present disclosure. [Diagram 2]

[0037] FIG. 1 illustrates an exemplary system for locating and mapping a computing device, in accordance with some examples of the present disclosure. [Diagram 3]

[0038] FIG. 1 illustrates an example process for visual feature matching to detect visual patterns in identification information output from a computing device, in accordance with some examples of the present disclosure. [Figure 4]

[0039] FIG. 1 illustrates an example system for determining a location of a computing device based on user gestures, in accordance with some examples of the present disclosure. [Diagram 5]

[0040] FIG. 1 illustrates an exemplary localization by an extended reality device using other computing devices, according to some examples of the present disclosure. [Figure 6]

[0041] FIG. 1 illustrates an exemplary joint localization by a device based on common landmarks, in accordance with some examples of the present disclosure. [Figure 7]

[0042] 1 illustrates an exemplary use case for controlling a computing device using an extended reality device, according to some examples of the present disclosure. [Figure 8]

[0043] 1 is a flowchart illustrating an example process for locating connected devices and mapping connected devices, according to some examples of the present disclosure. [Figure 9]

[0044] FIG. 1 illustrates an exemplary computing device architecture, in accordance with some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037]

[0045] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be implemented without these specific details. The figures and descriptions are not limiting.

[0038]

[0046] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0039]

[0047] As previously described, extended reality (e.g., augmented reality, virtual reality, etc.) devices, such as smart glasses and head mounted displays (HMDs), may implement cameras and various sensors to track the position of the extended reality (XR) device and other objects within a physical environment. The XR device may use such tracking information to provide a realistic XR experience to a user of the XR device. For example, the XR device may enable a user to experience or interact with an immersive virtual environment or content. To provide a realistic XR experience, XR technology generally aims to integrate virtual content with the physical world. In some examples, XR technology involves matching the relative poses and movements of objects and devices. For example, the XR device may use the tracking information to calculate the relative poses of devices, objects, and / or generate a map of the real-world environment to match the relative positions and movements of the devices, objects, and / or real-world environment. Using the pose and / or movement of one or more devices, objects, and / or the real-world environment, the XR device can anchor content to and / or enable interaction with the real-world environment in a convincing manner. Relative pose information can be used to match virtual content with the user's perceived movement and spatiotemporal state of devices, objects, and the real-world environment to support the XR experience and interaction with the real-world environment.

[0040]

[0048] In some examples, an electronic device, such as an XR device or other electronic device (e.g., a mobile phone, a smart wearable device, a tablet computer, a laptop computer, etc.), may be paired with a different device, such as a networked device (e.g., an Internet of Things (IoT) device or connected device, such as a smart hub, a light bulb, a speaker, a camera, a doorbell, a security system, a lock, a plug, a switch, a thermostat, an alarm system, a television, an appliance, etc.). For example, a user may configure an XR device to control a networked speaker, a networked light, a networked appliance, or other smart / connected device in a room. To illustrate, in the networked light example, a user may conveniently control the networked light through an XR device rather than using a light switch to turn the light on or off. However, the process for configuring an electronic device to control a networked device may involve manual work and may be difficult and even confusing for many users. The process for locating and mapping networked devices to enable an electronic device to know the location of the networked devices and control the networked devices may also be difficult and time consuming. Often, a networked device may change location, which may cause a previously mapped location of the smart device to become inaccurate / outdated and may affect the ability of the electronic device to control and / or interact with the networked device after the location change. Moreover, each time a mapped networked device moves to a different location, the electronic device may need to perform another complex and time-consuming process of locating the networked device and mapping the networked device.

[0041]

[0049] In some examples, the XR capabilities of the XR device may be leveraged to enable more intuitive and natural interaction with some devices, such as networked devices, from the XR device. For example, the hand tracking and gesture recognition capabilities of the XR device may be leveraged to enable a user of the XR device to use gestures and other XR-based interactions to control the networked device. To facilitate and / or improve such interactions, the XR device may locate and map the networked devices that the user desires to control from the XR device. As mentioned above, locating and mapping the networked devices may be a difficult task. In many cases, the XR device may not have information about its location to locate and map a particular networked device. For example, in a room with multiple networked devices, the XR device may not know where a particular networked device of interest is located, or even which of the networked devices in the room is the particular networked device of interest.

[0042]

[0050] To illustrate, in a room with multiple networked light bulbs, the XR device may not know where it is located in order to locate and map a particular networked light bulb from the multiple networked light bulbs. In some cases, there may be additional networked devices in nearby rooms that may be located even closer to the XR device than one or more of the networked devices in the room, increasing the difficulty of locating and mapping a particular networked device. Because many networked devices are mobile, even if a room or building has only one networked device, the networked device may change location, which may make the previously mapped location of the networked device inaccurate / outdated and prevent the XR device from controlling and / or interacting with the networked device in its new location.

[0043]

[0051] Described herein are systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for locating and mapping controllable devices, such as networked devices (e.g., smart devices, IoT devices, connected devices) for control by other electronic devices, such as XR devices. Although networked or smart devices (e.g., IoT devices and / or other devices with capabilities for Internet / network connectivity) are used herein as examples of controllable devices, the techniques described herein may be implemented by or for any device that is controllable by another device. In some examples, the systems and techniques described herein may enable smart devices to be located and mapped for XR control. An exemplary XR device may locate and map the smart device to understand where the smart device is located spatially in the real world and within the XR device's internal coordinate system. Locating and mapping the smart device for extended reality may enable the XR device to enable more intuitive interaction with the smart device from the XR device. In some examples, the XR device can use sensor data from one or more sensors on the XR device, e.g., image data from an image sensor, audio data from an audio sensor, etc., to locate the smart device and map the smart device.

[0044]

[0052] In some cases, the XR device can remotely trigger the smart device to generate an identification output that the XR device can detect and that can be used to locate and map the smart device. The XR device can identify and associate the identification output with the smart device for future control of the triggered smart device through the XR device. For example, a user of the XR device can remotely trigger the smart device from an application on the XR device, such as a registration application associated with the smart device. The smart device can generate the identification output based on the remote trigger. The XR device can use one or more sensors, such as image sensors and / or audio sensors, to detect the identification output from the smart device. The XR device can use the identification output to locate and map the smart device for current and / or future control of the smart device from the XR device.

[0045]

[0053] A smart device can generate an identification output in several ways. In some examples, the smart device can output a particular pattern that the XR device can recognize as an identification output. To illustrate, in one example involving a smart device with light emitting capabilities (e.g., a smart television (TV), light bulb, etc.), the smart device can output a pattern of dimming and brightening light. The XR device can detect the dimming and brightening pattern and recognize it as an identification output. In some cases, the dimming and brightening pattern may be visible to the human eye (e.g., within the visible light spectrum). In other cases, the dimming and brightening pattern may be imperceptible to the human eye in terms of wavelength and / or duration. As another example, the smart device can change the color temperature of the smart device according to a pattern that the XR device can detect and recognize as an identification output. The color temperature pattern may be visible to the human eye or imperceptible to the human eye (e.g., in wavelength and / or duration). In some cases, the smart device may emit a pattern of light (visible or invisible to the human eye) that the XR device can detect and recognize as an identity output. For example, the smart device may emit infrared (IR) light that the XR device can detect and recognize as an identity output.

[0046]

[0054] In some cases, the smart device can encode the identification output in the wireless signal(s) transmitted by the smart device. For example, the smart device can use Bluetooth to communicate with the XR device. The smart device can use Bluetooth to send a code or pattern representing the identification output to the XR device. The XR device can receive a Bluetooth communication from the smart device and identify the code or pattern in the Bluetooth communication. The XR device can use the identified code or pattern to locate the smart device as described further herein.

[0047]

[0055] In some examples, a smart device may have a display capability (e.g., a smart hub, TV, screen, etc.), an audio capability, and / or other output capability (e.g., emitting light, etc.). In some cases, a smart device with a display capability may output a display pattern (e.g., a Quick Response (QR) code, a barcode, an image, a video, an animation, a rendered object, a graphic, a visual pattern, etc.). The display pattern may be visible to the human eye or imperceptible (e.g., in wavelength and / or duration) to the human eye. The XR device may detect the display pattern and recognize it as an identification output. In some cases, a smart device with an audio capability may output an audio pattern (e.g., Morse code, noise, a sequence of tones, or any other audio pattern), which may be perceptible to the human ear or imperceptible to the human ear in duration and / or frequency (e.g., ultrasound). The XR device may detect the audio pattern and recognize it as an identification output. In other examples, the smart device may output any other type of pattern or combination of patterns, such as a combination of light patterns, color temperature patterns, display patterns, and / or audio patterns.

[0048]

[0056] In some examples, the XR device can detect the identification output from the smart device using a deep neural network. For example, to detect an identification output including a light pattern, a display pattern, and / or an audio pattern, the deep neural network can be pre-trained on a set of image data and / or audio data that includes examples of relevant identification outputs. In some examples, the XR device can detect the identification output via cross-correlation of a digest (e.g., over time) of output statistics, such as brightness and contrast statistics, that exceed a predetermined threshold. In some cases, the XR device can detect the identification output via audio or visual feature matching. For example, the XR device can identify keypoints in an image that captures the visual output, such as a displayed code, and extract features in the image that correspond to the visual output. The XR device can generate feature descriptions for the extracted features. The XR device can match the keypoints, features, and / or feature descriptions with keypoints, features, and / or feature descriptions associated with the visual output. The XR device can detect the visual output based on such feature matching. The XR device can perform similar feature matching with the audio data to detect the audio output.

[0049]

[0057] In some cases, the XR device can dynamically track the location of the smart device. For example, the XR device can perform a relocation to track the location of the smart device in or when the location of the smart device has changed since a previous location determination. The XR device can perform a mapping process periodically and / or as needed. In some examples, when the XR device thinks it is close to a previously mapped smart device, the XR device can trigger an identification output to check whether the smart device is still in the previously mapped location. If the XR device determines that the smart device is not located in an expected location, the XR device can periodically trigger an identification output. When the XR device finds the smart device based on the identification output, the XR device can update the location of the smart device in the XR device's map of the smart device.

[0050]

[0058] In some cases, the smart device can detect when the location of the smart device has changed and trigger a relocation by the XR device. For example, the smart device can detect that its location has changed based on the smart device's network signal and / or characteristics of the smart device's network signal (e.g., signal strength, etc.), sensor data (e.g., image data captured by an image sensor of the smart device, inertial data captured by an inertial measurement unit of the smart device, etc.). In some cases, when the smart device detects that its location has changed, the smart device can notify the XR device that the location of the smart device has changed. For example, the smart device can send a wireless signal (e.g., Bluetooth communication, WiFi communication, etc.) to the XR device. The wireless signal can include an indication (e.g., data, patterns, etc.) of the change in the location of the smart device. In another example, when the smart device detects that its location has changed, the smart device can output an identification output for detection by the XR device. The identification output can trigger a relocation by the XR device.

[0051]

[0059] In another example, a user of an XR device can perform actions that indicate a desire to interact with the smart device, such as pointing at the smart device, maintaining gaze toward the smart device, maintaining a head pose toward the smart device, touching the smart device, etc. In some cases, the actions can identify, suggest, or imply a location of the smart device. If a smart device is not mapped at such a location (or a smart device of interest is not mapped at such a location), the XR device can trigger an identification output for all known / mapped smart devices until a matching identification output is found. The XR device can then update the XR device's map of the smart device to include the smart device at the determined location of the smart device.

[0052]

[0060] In some examples, instead of (or in addition to) triggering the smart device to identify itself (e.g., via an identification output), the XR device can use hand tracking to locate and map the smart device. For example, a user can touch a smart device with a finger to be tracked / located by the XR device and tag its location in the XR device's map of the smart device with the identified smart device.

[0053]

[0061] In some cases, the XR device can leverage image and / or audio sensors on the smart device to locate and map the smart device and / or the XR device relative to one another. For example, the XR device can use one or more smart devices to locate and / or map the XR device. In some examples, the XR device can remotely trigger the smart device to enter a mapping mode. Based on the trigger, the smart device can begin looking for an identification output from the XR device. Upon detecting an identification output from the XR device, the smart device can locate itself relative to the XR device and relay the location information to the XR device. The XR device can convert the location information into a map of the XR device of the smart device and identify / associate the smart device with the determined location.

[0054]

[0062] In some cases, the XR device can leverage one or more smart devices with image and / or audio sensors to locate and map the smart devices. For example, the XR device can remotely trigger a smart device in the environment to enter a mapping mode. The smart device can share feature vectors (e.g., audio and / or visual feature vectors) with the XR device. The XR device can match the smart device feature vectors shared by one or more smart devices against feature vectors acquired by the XR device. The XR device can convert the location information into a map of the smart device and identify / associate the smart device with the determined location.

[0055]

[0063] Various aspects of the present application are described with reference to the figures.

[0056]

[0064] 1 is a diagram illustrating an example of an XR system 100 used to locate, map, and / or control a computing device 150, according to some examples of the disclosure. The XR system 100 may be used to control the computing device 150 using XR and / or any other techniques described herein. The computing devices 150 may include smart devices (e.g., IoT / connected devices, etc.), such as, for example, but not limited to, smart wearable devices (e.g., smart watches, rings, bracelets, gloves, medical / fitness trackers, etc.), smart locks, smart bicycles, smart security systems, smart lights, smart hubs, smart cameras, smart refrigerators, smart speakers, smart medical sensors, smart televisions, smart switches, smart assistants, smart thermostats, robotic vacuum cleaners, mobile computing devices (e.g., laptop computers, tablet computers, smartphones, etc.), video game systems, Global Positioning System (GPS) devices, smart alarms, smart scanners, smart plugs, smart assistants, smart appliances, smart appliances, embedded systems, smart home devices, or any other smart, connected, and / or wireless devices.

[0057]

[0065] The XR system 100 and the computing device 150 may be communicatively coupled to enable the XR system 100 to interact with and control the computing device 150. In some examples, the XR system 100 may implement one or more XR applications, such as, for example, but not limited to, an XR application for managing and / or controlling a computing device (e.g., the computing device 150), a smart home application, a video game application, a device control application, an autonomous driving application, a navigation application, a productivity application, a social media application, a communication application, a modeling application, a media application, an e-commerce application, a browser application, a design application, a map application, and / or any other XR application.

[0058]

[0066] In some examples, the XR system 100 may include an electronic device configured to use information regarding the relative attitude of the XR system 100 and / or the computing device 150 to provide one or more functions, such as XR functions (e.g., locating / tracking, detecting, classifying, mapping, content rendering, etc.), device management and / or control functions, gaming functions, autonomous driving or navigation functions, computer vision functions, robotics functions, etc. For example, in some cases, the XR system 100 may be an XR device (e.g., a head-mounted display, a head-up display device, smart glasses, etc.) configured to detect, locate, and map the location of the computing device 150, and communicate with the computing device 150 to control one or more operations / states of the computing device 150.

[0059]

[0067] In the illustrative example shown in FIG. 1, the XR system 100 may include one or more image sensors, such as image sensors 102A and 102B (hereinafter collectively referred to as “image sensors 102”), an audio sensor 104 (e.g., an ultrasonic sensor, a microphone, etc.), an inertial measurement unit (IMU) 106, and one or more computational components 110. In some cases, the XR system 100 may optionally include one or more other / additional sensors, such as, for example, but not limited to, a radar, a light detection and ranging (LIDAR) sensor, a touch sensor, a pressure sensor (e.g., a barometric air pressure sensor and / or any other pressure sensor), a gyroscope, an accelerometer, a magnetometer, and / or any other sensor. In some examples, the XR system 100 may include additional sensors and / or components, such as, for example, a light emitting diode (LED) device, a storage device, a cache, a communication interface, a display, a memory device, etc. An example architecture and example hardware components that may be implemented by the XR system 100 are further described below with respect to FIG.

[0060]

[0068] The XR system 100 may be part of or implemented by a single computing device or multiple computing devices. In some examples, the XR system 100 may be part of an electronic device(s), such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a gaming console, an XR device such as an HMD, a drone, a computer in a vehicle, an IoT (Internet of Things) device, a smart wearable device, or any other suitable electronic device(s).

[0061]

[0069] In some implementations, the image sensor 102, the audio sensor 104, the IMU 106, and / or the one or more computing components 110 may be part of the same computing device. For example, in some cases, the image sensor 102, the audio sensor 104, the IMU 106, and / or the one or more computing components 110 may be integrated with or into a camera system, a smartphone, a laptop, a tablet computer, a smart wearable device, an XR device such as an HMD, an IoT device, a gaming system, and / or any other computing device. In other implementations, the image sensor 102, the audio sensor 104, the IMU 106, and / or the one or more computing components 110 may be part of or implemented by two or more separate computing devices.

[0062]

[0070] The one or more computing components 110 of the XR system 100 may include, for example, but not limited to, a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. In some examples, the XR system 100 may include other processors, such as, for example, a computer vision (CV) processor, a neural network processor (NNP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The XR system 100 may use the one or more computing components 110 to perform various computing operations, such as, for example, extended reality operations (e.g., tracking, localization, object detection, classification, pose estimation, mapping, content anchoring, content rendering, etc.), device control operations, image / video processing, graphics rendering, machine learning, data processing, modeling, calculations, computer vision, and / or any other operations.

[0063]

[0071] In some cases, one or more computing components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof to perform any of the various operations described herein. In some examples, one or more computing components 110 may include more or fewer computing components than those shown in Figure 1. Moreover, CPU 112, GPU 114, DSP 116, and ISP 118 are merely illustrative examples of computing components provided for purposes of explanation.

[0064]

[0072] The image sensor 102 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device such as a television or computer, a camera, etc. In some cases, the image sensor 102 may be part of a camera or computing device, such as a digital camera, a video camera, an IP camera, a smartphone, a smart television, a gaming system, etc. Moreover, in some cases, the image sensor 102 may include multiple image sensors, such as rear and front sensor devices, and may be part of a dual camera or other multi-camera assembly (e.g., including two cameras, three cameras, four cameras, or other number of cameras).

[0065]

[0073] In some examples, each image sensor of image sensor 102 can capture image data, generate frames based on the image data, and / or provide image data or frames to one or more computation components 110 for processing. A frame can include a video frame of a video sequence or a still image. A frame can include a pixel array representing a scene. For example, a frame can be a Red-Green-Blue (RGB) frame having red, green, and blue color components per pixel, a Luma, Chroma Red, Chroma Blue (YCbCr) frame having a luma component and two chroma (color) components (chroma red and chroma blue) per pixel, or any other suitable type of color or monochrome picture.

[0066]

[0074] In some examples, one or more of the computing components 110 can perform XR processing operations using data from one or more of the image sensor 102, the audio sensor 104, the IMU 106, and / or the computing device 150. For example, in some cases, one or more of the computing components 110 can perform tracking, localization, object detection, object classification, pose estimation, shape estimation, mapping, content anchoring, content rendering, image processing, modeling, content generation, gesture detection, gesture recognition, and / or other operations based on data from the image sensor 102, the audio sensor 104, the IMU 106, and / or the computing device 150. In some examples, one or more of the computing components 110 can use data from one or more of the image sensor 102, the audio sensor 104, the IMU 106, and / or the computing device 150 to generate a three-dimensional (3D) map of the environment and track the location of the XR system 100 within the 3D environment. In some examples, the one or more computational components 110 may detect the computing device 150, locate the computing device 150, map the computing device 150, and / or control the operation / state of the computing device 150 based on data from the image sensor 102, the audio sensor 104, the IMU 106, and / or the computing device 150.

[0067]

[0075] In some examples, the one or more computation components 110 can implement one or more algorithms to track and estimate the relative pose of the XR system 100 and / or the computing device 150. In some cases, the one or more computation components 110 can receive image data captured by one or more of the image sensors 102 and perform pose estimation based on the received image data to calculate the relative pose of the computing device 150 and the XR system 100. In some examples, the one or more computation components 110 can receive sensor data (e.g., image data from the image sensor 102, the audio sensor 104, data from the IMU 106, etc.) and use such data to track and / or map the computing device 150. In some cases, the one or more computation components 110 can implement one or more computer vision models to calculate the relative pose of the computing device 150 and the XR system 100.

[0068]

[0076] In some cases, the XR system 100 can use one or more of the image sensors 102 to detect light patterns generated by the computing device 150. The light patterns can include identification information outputs that the XR system 100 can use to detect, locate, and map the computing device 150 as described further herein. In some examples, the light patterns can include a pattern(s) of varying light levels (e.g., dimming and brightening) emitted by the computing device 150, a pattern of varying color temperatures of light emitted by the computing device 150, a display pattern (e.g., a quick response (QR) code, a barcode, a displayed object, a displayed image, etc.), and / or any other light / visual pattern. In some cases, the light patterns can include light that is visible or invisible to the human eye. In some examples, the light patterns can include IR light emitted by the computing device 150.

[0069]

[0077] In some cases, the XR system 100 can use the audio sensor 104 to detect audio patterns generated by the computing device 150. The audio patterns can include identification information outputs that the XR system 100 can use to detect, locate, and map the computing device 150, as described further herein. In some examples, the audio patterns can include sound patterns, such as, for example, Morse code, noise sequences, and / or any other modulated audio or sound patterns. In some cases, the XR system 100 can use the audio sensor 104 to measure distance information to assist in tracking, locating, mapping, and / or other XR functions. For example, the XR system 100 can use the audio sensor 104 to measure the distance between the XR system 100 and the computing device 150.

[0070]

[0078] In some cases, the IMU 106 can detect acceleration, angular velocity, and / or orientation of the XR system 100 and generate measurements based on the detected acceleration. In some cases, the IMU 106 can detect and measure the orientation, linear velocity, and / or angular velocity of the XR system 100. For example, the IMU 106 can measure the movement and / or pitch, roll, and yaw of the XR system 100. In some examples, the XR system 100 can use measurements acquired by the IMU 106 and / or image data from one or more of the image sensors 102 to calculate the attitude of the XR system 100 in 3D space. In some cases, the XR system 100 can additionally or alternatively use sensor data from the audio sensor 104 and / or the computing device 150 to perform tracking, attitude estimation, mapping, and / or other operations.

[0071]

[0079] The computing device 150 may include one or more output devices 160 that may output / emit a pattern, such as, for example, a light pattern, an audio pattern, a radio pattern, a code or pattern encoded in a Bluetooth signal, and / or any combination thereof. The pattern may include an identification output that the XR system 100 may use to detect, locate, and map the computing device 150. The XR system 100 may detect the pattern emitted by the output device(s) 160 and use the pattern to locate and map the computing device 150, as described further herein.

[0072]

[0080] The output device(s) 160 may include one or more sensors and / or components, such as, for example, but not limited to, a display, a speaker, a microphone, an image sensor, an LED device, a light emitting device, a pressure sensor, an IMU, a communication interface (e.g., wireless radio, etc.), a radar, etc. In some examples, the computing device 150 may include one or more wireless communication interfaces (not shown) for communicating with the XR system 100. In some examples, the communication interface may include a wireless transmitter, a wireless transceiver, or any other means for wireless communication and / or for transmitting data. The communication interface may implement any wireless protocol and / or technology for communicating with the XR system 100, such as Wi-Fi, Bluetooth, ZigBee, cellular, etc. In other examples, the computing device 150 may include more or less sensors, output device(s), and / or components (of the same and / or different types) than those shown in FIG. 1. In some cases, computing device 150 may include one or more other components not shown in FIG. 1, such as, for example, but not limited to, a microcontroller, an actuator, a storage device, a memory, a cache, a processing device, etc.

[0073]

[0081] The components shown in FIG. 1 for the XR system 100 and the computing device 150 are merely illustrative examples provided for purposes of explanation. In other examples, the XR system 100 and / or the computing device 150 may include more or less components than those shown in FIG. 1. Although the XR system 100 and the computing device 150 are shown to include several components, one skilled in the art will appreciate that the XR system 100 and the computing device 150 may include more or less components than those shown in FIG. 1. For example, the XR system 100 and / or the computing device 150 may also include one or more other memory devices (e.g., RAM, ROM, cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, cache, storage devices, and / or other hardware or processing devices in some cases that are not shown in FIG. 1. Illustrative examples of computing devices and / or hardware components that may be implemented with the XR system 100 and / or the computing device 150 are described below with respect to FIG. 9.

[0074]

[0082] As mentioned above, the XR system 100 can enable XR interactions with smart devices (e.g., computing device 150). For example, the XR system 100 can enable a user wearing the XR system 100 to point at a smart light bulb to turn it on. As another example, the XR system 100 can stream a video feed of a nearby security camera and / or render a visual overlay (e.g., a transparent red outline, a message, etc.) on a door equipped with a security system. In some examples, the XR system 100 can share content from the XR system 100 to a nearby smart television and / or communicate with a smart hub / assistant, indicating to the smart hub / assistant that the user wearing the XR system is speaking with the smart hub / assistant, so the smart hub / assistant understands that the user is not speaking with another person or device.

[0075]

[0083] To facilitate such interactions, the XR system 100 can locate and map the smart device, as described further herein. Locating and mapping the smart device can enable the XR system 100 to know where the smart device is (spatially) in the internal coordinate system of the XR system 100. The XR system 100 can use this information to enable interactions with the smart device.

[0076]

[0084] 2 illustrates an example system 200 for locating and mapping a computing device 150. The computing device 150 may include a smart device (e.g., an IoT / connected device), such as, for example, but not limited to, a smart wearable device (e.g., a smart watch, a ring, a bracelet, a glove, a medical / fitness / health tracker, etc.), a smart lock, a smart bicycle, a smart security system, a smart light, a smart hub, a smart camera, a smart refrigerator, a smart speaker, a smart plug, a smart assistant, a smart medical sensor, a smart television, a GPS device, a mobile computer, a video game console, a smart alarm, a smart scanner, a connected appliance, a smart device, a smart home device, an embedded system, or any other smart, connected, and / or wireless device.

[0077]

[0085] In the example shown in FIG. 2, the XR system 100 can send an identification trigger 204 to the computing device 150 to trigger the computing device 150 to output an identification output 206. The identification trigger 204 can include a message and / or command configured to trigger the computing device 150 to generate an identification output 206 that the XR system 100 can use to locate and map the computing device 150 as described herein. In some cases, the XR system 100 can generate the identification trigger 204 in response to a user input that requests the identification trigger 204. For example, a user of the XR system 100 can select a control element on the XR application 202 in the XR system 100 to trigger the generation of the identification trigger 204. As another example, a user of the XR system 100 can provide a spoken input that requests the generation of the identification trigger 204, and the XR system 100 can recognize the spoken input using one or more voice recognition algorithms.

[0078]

[0086] In some cases, the identification trigger 204 may be generated in response to an event, such as, for example, a trigger from the computing device 150 (e.g., a message, an action, etc.), a determination (e.g., based on sensor data from one or more sensors) that the computing device 150 is nearby (e.g., within the proximity of the XR system 100), a time, etc. In some cases, the XR system 100 may periodically output the identification trigger 204 to trigger the identification output 206 when the computing device 150 is within range and / or area, or when the computing device 150 and the XR system 100 are connected to the same local network and / or communicatively coupled to each other (directly or indirectly). In other cases, the computing device 150 may trigger the XR system 100 to generate the identification trigger 204. For example, when the computing device 150 detects (e.g., based on sensor data) that the XR system 100 is within a range and / or area, the computing device 150 may send a message to the XR system 100 to trigger a discovery mode. The discovery mode can trigger the XR system 100 to generate an identification trigger 204 and look for an identification output 206 to locate and map the computing device 150 .

[0079]

[0087] The computing device 150 can receive the identification trigger 204 and output an identification output 206. The identification output 206 can include an output pattern that the XR system 100 can detect to locate and map the computing device 150. The output pattern (e.g., the identification output 206) can include a light pattern, a display pattern, a radio pattern, a Bluetooth pattern, and / or an audio pattern generated by the output device 160 of the computing device 150. For example, in some cases, the output device 160 can include a light emitting capability, and the output pattern can include a dimming and brightening light pattern. The dimming and brightening pattern can be visible to the human eye (e.g., within the visible light spectrum) or imperceptible to the human eye in terms of wavelength and / or duration. In some cases, the output pattern can include IR light emitted by the output device 160. As another example, the output pattern can include a pattern of changes in color temperature of the light emitted by the output device 160. The color temperature pattern may be visible to the human eye or may be imperceptible (eg, in wavelength and / or duration) to the human eye.

[0080]

[0088] In other examples, the output pattern may include a display pattern presented / displayed by the output device 160. The display pattern may include, for example, a code (e.g., a QR code, a barcode, a key, etc.), an image, one or more characters, a rendered content item (e.g., a virtual content item, a digital object, a graphic, etc.), and / or any visual pattern. The display pattern may be visible to the human eye or imperceptible (e.g., in wavelength and / or duration) to the human eye. In other examples, the output pattern may include an audio pattern (e.g., Morse code, a sound, and / or any other audio pattern), which may be perceptible to the human ear or imperceptible to the human ear in duration and / or frequency (e.g., ultrasound). In other examples, the output pattern may include a radio pattern (e.g., some special code, an on and off pattern, or a low power and high power pattern). In other examples, the output pattern may include any other type of pattern or combination of patterns, such as a combination of light patterns, color temperature patterns, visual patterns, and / or audio patterns.

[0081]

[0089] In some cases, the computing device 150 can encode the identification output 206 in the wireless signal(s) transmitted by the computing device 150. For example, the computing device 150 can use Bluetooth to communicate with the XR system 100. The computing device 150 can use Bluetooth to send the XR system 100 a code or pattern representing the identification output 206. The XR system 100 can receive the Bluetooth communication from the computing device 150 and identify the code or pattern in the Bluetooth communication. The XR system 100 can use the identified code or pattern to locate the computing device 150 as further described herein.

[0082]

[0090] In some cases, the computing device 150 may broadcast the identification output 206 with an output pattern periodically or in response to an identification trigger 204. In some cases, the computing device 150 may encode information in the identification output 206, such as information about the computing device 150, location information, an identifier, etc. The identification output 206 may be detected by a nearby device, such as the XR system 100, located within proximity to the computing device 150 and / or a position relative to the computing device 150. In some examples, the distance and / or position of the nearby device that allows detection of the identification output 206 by the nearby device (e.g., the XR system 100) may depend on the type of output pattern (e.g., light, audio, display, etc.), the configuration of the output pattern, how the output pattern is emitted, etc. For example, the XR system 100 may detect the light / display pattern from the computing device 150 when the light / display pattern and / or the output device 160 is within a view of one or more of the image sensors 102 of the XR system 100. As another example, the XR system 100 can detect an audio pattern from the computing device 150 when the distance between the output device 160 and the XR system 100 is such that the audio sensor 104 of the XR system 100 can detect / recognize the audio in the audio pattern emitted by the output device 160. As another example, the XR system 100 can detect a wireless pattern from the computing device 150 when the distance between the output device 160 and the XR system 100 is such that the communication interface of the XR system 100 can detect / recognize the wireless pattern emitted by the output device 160.

[0083]

[0091] In some cases, the computing device 150 may vary the type of output pattern in the identification information output 206 based on one or more factors, such as, for example, the location of the computing device 150 (e.g., a private location, a public location, a location with a certain noise level, a location with a certain light or darkness level, a location with a certain number or configuration of visual impairments, etc.), preferences / settings indicating whether nearby devices should have a view to the output device 160, the type of environment, the size and / or configuration of the room in which the computing device 150 is located, etc.

[0084]

[0092] For example, if the preference settings indicate that the output pattern may be made detectable by nearby devices that do not have a view of output device 160 (e.g., by nearby devices that do not have an image sensor with a view of output device 160), then computing device 150 may broadcast audio and / or light patterns that can be detected by nearby devices even if the nearby devices' image sensors do not have a view of output device 160 (e.g., and therefore cannot detect the display pattern presented on output device 160). If the preference settings indicate that the output pattern should instead be detectable only by nearby devices that have a view of output device 160 (e.g., by nearby devices that have image sensor(s) that have a view of output device 160), then computing device 150 may output a display pattern that can be detected by nearby devices when the nearby devices' image sensors have a view of output device 160 and therefore have the display pattern presented on output device 160.

[0085]

[0093] In some examples, the output pattern can include a sequence that can indicate to a device detecting the output pattern that the output pattern is an identifying information output. For example, the output pattern can include a sequence, such as a bit sequence, that distinguishes the output pattern from other types of light patterns, audio patterns, display patterns, and / or similar types of patterns.

[0086]

[0094] The XR system 100 can use one or more of the image sensors 102 and / or audio sensors 104 to detect the identification output 206 from the computing device 150. For example, if the identification output 206 includes a light / display pattern, the XR system 100 can use one or more of the image sensors 102 to detect / capture the light / display pattern. One or more of the image sensors 102 can capture an image(s) of the identification output 206, and the XR system 100 can use the image(s) to detect the identification output 206 (e.g., via deep neural networks, feature matching, cross-correlation of digests of light statistics, etc.). If the identification output 206 includes an audio pattern, the XR system 100 can use the audio sensor 104 to detect / capture the audio pattern. The audio sensor 104 may capture audio data at the identification information output 206 and use the audio data to detect the identification information output 206 (e.g., via a deep neural network, audio feature matching, etc.).

[0087]

[0095] In some examples, the XR system 100 can detect the identification output 206 when the XR system 100 is within a certain range of the computing device 150 and / or when the XR system 100 is at a certain location relative to the computing device 150. The proximity and / or location of the XR system 100 to be able to detect the identification output 206 can depend on one or more factors as previously described, such as the type of output pattern (e.g., display, light, audio, etc.), the environment and / or location of the computing device 150 and / or the XR system 100, the configuration of the output pattern (e.g., display size, display location, audio level, light level, etc.).

[0088]

[0096] When one or more of the image sensor 102 and / or audio sensor 104 detect / capture the identification output 206, the localization engine 212 of the XR system 100 can perform localization to locate the identification output 206 and the computing device 150. In some examples, the localization can be based on the identification output 206 detected by the XR system 100 and the tracking data 214 generated by the XR system 100. In some examples, the tracking engine 210 of the XR system 100 can use data from one or more of the image sensor 102, audio sensor 104, and / or IMU 106 to perform tracking to determine the position of the XR system 100 in the environment and / or relative to the computing device 150 and generate the tracking data 214. In some examples, the XR system 100 can use the tracking data 214 for XR functions, such as, for example, localization, tracking, mapping, pose estimation, etc.

[0089]

[0097] In some cases, the tracking data 214 may include data indicating a position and / or orientation of the XR system 100 in a scene / environment, a map (or a portion of a map) of the scene / environment of the XR system 100. In some cases, the map in the tracking data 214 may map a 3D environment and locate one or more features / objects within the 3D environment, such as the position and / or orientation of the XR system 100 and / or the computing device 150. In some examples, the XR system 100 may use the tracking data 214 to provide XR functionality and / or enable XR-based interaction with the scene / environment (and / or objects in the scene / environment), such as the computing device 150. For example, the XR system 100 may use the tracking data 214 to enable a user of the XR system 100 to interact with and / or control the computing device 150, such as through an XR interface on the XR system 100, using gestures. The gestures may include, for example, but are not limited to, hand gestures, gaze, pointing at the computing device 150, touching the computing device 150, placing the computing device 150 within the field of view (FOV) of an image sensor of the XR system 100, and / or any other gesture.

[0090]

[0098] In some cases, the localization engine 212 can use the tracking data 214 and sensor data from the image sensor 102 and / or audio sensor 104 to locate the computing device 150 and associate the computing device 150 with the identification output 206. The localization engine 212 can generate localization data 208 that locate the identification output 206 and associate the identification output 206 with the computing device 150. The XR application 202 on the XR system 100 can use the localization data 208 to map the computing device 150. For example, the XR application 202 can use the localization data 208 to update the device map 222 in the environment map 220 to include the computing device 150 and the location of the computing device 150 in the 3D environment. The environment map 220 can map the 3D environment of the XR system 100 to identify features / objects and corresponding locations in the 3D environment. The device map 222 may include a map of computing devices (e.g., computing devices 150) that have been located and mapped within the environment map 220. In some examples, the device map 222 may identify the located devices and their estimated positions in 3D space.

[0091]

[0099] The XR system 100 can use the environment map 220 and the device map 222 to enable a user of the XR system 100 to interact with and / or control the computing device 150 through an interface rendered on the XR system 100 (e.g., via the XR application 202), through movements, through gestures, and / or through any other XR input / action. For example, the XR system 100 can use the environment map 220 and the device map 222 to enable a user of the XR system 100 to intuitively interact with the computing device 150 using the XR system 100 (e.g., through gestures, movements, physical interactions, etc.). To illustrate, in a non-limiting example, when a user points to a particular location in 3D space, the XR system 100 can use the environment map 220 and the device map 222 to determine that the computing device 150 is located at the particular location. Based on the mapped environment and the mapped location of the computing device 150 in the environment map 220 and the device map 222, the XR system 100 can associate a gesture (e.g., pointing to a specific location) with the computing device 150, even when there are other computing devices nearby and / or in the 3D environment. In some examples, the XR system 100 can interpret a user pointing to a specific location as a request to control or interact with the computing device 150. For example, the XR system 100 can interpret a user pointing to a specific location as a request to turn on / off a light of the computing device 150.

[0092]

[0100] In some examples, the XR system 100 detects a user input for controlling the computing device 150 (e.g., via a gesture, an interface rendered on the XR system 100, etc.) and generates a command for controlling the computing device 150. The XR system 100 can send a command to the computing device 150 to trigger an action / state in the computing device 150. In some examples, the XR system 100 can determine that a user input for controlling a device corresponds to the computing device 150 based on the location of the computing device 150 determined using the environment map 220 and the device map 222. For example, the XR system 100 can determine which device in a particular location the user wants to control by determining that the computing device 150 is in a particular location based on the environment map 220 and the device map 222.

[0093]

[0101] In some examples, commands generated by the XR system 100 to control the computing device 150 may include one or more instructions to adjust one or more operations and / or states of the computing device 150. For example, in some cases, the XR application 202 may generate commands to instruct the computing device 150 to turn off or on, adjust the light level of the computing device 150, play or stop a sound (e.g., a song, an alarm, a message, a noise, a notification, etc.), change the power state of one or more components, start or stop a camera operation, generate an output, provide a data feed (e.g., video, image, audio) from the computing device 150 to the XR system 100, start or stop an operation in the computing device 150, and / or stop or implement any other operation and / or state. The computing device 150 may receive a command from the XR system 100 and execute the command to implement one or more operations and / or states associated with the command.

[0094]

[0102] In some cases, the XR system 100 may perform relocation of the computing device 150 periodically, on demand, and / or upon a relocation trigger. In some examples, the XR system 100 may dynamically track the location of the computing device 150. For example, the XR system 100 may perform relocation to track the location of the computing device 150 in or when the location of the computing device 150 has changed since a previous location. In some cases, the XR system 100 may perform a mapping process periodically and / or as needed. In some examples, when the XR system 100 considers it is close to a previously mapped device (e.g., the computing device 150), the XR system 100 may trigger an identification output to check whether the device is still at the previously mapped location. If the XR system 100 determines that the device is not located in an expected location (e.g., at a previously mapped location), the XR system 100 may trigger an identification output one or more times, or periodically. When the XR system 100 finds a device (e.g., the computing device 150) based on the identification information output, the XR system 100 can update the location of the device in the device map 222.

[0095]

[0103] In another example, a user of the XR system 100 may perform an action indicating a desire to interact with the computing device 150, such as pointing at the computing device 150, maintaining a gaze toward the computing device 150, maintaining a head pose toward the computing device 150, touching the computing device 150, etc. In some cases, the action may identify, suggest, or imply the location of the computing device 150. If the computing device 150 is not mapped at such a location (or the computing device 150 is not mapped at such a location), the XR system 100 may trigger an identification output for any known / mapped devices (e.g., for all devices in the device map 222) until a matching identification output is detected. The XR system 100 may then use the detected identification output to locate the computing device 150 and update the device map 222 to include the computing device 150 at the determined location of the computing device 150.

[0096]

[0104] In some cases, the computing device 150 can trigger a relocation by the XR system 100. In some examples, the computing device 150 can detect when the location of the computing device 150 has changed and trigger a relocation by the XR system 100. For example, the computing device 150 can detect that its location has changed based on the network signal of the computing device 150 and / or the characteristics of the network signal of the computing device 150 (e.g., signal strength, etc.), sensor data (e.g., image data captured by an image sensor of the computing device 150, inertial data captured by an inertial measurement unit of the computing device 150, etc.). In some cases, when the computing device 150 detects that its location has changed, the computing device 150 can notify the XR system 100 that the location of the computing device 150 has changed. For example, the computing device 150 can send a wireless signal (e.g., Bluetooth communication, WiFi communication, etc.) to the XR system 100. The wireless signal may include an indication (e.g., data, a pattern, etc.) of a change in location of the computing device 150. In another example, when the computing device 150 detects that its location has changed, the computing device 150 may output an identification output for detection by the XR system 100. The identification output may trigger a relocation by the XR system 100.

[0097]

[0105] As mentioned above, the XR system 100 can use image / video data captured by one or more of the image sensors 102 and / or audio data captured by the audio sensor 104 to detect the identification output 206 from the computing device 150. In some examples, the XR system 100 can detect the identification output 206 using a deep neural network. For example, to detect an identification output including a light pattern, a display pattern, a radio pattern, and / or an audio pattern, the XR system 100 can use a deep neural network pre-trained on a set of image data and / or audio data that includes examples of relevant identification outputs. In some examples, the XR system 100 can detect the identification output 206 via cross-correlation of digests of audio and / or light statistics (e.g., light level / pattern statistics, audio pattern / sequence statistics, etc.) that exceed a predetermined threshold. In some cases, the XR system 100 can detect the identification output 206 via audio or visual feature matching. For example, the XR system 100 can identify keypoints in an image that capture a visual pattern in the identification output 206, such as a displayed code, and extract features in the image that correspond to the visual pattern. The XR system 100 can generate feature descriptions for the extracted features. The XR system 100 can match the keypoints, features, and / or feature descriptions with keypoints, features, and / or feature descriptions associated with the visual pattern. The XR system 100 can detect the visual pattern based on such feature matching. The XR system 100 can perform similar feature matching with audio data to detect audio output.

[0098]

[0106] 3 illustrates an example process 300 for visual feature matching to detect visual patterns in an identification output from a computing device 150. In this example, a feature extraction engine 315 of the XR system 100 may process an image 302 of a visual object 306 associated with a given identification output and perform feature extraction 320 to extract features 324 from the image 302. The features 324 may include a feature vector corresponding to a feature of the visual object 306 and a description of the feature in the feature vector.

[0099]

[0107] The XR system 100 can capture an image 304 of the identification information output from the computing device 150. The image 304 can capture a visual object 306 included in the identification information output from the computing device 150. The XR system 100 can input the image 304 to a feature extraction engine 315 to perform feature extraction 322 to extract features 326 from the image 304. The features 326 can include a feature vector corresponding to a feature of the visual object 306 and a description of the feature in the feature vector.

[0100]

[0108] The XR system 100 can perform feature matching 330 using the features 324 from feature extraction 320 and the features 326 from feature extraction 322 to determine whether the features 324 and 326 match. The feature matching 330 can compare the features 324 and 326 to determine whether the features 324 and 326 both correspond to the same visual object (e.g., the visual object 306). If the feature matching 330 determines that the features 324 and 326 correspond to the same visual object, the XR system 100 can determine that the visual object 306 corresponds to the identification information output. The XR system 100 can detect and / or identify the identification information output from the computing device 150 based on the visual object 306 in the identification information output.

[0101]

[0109] In some cases, if image 302 and image 304 capture visual object 306 from different angles / views and visual object 306 is visually discernable from both angles / views, XR system 100 may perform a transformation, such as a homography transformation, to obtain matching feature vectors between image 302 and image 304. In some examples, XR system 100 may determine correspondence between features in image 302 and features in image 304 to determine a match.

[0102]

[0110] In some cases, feature 324 can include the 3D location of feature 324 and feature 326 can include the 3D location of feature 326. In some cases, the XR system 100 can use the 3D location of the feature to determine a correspondence between the features (e.g., to match the features). In some cases, the XR system 100 can use the 3D location of the feature to help locate an identification output associated with feature 326.

[0103]

[0111] 4 illustrates an example system 400 for locating a computing device 150 based on a user gesture. In this example, the user gesture is a hand gesture (e.g., pointing at the computing device 150 or touching the computing device 150 with a finger) and the location location uses hand tracking. However, in other examples, the user gesture can be any other type of gesture from a user and the location location can use any other relevant tracking of the user. For example, in some cases, the gesture can be maintaining a gaze toward the computing device, placing the computing device 150 within the field of view of an image sensor of the XR system 100, maintaining a certain head pose, etc. In some cases, the tracking can include eye / gaze tracking, head pose tracking, and / or other gesture tracking.

[0104]

[0112] The tracking engine 410 of the XR system 100 can use data from the image sensor 102 and the IMU 106 to track the location and orientation of the XR system 100, as previously described with respect to the tracking engine 210 shown in Figure 2. The tracking engine 410 can generate tracking data 412, which can include the location and orientation of the XR system 100 and a map of the 3D environment, as previously described with respect to the tracking data 214 shown in Figure 2.

[0105]

[0113] A user of the XR system 100 can use the hand 402 to point at the computing device 150 (and / or touch the computing device 150 with the fingers of the hand 402) for locating the computing device 150. In some examples, rather than the XR system 100 sending an identification trigger to trigger the computing device 150 to generate an identification output for locating the computing device 150, the user can trigger the locating and mapping of the computing device 150 by a hand gesture (e.g., by pointing at and / or touching the computing device 150) as shown in FIG. 4. In some cases, the XR system 100 can dynamically detect the hand gesture and trigger the locating. In other cases, the user of the XR system 100 can trigger the detection (and locating) of the hand gesture by interacting with an interface rendered by the XR application 202, such as selecting a control or setting in the interface rendered by the XR application 202.

[0106]

[0114] To identify the location of the computing device 150, one or more of the image sensors 102 of the XR system 100 can capture an image of the hand 402 performing a gesture, such as pointing to the computing device 150, touching the computing device 150, etc. In other examples, to identify the location of the computing device 150, one or more image sensors 102 can capture an image of the computing device 150 and / or any other part of the user. For example, one or more image sensors 102 can capture an image of the computing device 150 after it is positioned by the user within the field of view of the one or more image sensors 102. As another example, one or more image sensors 102 can capture one or more images of the user's eyes or head to track the user's gaze and / or head pose.

[0107]

[0115] The XR system 100 can use the action tracking engine 414 for action tracking (e.g., tracking an action, activity, or event). For example, the action tracking engine 414 can use the image of the hand 402 (and / or any other image as described above) to perform hand tracking to estimate the pose of the hand 402 in 3D space. In some examples, the XR system 100 can also use the tracking data 412 from the tracking engine 410 for action tracking (e.g., hand tracking, head pose tracking, gaze tracking, gesture tracking, computing device tracking, tracking other actions by the user, etc.). For example, the action tracking engine 414 of the XR system 100 can perform hand tracking using images of the hand 402 from one or more of the image sensors 102, a 3D map of the environment from the tracking data 412, and the location and pose of the XR system 100 identified in the tracking data 412.

[0108]

[0116] The localization engine 416 of the XR system 100 can use the action tracking results (e.g., hand tracking, object tracking, user action tracking, gesture tracking, etc.) from the action tracking engine 414 and the tracking data 412 from the tracking engine 410 to perform localization of the computing device 150. The localization can identify the location of the computing device 150 in the 3D environment / space. Based on the localization, the XR system 100 can generate localization data 418 that identifies the location of the computing device 150. The XR application 202 can use the localization data 418 to update the device map 222 to include the location of the computing device 150. In some examples, the XR application 202 can tag a location in the device map 222 that corresponds to the determined location of the computing device 150 with the computing device 150 identified by the XR system 100. The XR system 100 can use the updated device map to enable controlling and interacting with the computing device 150 as previously described.

[0109]

[0117] In some cases, the localization engine 416 may use triangulation or any localization algorithm to locate the computing device 150. In some examples, to locate the computing device 150, the localization engine 416 may correlate a region of interest (ROI) tracked by the tracking engine 414 to the environment map 220 to correlate an action (e.g., gesture, etc.) captured in the ROI with a 3D environment / space. For example, the localization engine 416 may use tracking results (e.g., determined pose information, movement information, etc.) from the action tracking engine 414 and tracking data 412 from the tracking engine 410 to determine an action (e.g., gesture, etc.) captured in the ROI and, in some cases, corresponding movement and / or position information. In some examples, the action may indicate a location of the computing device 150. The localization engine 416 can use the corresponding movement and / or position information to determine the position of the action and / or the object associated with the action (e.g., hand, finger, head, eye, another user body part, computing device 150, and / or any other object) in the 3D environment / space. Example localization engine 416 can use the position of the action and / or the associated object to determine the location of computing device 150 in the coordinate system of XR system 100. In some aspects, localization engine 416 can use the location of computing device 150 in the coordinate system of XR system 100 to correlate computing device 150 with a location in the environment map 220 and / or device map 222.

[0110]

[0118] In some cases, the XR system 100 can use one or more computing devices to locate one or more computing devices and / or the XR system 100 relative to each other and map one or more computing devices. For example, the XR system 100 can use one or more computing devices to locate the XR system 100 and one or more computing devices relative to the XR system 100 and map one or more computing devices based on the location information. In some examples, the XR system 100 can remotely trigger the computing device(s) to enter a mapping mode. Based on the trigger, the computing device(s) can begin looking for an identification output from the XR system 100. Upon detecting an identification output from the XR system 100, the computing device(s) can locate itself relative to the XR system 100 and relay the location information to the XR system 100. The XR system 100 can convert the location information into the XR system 100's map of devices (e.g., device map 222) and identify / associate the computing device(s) with the determined location(s).

[0111]

[0119] 5 illustrates an example location 500 by the XR system 100 using computing devices 530, 532, and 534. The computing devices 530, 532, and 534 can include smart devices (e.g., IoT / connected devices, etc.), such as the computing device 150. Moreover, each of the computing devices 530, 532, and 534 can include one or more sensors, such as, for example, but not limited to, image sensor(s), audio sensor(s), IMU, radar, pressure sensor, etc. In this illustrative example, the computing devices 530 and 534 are shown as camera devices with image sensors for capturing image / video data, and the computing device 532 is shown as a smart television with at least one audio sensor. However, in other examples, the computing devices 530, 532, and 534 can include other types of devices with other types of sensors (instead of or in addition to the above-mentioned sensors).

[0112]

[0120] The computing devices 530, 532, and 534 can use their sensors to detect identification information output from the XR system 100 and determine their own location relative to the XR system 100, as described further herein. For example, when a user 502 wearing the XR system 100 moves from the computing devices 530, 532, and 534 into the FOV of each computing device, the computing devices can detect identification information output from the XR system 100 and determine their own location relative to the XR system 100. The computing devices can send location information to the XR system 100, which can convert the location information into a device map (e.g., device map 222) in the XR system 100 to obtain a mapping of each computing device.

[0113]

[0121] 5, the XR system 100 can send an identification trigger 504 to the computing device 530 to remotely trigger the computing device 530 to enter the mapping mode. In other examples, the computing device 530 can enter the mapping mode periodically or based on another event / trigger. For example, in some cases, the computing device 530 may enter the mapping mode upon detecting that the XR system 100 is within the FOV of the computing device 530, upon detecting that the XR system 100 is within a threshold range of the computing device 530, upon detecting a spoken command from the user 502 requesting the computing device 530 to enter the mapping mode, upon detecting a voice from the user 502 indicating that the user 502 may be using the computing device 530 and may be within proximity to the computing device 530, upon receiving a communication from another computing device (e.g., computing device 532, computing device 534) identifying the detected location of the XR system 100 indicating that the XR system 100 may be moving towards the location of the computing device 530 (and / or may be within the FOV and / or range of the computing device 530 at a future time and / or within a future time window), and / or upon any other event / trigger.

[0114]

[0122] After sending the identification trigger 504, the XR system 100 can generate an identification output 510 for detection by the computing device 530. In this example, the identification output 510 represents or includes a pattern displayed by the XR system 100 (e.g., by a display on the XR system 100 and visible to the computing device 530) or on the XR system 100 (e.g., a pattern on the XR system 100 itself). However, in other examples, the identification output 510 can include any other pattern (e.g., a light pattern, an audio pattern, a combination of patterns, etc.), as previously described with respect to the identification output 206 shown in FIG. 2.

[0115]

[0123] In some examples, the XR system 100 can output the identification output 510 once after sending the identification trigger 504. For example, the XR system 100 can output the identification output 510 and not output additional identification outputs until a certain amount of time has passed or the XR system 100 moves to a different location / position (and / or a threshold distance and / or position change). In some cases, the XR system 100 can output the identification output 510 and maintain the identification output 510 (e.g., continue to emit the identification output 510, continue to display the identification output 510, etc.) for a period of time. In other examples, the XR system 100 can output the identification output 510 multiple times (e.g., periodically, etc.).

[0116]

[0124] Upon receiving the identification trigger 504 (or upon detecting any other event / trigger configured to trigger the mapping mode, as previously described), the computing device 530 may enter the mapping mode. Once the computing device 530 enters the mapping mode, the computing device 530 may begin searching for an identification output 510 from the XR system 100. As shown, at time t1, the XR system 100 is within the FOV of the computing device 530. Since the identification output 510 in this example is a display pattern, the computing device 530 may detect the identification output 510 when the XR system 100 is within the FOV of the computing device 530 at time t1. However, in other examples, the identification output 510 may include another type of pattern that may be detected when the XR system 100 is not within the FOV of the computing device 530, such as an audio pattern, a pattern of output light levels, etc. In such an example, the computing device 530 may detect the identification information output 510 when the XR system 100 is within a certain range (which may vary based on one or more factors as previously described) of the computing device 530, even if the XR system 100 is not within the FOV of the computing device 530.

[0117]

[0125] The computing device 530 can detect the identification output 510 from the XR system 100 and use the identification output 510 to locate itself in 3D space relative to the XR system 100. For example, the computing device 530 can locate its relative location to the XR system 100 in an internal coordinate system of the computing device 530. In some examples, the computing device 530 can identify and / or locate the XR system 100 based on the identification output 510 and sensor data from one or more sensors of the computing device 530. In some cases, the computing device 530 can use one or more sensors of the computing device 530 to track / map its location in 3D space and determine its location relative to the XR system 100 in the internal coordinate system of the computing device 530. In other cases, the computing device 530 can be pre-configured with location information identifying its location in 3D space. In such an example, the computing device 530 may use the pre-configured location information to determine its location relative to the XR system 100.

[0118]

[0126] After the computing device 530 determines its own position relative to the XR system 100, the computing device 530 can send positioning data 520 to the XR system 100. The positioning data 520 can indicate the location of the computing device 530 relative to the XR system 100. For example, the positioning data 520 can indicate that, within the coordinate system of the computing device 530, the computing device 530 is a distance above / below and / or a distance away from the XR system 100 (e.g., in a range / proximity). In some cases, when indicating a distance away from the XR system 100, the positioning data 520 can also indicate a direction of the distance away, such as back, front, left or right, north, south, east or west, or any other indication of direction. In some examples, the positioning data 520 can indicate one or more coordinates of the location of the computing device 150 in 3D space.

[0119]

[0127] The XR system 100 can receive the location data 520 and use the location data 520 to locate and map the computing device 530. In some examples, the XR system 100 can also use the location data 520 to locate itself relative to the computing device 530. Based on the location of the computing device 530 relative to the XR system 100, the XR system 100 can update its device map (e.g., device map 222) to include an indication of the location of the computing device 530. In some examples, the XR system 100 can convert the location information in the location data 520 into its own coordinate system and map. For example, if the location data 520 indicates that the computing device 530 is a distance above / below and a distance away from the XR system 100, the XR system 100 can use such information to determine that the computing device 530 is a distance above / below and a distance away from the current location of the XR system 100. The XR system 100 can determine the location of the computing device 530 within the internal coordinate system of the XR system 100 and update the device map of the XR system 100 to associate the location with the computing device 530.

[0120]

[0128] At time t2, the XR system 100 can send an identification trigger 506 to the computing device 532 to remotely trigger the computing device 532 to enter the mapping mode. As mentioned above, in other examples, the computing device 532 can enter the mapping mode periodically or based on another event / trigger. After sending the identification trigger 506, the XR system 100 can generate an identification output 512 for detection by the computing device 532. In this example, the identification output 512 represents or includes an audio pattern emitted by the XR system 100 (e.g., by a speaker on the XR system 100). However, in other examples, the identification output 512 can include any other pattern (e.g., a light pattern, a display pattern, a combination of patterns, etc.), as previously described.

[0121]

[0129] Upon receiving the identification trigger 506 (or upon detecting any other event / trigger configured to trigger the mapping mode, as previously described), the computing device 532 may enter the mapping mode. Once the computing device 532 enters the mapping mode, the computing device 532 may begin to search for the identification output 512 from the XR system 100. In this example, the computing device 532 may use an audio sensor to capture / detect an audio pattern in the identification output 512. The computing device 532 may detect the identification output 512 from the XR system 100 and use the identification output 512 to locate itself in 3D space relative to the XR system 100, as previously described.

[0122]

[0130] After the computing device 532 determines its own position relative to the XR system 100, the computing device 532 can send positioning data 522 to the XR system 100. The positioning data 522 can indicate the location of the computing device 532 relative to the XR system 100. The XR system 100 can receive the positioning data 522 and use the positioning data 522 to locate and map the computing device 532. In some examples, the XR system 100 can also use the positioning data 522 to determine its own position relative to the computing device 532.

[0123]

[0131] Based on the location of the computing device 532 relative to the XR system 100, the XR system 100 can then update its device map (e.g., device map 222) to include an indication of the location of the computing device 532. In some examples, the XR system 100 can convert the location information in the localization data 522 into its own coordinate system and map. For example, if the localization data 522 indicates that the computing device 532 is a distance above / below and away from the XR system 100, the XR system 100 can use such information to determine that the computing device 532 is a distance above / below and away from the current location of the XR system 100. The XR system 100 can determine the location of the computing device 532 within the internal coordinate system of the XR system 100 and update the device map of the XR system 100 to associate the location with the computing device 532.

[0124]

[0132] To initiate an attempt to locate the computing device 534, the XR system 100 can send an identification trigger 508 to the computing device 534 to remotely trigger the computing device 534 to enter a mapping mode. In other examples, the computing device 534 can enter the mapping mode periodically or based on another event / trigger.

[0125]

[0133] After sending the identification trigger 508, the XR system 100 can generate an identification output 514 for detection by the computing device 534. In this example, the identification output 514 represents or includes a light pattern emitted by the XR system 100 (e.g., by a light emitting device on the XR system 100). However, in other examples, the identification output 514 can include any other pattern (e.g., a display pattern, an audio pattern, a combination of patterns, etc.), as previously described.

[0126]

[0134] Upon receiving the identification trigger 508 (or upon detecting any other event / trigger configured to trigger the mapping mode, as previously described), the computing device 534 may enter the mapping mode. Once the computing device 534 enters the mapping mode, the computing device 534 may begin searching for an identification output 514 from the XR system 100. As shown, at time t3, the computing device 534 may detect the identification output 514 from the XR system 100. The computing device 534 may detect the identification output 514 from the XR system 100 and use the identification output 514 to locate itself in 3D space relative to the XR system 100. For example, the computing device 534 may locate its relative location relative to the XR system 100 in the internal coordinate system of the computing device 534.

[0127]

[0135] After the computing device 534 determines its own position relative to the XR system 100, the computing device 534 can send the positioning data 524 to the XR system 100. The positioning data 524 can indicate the location of the computing device 534 relative to the XR system 100. The XR system 100 can receive the positioning data 524 and use the positioning data 524 to locate and map the computing device 534. In some examples, the XR system 100 can also use the positioning data 524 to determine its own position. Based on the location of the computing device 534 relative to the XR system 100, the XR system 100 can update its device map (e.g., device map 222) to include an indication of the location of the computing device 534. In some examples, the XR system 100 can convert the location information in the positioning data 524 into its own coordinate system and map. The XR system 100 can determine the location of the computing device 534 within the internal coordinate system of the XR system 100 and update the device map of the XR system 100 to associate the location with the computing device 534.

[0128]

[0136] In some examples, the XR system 100 can implement location determination 500 to efficiently locate and map several devices. For example, if a user wants to locate all smart devices in a particular location, such as the user's home, the user can move around the location while wearing the XR system 100. The XR system 100 can implement location determination 500 to locate the smart devices as the user moves to different areas of the location. The XR system 100 can use the location information to map each of the smart devices in the particular location. Knowing the location of each of the smart devices based on the mapping of the smart devices, the XR system 100 can enable the user to use the XR system 100 to interact with and control any of the smart devices. In some examples, a user wearing the XR system 100 can use gestures and / or input devices to interact with or control any of the smart devices using the XR system 100. In some examples, a user wearing the XR system 100 can interact with an interface on the XR system 100 to control or otherwise interact with any of the smart devices.

[0129]

[0137] 5 uses multiple computing devices (e.g., computing devices 530, 532, and 534), other examples of location location 500 may be implemented using the XR system 100 and a single other computing device (e.g., computing device 530, computing device 532, or computing device 534). In other examples, location location 500 shown in FIG. 5 may be implemented using more or fewer computing devices than those shown in FIG. 5.

[0130]

[0138] 6 is a diagram illustrating an example joint localization 600 by a device based on a common landmark. The XR system 100 can send an identification trigger 602 to the computing device 610 to trigger the computing device 610 to enter a mapping mode. In this example, the computing device 610 represents a camera device. However, the computing device 610 can include any smart device (e.g., IoT / connected device, etc.), such as the computing device 150.

[0131]

[0139] The computing device 610 can receive an identification trigger 602 from the XR system 100 and enter a mapping mode. In the mapping mode, the computing device 610 can locate the landmark 620 relative to its own coordinate system. For example, the computing device 610 can obtain image data capturing the landmark 620 and process the image data using one or more localization algorithms to determine the location of the landmark 620 relative to the computing device 610. The computing device 610 can then send location information 604 to the XR system 100. In some cases, the location information 604 can include localization data as previously described. In some examples, the location information 604 can describe and / or identify the location of the landmark 620 relative to the computing device 610. The XR system 100 can use the location of the landmark 620 described and / or identified in the location information 604 to locate the computing device 610, as further described herein. In some cases, the location information 604 may describe and / or identify a location of the landmark 620 within or relative to the coordinate system of the computing device 610 or the XR system 100. For example, the location information 604 may describe and / or identify a location of the landmark 620 in the coordinate system of the computing device 610. The XR system 100 may transform the location of the landmark 620 from the coordinate system of the computing device 610 to the coordinate system of the XR system 100, as described further below.

[0132]

[0140] The XR system 100 can also identify the position of the landmark 620 to determine the location of the landmark 620 relative to the XR system 100 and / or the coordinate system of the XR system 100. The XR system 100 can use the location information 604 from the computing device 610 and the location of the landmark 620 relative to the XR system 100 and / or the coordinate system of the XR system 100 to identify the location of the computing device 610. In some examples, the XR system 100 can use the location of the landmark 620 relative to the XR system 100 (and / or the coordinate system of the XR system 100) to convert the location of the landmark 620 relative to the computing device 610 (and / or the coordinate system of the computing device 610) to the location of the computing device in the coordinate system of the XR system 100. For example, the XR system 100 can determine the location of the computing device 610 relative to the coordinate system of the XR system 100 based on the location of the landmark 620 relative to the coordinate system of the XR system 100 and the location of the landmark 620 relative to the coordinate system of the computing device 610. In this example, the XR system 100 can convert the location of the computing device 610 relative to the coordinate system of the computing device 610 to the location of the computing device 610 relative to the coordinate system of the XR system 100.

[0133]

[0141] Based on the location of the computing device 610 relative to the XR system 100, the XR system 100 can map the computing device 610. For example, the XR system 100 can update a device map (e.g., device map 222) in the XR system 100 to associate the location of the computing device 610 with the computing device 610.

[0134]

[0142] In some cases, the location information 604 from the computing device 610 may include a feature vector describing features of the landmark 620. For example, the computing device 610 may extract features corresponding to the landmark 620 from image data capturing the landmark 620. The computing device 610 may then provide the extracted features to the XR system 100. The XR system 100 may use the features from the computing device 610 to verify that the location information 604 from the computing device 610 corresponds to the landmark 620 and to ensure that the localization information determined by the XR system 100 for a landmark (e.g., the landmark 620) and the location information 604 from the computing device 610 correspond to the same landmark. For example, the XR system 100 may determine a feature vector associated with the landmark 620. The XR system 100 may compare the feature vector it estimates with the feature vector from the computing device 610. The XR system 100 may verify that the feature vector from the computing device 610 corresponds to the same landmark as the feature vector estimated by the XR system 100. For example, the XR system 100 can determine whether the feature vectors match and / or have correspondence.

[0135]

[0143] The XR system 100 can verify that the feature vectors correspond to the same landmarks to ensure that the XR system 100 does not rely on incorrect localization information to locate the computing device 610. For example, the XR system 100 can ensure that it locates the computing device 610 based on the locations of the XR system 100 and the computing device 610 relative to the same landmarks.

[0136]

[0144] As previously described, the XR system 100 can map controllable devices to allow a user to control and / or interact with the controllable devices via the XR system 100 using several types of user inputs, such as gestures and XR inputs. FIG. 7 illustrates an exemplary use case 700 for controlling a computing device 150 using the XR system 100. In this example, the computing device 150 represents a smart light bulb. The XR system 100 can detect an input from a user of the XR system 100 requesting to change the state of the computing device 150 from a first state 710 to a second state 720.

[0137]

[0145] For example, the XR system 100 can detect a gesture by a user, such as pointing (e.g., via a finger, controller, hand, head, input device, etc.) at the computing device 150, waving in a direction and / or location (e.g., close to the computing device 150), maintaining gaze toward the computing device 150, and / or any other gesture. The XR system 100 can interpret the gesture as an input to adjust the state of the computing device 150 from a first state 710 to a second state 720. In another example, the XR system 100 can receive an input from a user through an interface rendered by the XR system 100. The input from the user can include a request to adjust the state of the computing device 150 from the first state 710 to the second state 720.

[0138]

[0146] Based on the input from the user, the XR system 100 can generate a command 702 to trigger the computing device 150 to adjust the state of the computing device 150 from a first state 710 to a second state 720. The XR system 100 can send the command 702 to the computing device 150. The computing device 150 can receive the command 702 and execute the command 702 to change the state of the computing device 150 to the second state 720.

[0139]

[0147] As mentioned above, the command 702 includes a command to change the state of the computing device 150 from a first state 710 to a second state 720. The first state 710 shown in this example is an off state, and the second state 720 shown in this example is an on state. Thus, as shown in FIG. 7, the XR system 100 can use the command 702 to turn on the smart light bulb represented by the computing device 150. In other examples, the command 702 can include a command to change the operation / state of the smart light bulb represented by the computing device 150 to any other state. For example, the command 702 can include a command to turn off the smart light bulb (from an on state), change the light level of the smart light bulb, change the color temperature of the smart light bulb, start / stop audio playback (e.g., sound, music, etc.) if the smart light bulb has audio capabilities, trigger communication and / or connection by the smart light bulb with another device(s), generate a particular light pattern(s), or implement any other state / action or combination of states / actions.

[0140]

[0148] The XR system 100 can similarly use commands to control the state / operation of any other type of computing device 150, such as, for example, a smart speaker, a smart television, a smart lock, a smart refrigerator, a smart camera, a smart wearable device, a smart security system, a smart thermostat, a smart sensor, a smart fitness / health tracker, a smart hub, a smart switch (e.g., a smart light switch, etc.), a smart appliance, a smart plug, a television, etc.

[0141]

[0149] 8 is a flow chart illustrating an example process 800 for locating and mapping a connected or networked device (e.g., a smart device such as the computing device 150). At block 802, the process 800 may include receiving, by the electronic device (e.g., the XR system 100), an identification output from a connected device (e.g., the computing device 150) that is directly or indirectly coupled to the electronic device. In some cases, the identification output may be reflected in or encoded in a wireless signal from the connected device. In some examples, the identification output may include an audio pattern, a display pattern, a light pattern, a wireless pattern, and / or a combination thereof. For example, the identification information output may include a pattern of light levels (e.g., a pattern of dimming and brightening light levels), a pattern of color temperature changes, a sound pattern (e.g., noise, music, Morse code, a sequence of tones, etc.), a pattern or code encoded in a wireless signal (e.g., a Bluetooth signal, a WiFi signal, etc.), a display pattern (e.g., an image, a Quick Response (QR) code, a bar code, one or more displayed characters / symbols, a digital content item, a digital object, a video, a virtual content, a digital rendering, a graphic, a communication code, etc.), combinations thereof, and the like.

[0142]

[0150] In some cases, the light pattern may include a pattern of light dimming and brightening levels emitted by the electronic device, a pattern of color temperature change of light emitted by the electronic device, and / or a sequence of blinking lights emitted by the electronic device. In some cases, the light pattern may include light that is visible or invisible to the human eye. For example, the light pattern may include IR light or a pattern of IR light. In some cases, the display pattern may include a code (e.g., a QR code, a bar code, a key, etc.) displayed on a display device of the electronic device, an image displayed on a display device of the electronic device, a visual pattern displayed on a display device of the electronic device, and / or a digital object displayed on a display device of the electronic device. In some cases, the audio pattern may include an audio code output by the electronic device and / or a sequence of sounds output by the electronic device. In some cases, the communication code may be generated from a communication interface of the electronic device.

[0143]

[0151] In some examples, the electronic device can trigger an identification output from a connected device. In some cases, the electronic device can trigger an identification output based on an identification trigger (e.g., identification trigger 204) sent to the connected device. For example, the electronic device can send an identification trigger to the connected device to trigger an identification output from the connected device. To illustrate, the electronic device can send a signal to the connected device configured to trigger an identification output from the connected device and receive an identification output from the connected device. In some cases, the electronic device can receive one or more identification outputs from a plurality of identification outputs broadcast by the connected device. The one or more identification outputs can include an identification output.

[0144]

[0152] In some aspects, receiving the identification information output can include detecting the identification information output from the connected device. In some examples, the electronic device can detect the identification information output using one or more sensors on the electronic device, such as an image sensor(s), a light sensor, and / or an audio sensor(s). For example, the electronic device can detect the identification information output based on audio data of the identification information output captured by an audio sensor at the electronic device. As another example, the electronic device can detect the identification information output based on image data of the identification information output captured by an image sensor at the electronic device.

[0145]

[0153] At block 804, the process 800 may include mapping the connected device in a coordinate system of the electronic device based on the identification information output from the connected device. In some examples, the electronic device may update a device map (e.g., device map 222) to identify the location of the connected device.

[0146]

[0154] In some examples, process 800 may include determining a location of the connected device relative to the electronic device based on the identification information output, and mapping the connected device in a coordinate system of the electronic device based on the location of the connected device.

[0147]

[0155] In some examples, process 800 can include sending, by the electronic device, to the connected device a signal (e.g., a message, packet, command, output, etc.) configured to trigger an additional identification output from the connected device, sending, by the electronic device, one or more additional signals at one or more different times to the connected device based on failure to receive the additional identification output, receiving, from the connected device, the one or more additional signals configured to trigger the additional identification output from the connected device, determining a location of the connected device based on the additional identification output, and updating a device map (e.g., device map 222) to associate the connected device with the location of the connected device. In some examples, mapping the connected device in a coordinate system of the electronic device can include updating the device map to associate the connected device with the location of the connected device.

[0148]

[0156] In some cases, sending one or more additional signals at one or more different times may include determining a failure to receive the additional identification information output while the electronic device is within a threshold proximity to a previously determined location (e.g., a mapped location) of the connected device and / or the previously determined location (e.g., a mapped location) of the connected device is within a field of view of an image sensor of the electronic device, and sending one or more additional signals to the connected device based on the failure to receive the additional identification information output.

[0149]

[0157] In some examples, sending one or more additional signals at one or more different times may include periodically sending one or more additional signals until a predetermined amount of time has elapsed without receiving an additional identification information output from the connected device and / or until a number of additional signals have been sent without receiving an additional identification information output from the connected device.

[0150]

[0158] In some examples, receiving an identification output from the connected device can include detecting one or more patterns in the identification output (e.g., an audio pattern, a display pattern, a pattern in a wireless signal, and / or a light pattern) and recognizing the one or more patterns as the identification output from the connected device. In some cases, the electronic device can detect the one or more patterns based on data from one or more sensors of the electronic device and recognize the one or more patterns (e.g., an audio pattern, a display pattern, a pattern in a wireless signal, and / or a light pattern) as the identification output. In some examples, the one or more sensors can include an image sensor(s), an audio sensor(s), and / or a light sensor(s). In some cases, the electronic device can recognize the one or more patterns (e.g., an audio pattern, a display pattern, a pattern in a wireless signal, and / or a light pattern) using a neural network, visual feature matching, and / or audio feature matching.

[0151]

[0159] In some examples, process 800 can include receiving a request to trigger an action of a connected device (e.g., adjust a state, perform an operation, implement a setting, etc.), identifying a location of the connected device based on the mapping of the connected device, and sending, by the electronic device, a command to the connected device configured to trigger an action by the connected device. In some examples, the request to trigger an action by the connected device can include a request to send a video feed captured by the connected device to the electronic device, and process 800 can include sending the command to the connected device, receiving the video feed from the connected device, and displaying the video feed at the electronic device. The command can be configured to trigger the connected device to send the video feed to the electronic device.

[0152]

[0160] In some examples, the request to trigger an action by the connected device may include a request to trigger the connected device to display content from the electronic device (e.g., virtual content or any other content rendered and / or generated by the electronic device), and process 800 may include sending data to the connected device including the command and the content from the electronic device. In some cases, the command is configured to trigger the connected device to display the content from the electronic device.

[0153]

[0161] In some cases, the request to trigger an action by the connected device may include a request to adjust a power mode of the connected device (e.g., turn on the connected device, turn off the connected device, set the connected device to a sleep state, reduce or increase the power or operating mode of the connected device, etc.), and the command may be configured to adjust the power mode of the connected device.

[0154]

[0162] In some examples, the connected device may include a light bulb, and the request to trigger an action by the connected device may include a request to adjust the brightness and / or color of the light emitted by the light bulb. In some cases, the command is configured to trigger the light bulb to adjust the brightness and / or color of the light emitted by the light bulb.

[0155]

[0163] In some examples, the process 800 can include detecting a gesture by a user associated with the electronic device, triggering one or more identification outputs from one or more connected devices in the physical environment based on the gesture, receiving a particular identification output from the one or more identification outputs, and determining that the connected device is located at a location based on the particular identification output. In some examples, the gesture can indicate a location of the connected device. In some examples, the particular identification output originates from the location. In some examples, the process 800 can include updating a device map to associate the connected device with the location of the connected device. In some examples, the gesture can include a hand gesture, a gaze, a head pose, pointing, placing the connected device within the FOV of the electronic device, touching the connected device, and / or any other gesture.

[0156]

[0164] In some examples, process 800 may include triggering one or more connected devices to enter a mapping mode, sending a device identification output to the one or more connected devices, receiving location information from the one or more connected devices, and mapping the one or more connected devices in a coordinate system of the electronic device based on the location information from the one or more connected devices. In some cases, the location information may indicate a location of the one or more connected devices relative to the electronic device.

[0157]

[0165] In some examples, process 800 may include triggering one or more connected devices to enter a mapping mode, receiving location information from the one or more connected devices, determining second locations of the one or more connected devices based on the location information and a third location of a landmark relative to the electronic device, and mapping the one or more connected devices in a coordinate system of the electronic device based on the second locations of the one or more connected devices. In some cases, the location information may indicate a first location of the landmark relative to the connected device.

[0158]

[0166] In some examples, the process 800 may be performed by one or more computing devices or apparatuses. In one illustrative example, the process 800 may be performed by the XR system 100 shown in FIG. 1. In some examples, the process 800 may be performed by one or more computing devices having a computing device architecture 900 shown in FIG. 9. In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of the process 800. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data and / or other sensor measurements. For example, the computing device may include a smartphone, a head-mounted display, a mobile device, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying the images. In some examples, the one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to communicate data.

[0159]

[0167] The components of a computing device may be implemented in circuits. For example, the components may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry) and / or may include and / or be implemented using electronic circuitry or other electronic hardware that may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The computing device may further include a display (as an example of an output device, or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0160]

[0168] Process 800 is illustrated as a logical flow diagram, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0161]

[0169] Further, process 800 may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0162]

[0170] 9 illustrates an exemplary computing device architecture 900 of an exemplary computing device that can implement various techniques described herein. For example, the computing device architecture 900 can implement at least some portions of the XR system 100 shown in FIG. 1 or the computing device 150 shown in FIG. 1. The components of the computing device architecture 900 are shown in electrical communication with each other using a connection 905, such as a bus. The exemplary computing device architecture 900 includes a processing unit (CPU or processor) 910 and a computing device connection 905 that couples various computing device components, including a computing device memory 915, such as a read only memory (ROM) 920 and a random access memory (RAM) 925, to the processor 910.

[0163]

[0171] The computing device architecture 900 may include a cache of high speed memory directly connected to the processor 910, in close proximity to the processor 910, or integrated as part of the processor 910. The computing device architecture 900 may copy data from the memory 915 and / or storage device 930 to the cache 912 for quick access by the processor 910. In this manner, the cache may provide performance improvements that avoid processor 910 delays while waiting for data. These and other modules may control or be configured to control the processor 910 to perform various actions. Other computing device memories 915 may also be available for use. The memory 915 may include multiple different types of memory with different performance characteristics. The processor 910 may include any general purpose processor and hardware or software services stored in the storage device 930 and configured to control the processor 910, as well as special purpose processors where software instructions are built into the processor design. The processor 910 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0164]

[0172] To enable user interaction with the computing device architecture 900, the input device 945 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 935 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 900. The communication interface 940 can generally govern and manage user input and computing device output. There is no restriction to operating on any particular hardware configuration, and thus the basic features herein can be easily substituted with improved hardware or firmware configurations as they are developed.

[0165]

[0173] The storage device 930 is a non-volatile memory and may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 925, a read only memory (ROM) 920, and hybrids thereof. The storage device 930 may include software, code, firmware, etc. for controlling the processor 910. Other hardware or software modules are contemplated. The storage device 930 may be connected to the computing device connections 905. In one aspect, a hardware module that performs a particular function may include software components stored in a computer readable medium in relation to the necessary hardware components, such as the processor 910, the connections 905, the output device 935, etc., to perform that function.

[0166]

[0174] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0167]

[0175] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0168]

[0176] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks comprising devices, device components, steps or routines in a method implemented in software, or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0169]

[0177] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when an operation of a process is completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0170]

[0178] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.

[0171]

[0179] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.

[0172]

[0180] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0173]

[0181] In the above description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various ways, and the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be considered illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0174]

[0182] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0175]

[0183] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0176]

[0184] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communications interface).

[0177]

[0185] Claim language or other language in this disclosure reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not recited in the set of A and B.

[0178]

[0186] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0179]

[0187] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0180]

[0188] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0181]

[0189] Illustrative examples of the present disclosure include the following:

[0182]

[0190] Aspect 1. An apparatus for locating one or more devices, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors configured to: receive an identification output from a connected device coupled to the apparatus; and map the connected device in a coordinate system of the apparatus based on the identification output from the connected device, the identification output comprising at least one of an audio pattern, a display pattern, and a light pattern.

[0183]

[0191] Aspect 2. The apparatus of aspect 1, wherein the identification information output comprises at least one of an audio pattern, a display pattern, a light pattern, and a wireless signal.

[0184]

[0192] Aspect 3. An apparatus as described in any of aspects 1 to 2, wherein the one or more processors are configured to determine a location of the connected device based on the identification information output, and to map the connected device in a coordinate system of the apparatus based on the location of the connected device.

[0185]

[0193] Aspect 4. The apparatus of any of aspects 1 to 3, wherein the one or more processors are configured to: send a signal to the connected device configured to trigger an additional identification information output from the connected device; send one or more additional signals to the connected device at one or more different times, receive an additional identification information output from the connected device, where the one or more additional signals are configured to trigger an additional identification information output from the connected device; and determine a location of the connected device based on the additional identification information output, wherein the one or more processors are configured to update a device map to associate the connected device with a location of the connected device to map the connected device in a coordinate system of the apparatus.

[0186]

[0194] Embodiment 5. The device of any of embodiments 1 to 4, wherein to send one or more additional signals at one or more different times, the one or more processors are configured to: determine a failure to receive the additional identification information output while the device is within a threshold proximity to the mapped location of the connected device and the mapped location of the connected device is within a field of view of an image sensor of the device; and send one or more additional signals to the connected device based on the failure to receive the additional identification information output.

[0187]

[0195] Embodiment 6. The apparatus of any of embodiments 1 to 5, wherein to send one or more additional signals at one or more different times, the one or more processors are configured to periodically send one or more additional signals until at least one of: a predetermined amount of time has elapsed without receiving an additional identification information output from the connected device; and a number of additional signals have been sent without receiving an additional identification information output from the connected device.

[0188]

[0196] Aspect 7. An apparatus as described in any of aspects 1 to 6, wherein to receive an identification information output from a connected device, the one or more processors are configured to detect one or more patterns in the identification information output based on data from one or more sensors of the apparatus, and recognize the one or more patterns as an identification information output from the connected device.

[0189]

[0197] Embodiment 8. An apparatus as described in any of embodiments 1 to 7, wherein the one or more patterns comprise an audio pattern and the one or more sensors comprise an audio sensor.

[0190]

[0198] Embodiment 9. An apparatus according to any of embodiments 1 to 8, wherein the one or more patterns comprise a display pattern and the one or more sensors comprise an image sensor.

[0191]

[0199] Embodiment 10. An apparatus according to any of embodiments 1 to 9, wherein the one or more patterns comprise light patterns and the one or more sensors comprise light sensors.

[0192]

[0200] Aspect 11. An apparatus as described in any of aspects 1 to 10, wherein the one or more processors are configured to use at least one of a neural network and feature matching to recognize one or more patterns as identification information output from a connected device.

[0193]

[0201] Aspect 12. An apparatus as described in any of aspects 1 to 11, wherein one or more processors are configured to receive a request to trigger an action by a connected device, identify a location of the connected device based on a mapping of the connected device, and send a command to the connected device configured to trigger an action by the connected device.

[0194]

[0202] Aspect 13. The apparatus of any of aspects 1 to 12, wherein the request to trigger an action by the connected device comprises a request to send a video feed captured by the connected device to the apparatus, wherein the one or more processors are configured to send a command to the connected device, wherein the command is configured to trigger the connected device to send the video feed to the apparatus, receive a video feed from the connected device, and display the video feed at the apparatus.

[0195]

[0203] Aspect 14. The apparatus of any of aspects 1 to 13, wherein the request to trigger an action by the connected device comprises a request to trigger the connected device to display content from the device, and wherein to send the command to the connected device, the one or more processors are configured to send data to the connected device comprising the command and the content from the device, wherein the command is configured to trigger the connected device to display the content from the device.

[0196]

[0204] Aspect 15. The apparatus of any of aspects 1 to 14, wherein the request to trigger an action by the connected device comprises a request to adjust a power mode of the connected device, wherein the command is configured to adjust a power mode of the connected device.

[0197]

[0205] Aspect 16. The apparatus of any of aspects 1 to 15, wherein the connected device comprises a light bulb, and wherein the request to trigger an action by the connected device comprises a request to adjust at least one of a brightness and a color of light emitted by the light bulb, and wherein the command is configured to trigger the light bulb to adjust at least one of the brightness and the color of the light emitted by the light bulb.

[0198]

[0206] Aspect 17. An apparatus as described in any of aspects 1 to 16, wherein one or more processors are configured to detect a gesture by a user associated with the apparatus, trigger one or more identification information outputs from one or more connected devices in the physical environment based on the gesture, where the gesture indicates a location of the connected device, receive a particular identification information output from the one or more identification information outputs, and determine that the connected device is located at the location based on the particular identification information output, where the particular identification information output originates from the location.

[0199]

[0207] Aspect 18. The apparatus of any of aspects 1 to 17, wherein the one or more processors are configured to update the device map to associate the connected device with a location of the connected device in a coordinate system of the apparatus to map the connected device.

[0200]

[0208] Aspect 19. A device as described in any of aspects 1 to 18, wherein the gesture comprises at least one of pointing to one or more connected devices, touching one or more connected devices, placing one or more connected devices within a field of view of an image sensor of the device, a hand gesture, and a gaze.

[0201]

[0209] Aspect 20. An apparatus as described in any of aspects 1 to 19, wherein the one or more processors are configured to: receive a voice command by a user associated with the apparatus; trigger one or more identification information outputs from one or more connected devices in the physical environment based on the voice command, the voice command indicating a location of the connected device; receive a particular identification information output from the one or more identification information outputs; and determine that the connected device is located at the location based on the particular identification information output, the particular identification information output emanating from the location.

[0202]

[0210] Aspect 21. An apparatus as described in any of aspects 1 to 20, wherein the one or more processors are configured to trigger one or more connected devices to enter a mapping mode, send a device identification information output to the one or more connected devices, receive location information from the one or more connected devices, and map the one or more connected devices in a coordinate system of the apparatus based on the location information from the one or more connected devices, where the location information indicates a location of the one or more connected devices relative to the apparatus.

[0203]

[0211] Aspect 22. An apparatus as described in any of aspects 1 to 21, wherein the one or more processors are configured to: trigger one or more connected devices to enter a mapping mode; receive location information from the one or more connected devices; determine a second location of the one or more connected devices based on the location information, where the location information indicates a first location of a landmark relative to the one or more connected devices, and a third location of the landmark relative to the apparatus; and map the one or more connected devices in a coordinate system of the apparatus based on the second locations of the one or more connected devices.

[0204]

[0212] Aspect 23. A device as described in any of aspects 1 to 22, wherein the identification information output comprises a light pattern, wherein the light pattern comprises at least one of a pattern of dimming and brightening levels of light emitted by the device, a pattern of color temperature changes of the light emitted by the device, and a sequence of blinking light emitted by the device.

[0205]

[0213] Aspect 24. A device as described in any of aspects 1 to 23, wherein the identification information output comprises a display pattern, wherein the display pattern comprises at least one of a code displayed on a display device of the device, an image displayed on a display device of the device, a visual pattern displayed on a display device of the device, and a digital object displayed on a display device of the device.

[0206]

[0214] Aspect 25. The device of any of aspects 1 to 24, wherein the identification information output comprises an audio pattern, wherein the audio pattern comprises at least one of an audio code output by the device and a sequence of sounds output by the device.

[0207]

[0215] Aspect 26. An apparatus as described in any of aspects 1 to 25, wherein the one or more processors are configured to send a signal to a connected device configured to trigger an identification information output from the connected device, and to receive an identification information output from the connected device.

[0208]

[0216] Aspect 27. The apparatus of any of aspects 1 to 26, wherein to receive the identification information output, the one or more processors are configured to receive one or more identification information outputs from a plurality of identification information outputs broadcast by connected devices, the one or more identification information outputs comprising the identification information output.

[0209]

[0217] Aspect 28. An apparatus described in any of aspects 1 to 27, wherein the apparatus is an extended reality device.

[0210]

[0218] Aspect 29. An apparatus described in any of aspects 1 to 28, wherein the apparatus comprises an extended reality device.

[0211]

[0219] Embodiment 30. An apparatus according to any one of embodiments 1 to 29, wherein the apparatus is a mobile device.

[0212]

[0220] Aspect 31. An apparatus as described in any of aspects 1 to 30, wherein the apparatus comprises a mobile device.

[0213]

[0221] Aspect 32. An apparatus as described in any of aspects 1 to 31, wherein the apparatus is at least one of a display configured to display data and a camera configured to capture image data.

[0214]

[0222] Aspect 33. An apparatus as described in any of aspects 1 to 32, wherein the apparatus comprises at least one of a display configured to display data and a camera configured to capture image data.

[0215]

[0223] Aspect 34. An apparatus as described in any of aspects 1 to 33, wherein the connected device is at least one of a light bulb, a speaker, a television, a microphone, one or more sensors, a camera, a thermostat, and a wearable device.

[0216]

[0224] Aspect 35. An apparatus as described in any of aspects 1 to 34, wherein the connected device comprises at least one of a light bulb, a speaker, a television, a microphone, one or more sensors, a camera, a thermostat, and a wearable device.

[0217]

[0225] Embodiment 36. The apparatus of any of embodiments 1 to 35, wherein the identification information output comprises a received radio signal.

[0218]

[0226] Aspect 37. A method for locating one or more devices, the method comprising: receiving an identification output from a connected device coupled to an electronic device; and mapping the connected device in a coordinate system of the electronic device based on the identification output from the connected device.

[0219]

[0227] Aspect 38. The method of aspect 37, wherein the identification information output comprises at least one of an audio pattern, a display pattern, a light pattern, and a wireless signal.

[0220]

[0228] Aspect 39. The method of any of aspects 37 to 38, further comprising determining a location of the connected device based on the identification information output, and mapping the connected device in a coordinate system of the electronic device based on the location of the connected device.

[0221]

[0229] Aspect 40. The method of any of aspects 37 to 39, further comprising: sending a signal to the connected device configured to trigger an additional identification information output from the connected device, sending one or more additional signals to the connected device at one or more different times, receiving an additional identification information output from the connected device, where the one or more additional signals are configured to trigger an additional identification information output from the connected device, and determining a location of the connected device based on the additional identification information output, wherein mapping the connected device in a coordinate system of the electronic device comprises updating a device map to associate the connected device with the location of the connected device.

[0222]

[0230] Aspect 41. The method of any of aspects 37 to 40, wherein sending one or more additional signals at one or more different times comprises determining a failure to receive the additional identification information output while at least one of the electronic device is within a threshold proximity to the mapped location of the connected device and the mapped location of the connected device is within a field of view of an image sensor of the electronic device, and sending one or more additional signals to the connected device based on the failure to receive the additional identification information output.

[0223]

[0231] Aspect 42. The method of any of aspects 37 to 41, wherein sending one or more additional signals at one or more different times comprises periodically sending one or more additional signals until at least one of: a predetermined amount of time has elapsed without receiving an additional identification information output from the connected device; and a number of additional signals are sent without receiving an additional identification information output from the connected device.

[0224]

[0232] Aspect 43. The method of any of aspects 37 to 42, wherein receiving an identification information output from the connected device comprises detecting one or more patterns in the identification information output based on data from one or more sensors of the electronic device, and recognizing the one or more patterns as the identification information output from the connected device.

[0225]

[0233] Embodiment 44. The method of any of embodiments 37 to 43, wherein the one or more patterns comprise an audio pattern and the one or more sensors comprise an audio sensor.

[0226]

[0234] Embodiment 45. The method of any of embodiments 37 to 44, wherein the one or more patterns comprise a display pattern and the one or more sensors comprise an image sensor.

[0227]

[0235] Embodiment 46. The method of any of embodiments 37 to 45, wherein the one or more patterns comprise light patterns and the one or more sensors comprise light sensors.

[0228]

[0236] Aspect 47. A method as described in any of aspects 37 to 46, wherein recognizing one or more patterns as an identification information output from the connected device comprises recognizing one or more patterns as an identification information output using at least one of a neural network and feature matching.

[0229]

[0237] Aspect 48. The method of any of aspects 37 to 47, further comprising receiving a request to trigger an action by a connected device, identifying a location of the connected device based on a mapping of the connected device, and sending a command to the connected device configured to trigger an action by the connected device.

[0230]

[0238] Aspect 49. The method of any of aspects 37 to 48, wherein the request to trigger an action by the connected device comprises a request to send a video feed captured by the connected device to the electronic device, the method further comprising sending a command to the connected device, receiving a video feed from the connected device, wherein the command is configured to trigger the connected device to send the video feed to the electronic device, and displaying the video feed at the electronic device.

[0231]

[0239] Aspect 50. The method of any of aspects 37 to 49, wherein the request to trigger an action by the connected device comprises a request to trigger the connected device to display content from the electronic device, and wherein sending the command to the connected device comprises sending data to the connected device comprising the command and the content from the electronic device, and wherein the command is configured to trigger the connected device to display the content from the electronic device.

[0232]

[0240] Aspect 51. The method of any of aspects 37 to 50, wherein the request to trigger an action by the connected device comprises a request to adjust a power mode of the connected device, wherein the command is configured to adjust a power mode of the connected device.

[0233]

[0241] Aspect 52. The method of any of aspects 37 to 51, wherein the connected device comprises a light bulb, and wherein the request to trigger an action by the connected device comprises a request to adjust at least one of a brightness and a color of light emitted by the light bulb, and wherein the command is configured to trigger the light bulb to adjust at least one of the brightness and the color of the light emitted by the light bulb.

[0234]

[0242] Aspect 53. A method according to any of aspects 37 to 52, further comprising: detecting a gesture by a user associated with the electronic device; triggering one or more identification information outputs from one or more connected devices in the physical environment based on the gesture, the gesture indicating a location of the connected device; receiving a particular identification information output from the one or more identification information outputs; and determining that the connected device is located at the location based on the particular identification information output, the particular identification information output emanating from the location.

[0235]

[0243] Aspect 54. The method of any of aspects 37 to 53, wherein mapping the connected device in a coordinate system of the electronic device comprises updating a device map to associate the connected device with a location of the connected device.

[0236]

[0244] Aspect 55. A method as described in any of aspects 37 to 54, wherein the gesture comprises at least one of pointing to one or more connected devices, touching one or more connected devices, placing one or more connected devices within a field of view of an image sensor of the electronic device, a hand gesture, and a gaze.

[0237]

[0245] Aspect 56. A method according to any of aspects 37 to 55, further comprising receiving a voice command by a user associated with the electronic device, triggering one or more identification information outputs from one or more connected devices in the physical environment based on the voice command, the voice command indicating a location of the connected device, receiving a particular identification information output from the one or more identification information outputs, and determining that the connected device is located at the location based on the particular identification information output, the particular identification information output emanating from the location.

[0238]

[0246] Aspect 57. The method of any of aspects 37 to 56, further comprising triggering one or more connected devices to enter a mapping mode, sending a device identification information output to the one or more connected devices, receiving location information from the one or more connected devices, and mapping the one or more connected devices in a coordinate system of the electronic device based on the location information from the one or more connected devices, where the location information indicates a location of the one or more connected devices relative to the electronic device.

[0239]

[0247] Aspect 58. The method of any of aspects 37 to 57, further comprising: triggering one or more connected devices to enter a mapping mode; receiving location information from the one or more connected devices; determining a second location of the one or more connected devices based on the location information, where the location information indicates a first location of a landmark relative to the one or more connected devices, and a third location of the landmark relative to the electronic device; and mapping the one or more connected devices in a coordinate system of the electronic device based on the second locations of the one or more connected devices.

[0240]

[0248] Aspect 59. A method as described in any of aspects 37 to 58, wherein the identification information output comprises a light pattern, wherein the light pattern comprises at least one of a pattern of dimming and brightening levels of light emitted by the electronic device, a pattern of color temperature changes of light emitted by the electronic device, and a sequence of blinking light emitted by the electronic device.

[0241]

[0249] Aspect 60. The method of any of aspects 37 to 59, wherein the identification information output comprises a display pattern, wherein the display pattern comprises at least one of a code displayed on a display device of the electronic device, an image displayed on a display device of the electronic device, a visual pattern displayed on a display device of the electronic device, and a digital object displayed on a display device of the electronic device.

[0242]

[0250] Aspect 61. The method of any of aspects 37 to 60, wherein the identification information output comprises an audio pattern, wherein the audio pattern comprises at least one of an audio code output by the electronic device and a sequence of sounds output by the electronic device.

[0243]

[0251] Aspect 62. The method of any of aspects 37 to 61, further comprising sending a signal to the connected device configured to trigger an identification information output from the connected device, and receiving the identification information output from the connected device.

[0244]

[0252] Aspect 63. The method of any of aspects 37 to 62, wherein receiving an identification information output comprises receiving one or more identification information outputs from a plurality of identification information outputs broadcast by connected devices, the one or more identification information outputs comprising the identification information output.

[0245]

[0253] Aspect 64. The method of any of aspects 37 to 63, wherein the connected device is at least one of a light bulb, a speaker, a television, a microphone, one or more sensors, a camera, a thermostat, and a wearable device.

[0246]

[0254] Embodiment 65. The method of any of embodiments 37 to 64, wherein the connected device comprises at least one of a light bulb, a speaker, a television, a microphone, one or more sensors, a camera, a thermostat, and a wearable device.

[0247]

[0255] Embodiment 66. The method of any of embodiments 37 to 65, wherein the identification information output comprises a received radio signal.

[0248]

[0256] Embodiment 67. A method according to any of embodiments 37 to 66, wherein the electronic device is or comprises an XR device.

[0249]

[0257] Embodiment 68. The method of any of embodiments 37 to 67, wherein the electronic device is or comprises a mobile device.

[0250]

[0258] Embodiment 69. The method of any one of embodiments 37 to 68, wherein the electronic device comprises a camera.

[0251]

[0259] Embodiment 70. The method of any of embodiments 37 to 69, wherein the electronic device comprises a display device.

[0252]

[0260] Aspect 71. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 37 to 70.

[0253]

[0261] Embodiment 72. An apparatus comprising means for carrying out the method according to any of embodiments 37 to 70.

[0254]

[0262] Embodiment 73. The apparatus of embodiment 72, wherein the apparatus is or comprises an electronic device.

[0255]

[0263] Aspect 74. The apparatus of any of aspects 72 to 73, wherein the apparatus comprises a mobile device.

[0256]

[0264] Embodiment 75. An apparatus according to any of embodiments 72 to 74, wherein the apparatus comprises a camera.

[0257]

[0265] Embodiment 76. The apparatus of any of embodiments 72 to 75, wherein the apparatus comprises a display device.

[0258]

[0266] Aspect 77. An apparatus for determining a location of one or more devices, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors configured to determine location data for the connected device based on at least one of a first identification output of the connected device, a second identification output of the apparatus, action tracking data, and first joint localization information; and map the connected device in a coordinate system of the apparatus based on the determined location data.

[0259]

[0267] Aspect 78. The apparatus of aspect 77, wherein the identification information output comprises at least one of an audio pattern, a display pattern, and a light pattern.

[0260]

[0268] Embodiment 79. An apparatus as described in any of embodiments 77 to 78, wherein the one or more processors are configured to determine location data for the connected device by detecting a first identification information output from the connected device and determining the location data based on the first identification information.

[0261]

[0269] Aspect 80. The apparatus of any of aspects 77 to 79, wherein the one or more processors are configured to determine location data for the connected device by sending the second identification information output to the connected device, receiving location information from the connected device, and determining the location data based on the location information, where the location information indicates a location of the connected device relative to the apparatus.

[0262]

[0270] Aspect 81. A device described in any of aspects 77 to 80, wherein one or more processors are configured to determine tracking data for the device and to determine location data based on the tracking data and the action tracking data.

[0263]

[0271] Aspect 82. An apparatus as described in any of aspects 77 to 81, wherein the one or more processors are configured to determine location data for the connected device by receiving first joint localization information from the connected device, determining second joint localization information, where the first joint localization information indicates a first location of a landmark relative to the connected device, and determining the location data based on the first joint localization information and the second joint localization information, where the second joint localization information indicates a second location of the landmark relative to the apparatus.

[0264]

[0272] Aspect 83. The apparatus of any of aspects 77 to 82, wherein the first joint localization information includes a feature vector describing features of the landmark, and the one or more processors are configured to use the feature vector to verify that the first joint localization information corresponds to the landmark.

[0265]

[0273] Embodiment 84. An apparatus according to any of embodiments 77 to 83, wherein the apparatus is an XR device or comprises an XR device.

[0266]

[0274] Aspect 85. An apparatus as described in any of aspects 77 to 84, wherein the apparatus is or comprises a mobile device.

[0267]

[0275] Aspect 86. A method for determining a location of one or more devices, the method comprising: determining location data for the connected device based on at least one of a first identification output of the connected device, a second identification output of the electronic device, action tracking data, and first joint location information; and mapping the connected device in a coordinate system of the electronic device based on the determined location data.

[0268]

[0276] Aspect 87. The method of aspect 86, wherein the identification information output comprises at least one of an audio pattern, a display pattern, and a light pattern.

[0269]

[0277] Aspect 88. The method of any of aspects 86 to 87, further comprising determining location data for the connected device by detecting a first identification information output from the connected device and determining the location data based on the first location information.

[0270]

[0278] Aspect 89. The method of any of aspects 86 to 88, further comprising determining location data for the connected device by sending a second identification information output to the connected device, receiving location information from the connected device, and determining the location data based on the location information, where the location information indicates a location of the connected device relative to the electronic device.

[0271]

[0279] Example 90. The method of any of examples 86 to 89, further comprising determining tracking data about the electronic device and determining location data based on the tracking data and the action tracking data.

[0272]

[0280] Aspect 91. The method of any of aspects 86 to 90, further comprising determining location data for the connected device by receiving first joint location information from the connected device, determining second joint location information, where the first joint location information indicates a first location of a landmark relative to the connected device, and determining the location data based on the first joint location information and the second joint location information, where the second joint location information indicates a second location of the landmark relative to the electronic device.

[0273]

[0281] Aspect 92. The method of any of aspects 86 to 91, wherein the first joint localization information includes a feature vector describing characteristics of the landmark, and the method further comprises using the feature vector to verify that the first joint localization information corresponds to the landmark.

[0274]

[0282] Embodiment 93. A method according to any of embodiments 86 to 92, wherein the electronic device is or comprises an XR device.

[0275]

[0283] Embodiment 94. The method of any of embodiments 86 to 93, wherein the electronic device is or comprises a mobile device.

[0276]

[0284] Embodiment 95. An apparatus comprising means for carrying out the method according to any of embodiments 86 to 94.

[0277]

[0285] Embodiment 96. The apparatus of embodiment 95, wherein the apparatus is an electronic device or comprises an electronic device.

[0278]

[0286] Aspect 97. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 86 to 94.

[0279]

[0287] Embodiment 98. An apparatus according to any of embodiments 1 to 36, wherein the connected device is directly or indirectly coupled to the apparatus.

[0280]

[0288] Embodiment 99. The method of any of embodiments 37 to 70, wherein the connected device is directly or indirectly coupled to an electronic device.

Claims

1. An apparatus for identifying the position of one or more devices, the apparatus comprising: a memory; one or more processors coupled to the memory; wherein the one or more processors are configured to: send a signal to a connected device coupled to the apparatus to trigger an identification information output from the connected device; receive the identification information output from the connected device in response to the signal, the identification information output comprising at least one of an audio pattern, a display pattern, and an optical pattern; identify the position of the connected device in the coordinate system of the apparatus and map the connected device for use in controlling the connected device from the apparatus based on the identification information output from the connected device; An apparatus configured to perform the above.

2. The apparatus according to claim 1, wherein the one or more processors are configured to: determine the location of the connected device based on the identification information output; map the connected device in the coordinate system of the apparatus based on the location of the connected device. An apparatus configured to perform the above.

3. The one or more processors are configured to: send a signal to the connected device to trigger an additional identification information output from the connected device; send one or more additional signals to the connected device at one or more different times based on a failure to receive the additional identification information output, the one or more additional signals being configured to trigger the additional identification information output from the connected device; receive the additional identification information output from the connected device; determine the location of the connected device based on the additional identification information output. An apparatus configured to perform the above. Here, in the coordinate system of the device, to map the connected device, the one or more processors are configured to update a device map to associate the connected device with the location of the connected device, and optionally, to send the one or more additional signals at one or more different times, the one or more processors are determining the failure to receive the additional identification information output between at least one of: the device being within a threshold proximity to the mapped location of the connected device, and the mapped location of the connected device being within the field of view of the image sensor of the device; sending the one or more additional signals to the connected device based on the failure to receive the additional identification information output, and / or periodically sending the one or more additional signals until at least one of: a predetermined amount of time elapses without receiving the additional identification information output from the connected device, and several additional signals are sent without receiving the additional identification information output from the connected device. The device according to claim 1, configured to perform the above. **Claim 4** To receive the identification information output from the connected device, the one or more processors are detecting one or more patterns in the identification information output based on data from one or more sensors of the device, and optionally, the one or more sensors comprise at least one of an audio sensor, an image sensor, and a light sensor; recognizing the one or more patterns as the identification information output from the connected device, and optionally, the one or more processors are configured to recognize the one or more patterns as the identification information output from the connected device using at least one of a neural network and feature matching. The device according to claim 1, configured to perform the above. **Claim 5** The one or more processors are Receiving a request to trigger an action by the connected device; Identifying the location of the connected device based on the mapping of the connected device; Sending a command to the connected device configured to trigger the action by the connected device The apparatus according to claim 1, configured to perform.

6. The request for triggering the action by the connected device is A request to send a video feed captured by the connected device to the apparatus, wherein the one or more processors Sending the command to the connected device, wherein the command is configured to trigger the connected device to send the video feed to the apparatus. Receiving the video feed from the connected device; Displaying the video feed in the apparatus Configured to perform A request to trigger the connected device to display content from the apparatus, wherein, to send the command to the connected device, the one or more processors Sending data including the command and the content from the apparatus to the connected device, wherein the command is configured to trigger the connected device to display the content from the apparatus. Configured to perform, and / or A request to adjust the power mode of the connected device, wherein the command is configured to adjust the power mode of the connected device. The apparatus according to claim 5, comprising.

7. The connected device comprises a light bulb, wherein the request for triggering the action by the connected device comprises a request to adjust at least one of the brightness and color of the light emitted by the light bulb, and wherein the command is configured to trigger the light bulb to adjust at least one of the brightness and color of the light emitted by the light bulb. The apparatus according to claim 5.

8. The one or more processors are Detecting gestures by a user associated with the device, wherein the gesture indicates the location of the connected device Triggering one or more identification information outputs from one or more connected devices in a physical environment based on the gesture Receiving a specific identification information output from the one or more identification information outputs, wherein the specific identification information output is transmitted from the location Determining that the connected device is located at the location based on the specific identification information output Configured to perform, and optionally, for mapping the connected device in the coordinate system of the device, the one or more processors Updating a device map to associate the connected device with the location of the connected device The device according to claim 1, configured to perform

9. The one or more processors Triggering one or more connected devices to enter a mapping mode Sending device identification information output to the one or more connected devices Receiving location identification information from the one or more connected devices, wherein the location identification information indicates the location of the one or more connected devices relative to the device Mapping the one or more connected devices in the coordinate system of the device based on the location identification information from the one or more connected devices The device according to claim 1, configured to perform

10. The one or more processors Triggering one or more connected devices to enter a mapping mode Receiving location identification information from the one or more connected devices, wherein the location identification information indicates a first location of a landmark relative to the one or more connected devices Determining a second location of the one or more connected devices based on the location identification information and a third location of the landmark relative to the device Mapping the one or more connected devices in the coordinate system of the device based on the second location of the one or more connected devices The apparatus according to claim 1, configured to perform

11. wherein the light pattern comprises at least one of a pattern of light dimming and brightening levels emitted by the apparatus, a pattern of color temperature change of light emitted by the apparatus, and a sequence of flashing lights emitted by the apparatus, wherein the display pattern comprises at least one of a code displayed on the display device of the apparatus, an image displayed on the display device of the apparatus, a visual pattern displayed on the display device of the apparatus, and a digital object displayed on the display device of the apparatus, and / or wherein the audio pattern comprises at least one of an audio code output by the apparatus and a sequence of sounds output by the apparatus, the apparatus according to claim 1.

12. To receive the identification information output, the one or more processors receive one or more identification information outputs from a plurality of identification information outputs broadcast by the connected device, the one or more identification information outputs comprising the identification information output, The apparatus according to claim 1, configured to perform

13. The apparatus is an extended reality device, a mobile device, and / or at least one of a display configured to display data and a camera configured to capture image data, the apparatus according to claim 1.

14. The connected device is at least one of a light bulb, a speaker, a television, a microphone, one or more sensors, a camera, a thermostat, and a wearable device, the apparatus according to claim 1.

15. A method for identifying the position of one or more devices, the method comprising sending a signal to a connected device coupled to an electronic device, the signal being configured to trigger an identification information output from the connected device, receiving the identification information output from the connected device in response to the signal, the identification information output comprising at least one of an audio pattern, a display pattern, and a light pattern Based on the output of the identification information from the connected device, in order to be used when controlling the connected device from the electronic device, in the coordinate system of the electronic device, identify the position of the connected device and map the connected device A method comprising.

Images