Controlling device and process settings based on radio frequency detection

Abstract

Systems and techniques for extended reality optimization using radio frequency (RF) sensing are disclosed. An example method can include acquiring RF sensing data, determining a reflection path of one or more reflected RF signals based on the RF sensing data, comparing the one or more reflection paths, where each reflected RF signal includes a reflection of a transmitted signal from one or more objects in physical space, to a field of view (FOV) of an image sensor of a device, and triggering an action by the device and / or the image sensor based on the comparison.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE

[0001] The present disclosure relates generally to radio frequency sensing. For example, aspects of the present disclosure relate to controlling device and / or process settings based on radio frequency sensing. [Background technology]

[0002]

[0002] Wireless electronic devices can provide a variety of wireless services, such as, for example, geolocation, mapping, and route discovery, among others. To implement various wireless functions, wireless electronic devices can include hardware and software components configured to transmit and receive radio frequency (RF) signals. For example, wireless electronic devices can be configured to communicate via Wi-Fi, 5G / New Radio (NR), Bluetooth, and / or Ultra Wideband (UWB), among others.

[0003]

[0003] In some cases, wireless electronic devices can also implement digital cameras to capture videos and / or images. For example, wireless devices such as phones, connected vehicles, computers, gaming systems, wearable devices, smart home assistants, etc. are often equipped with cameras. The cameras enable the electronic devices to capture videos and / or images. The videos and / or images can be captured for recreational applications, professional photography, surveillance, extended reality, and automation, among other applications. Moreover, cameras are increasingly equipped with specific functions to modify and / or manipulate the videos and / or images for various effects and / or applications. For example, many cameras are equipped with video / image processing capabilities to detect objects on the captured images, generate different image and / or video effects, etc. Summary of the Invention

[0004]

[0004] The following provides a simplified summary related to one or more aspects disclosed herein. Therefore, the following summary should not be considered an extensive overview related to all contemplated aspects, nor should the following summary be considered to identify key or critical elements related to all contemplated aspects or to define the scope related to a particular aspect. Thus, the following summary has the sole purpose of presenting, in a simplified form, some concepts related to one or more aspects related to the mechanisms disclosed herein, prior to the detailed description presented below.

[0005]

[0005] Systems, methods, apparatus, and computer-readable media for controlling devices and / or processing settings based on radio frequency (RF) sensing are disclosed. According to at least one example, a method for controlling devices and / or processing settings based on RF sensing is provided. The method can include acquiring radio frequency (RF) sensing data, determining one or more reflected paths of one or more reflected RF signals based on the RF sensing data, comparing the one or more reflected paths, where each reflected RF signal comprises a reflection of a transmitted RF signal from one or more objects in a physical space, to a field-of-view (FOV) of an image capturing device associated with a mobile device, and triggering an action by at least one of the image capturing device and the mobile device based on the comparison.

[0006] According to at least one example, a non-transitory computer-readable medium for controlling device and / or processing settings based on RF detection is provided. The non-transitory computer-readable medium can include instructions that, when executed by one or more processors, cause the one or more processors to obtain radio frequency (RF) detection data, determine one or more reflection paths of one or more reflected RF signals based on the RF detection data, compare the one or more reflection paths, where each reflected RF signal comprises a reflection of the transmitted RF signal from one or more objects in physical space, to a field of view (FOV) of an image capture device associated with the apparatus, and trigger an action by at least one of the image capture device and the apparatus based on the comparison.

[0007] According to at least one example, an apparatus is provided for controlling device and / or processing settings based on RF detection. The apparatus may include a memory and one or more processors coupled to the memory, the one or more processors configured to obtain radio frequency (RF) detection data, determine one or more reflection paths of one or more reflected RF signals based on the RF detection data, compare the one or more reflection paths, where each reflected RF signal comprises a reflection of a transmitted RF signal from one or more objects in a physical space, to a field of view (FOV) of an image capture device associated with the apparatus, and trigger an action by at least one of the image capture device and the apparatus based on the comparison.

[0008] According to at least one example, another apparatus is provided for controlling device and / or processing settings based on RF detection. The apparatus may include means for acquiring radio frequency (RF) detection data, determining one or more reflection paths of one or more reflected RF signals based on the RF detection data, comparing the one or more reflection paths, where each reflected RF signal comprises a reflection of a transmitted RF signal from one or more objects in physical space, to a field of view (FOV) of an image capture device associated with the apparatus, and triggering an action by at least one of the image capture device and the apparatus based on the comparison.

[0009]

[0009] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may determine, based on the comparison, that the one or more objects are outside the FOV of the image capture device, and based on determining that the one or more objects are outside the FOV of the image capture device, set a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device.

[0010]

[0010] In some aspects, the methods, non-transitory computer-readable media, and devices described above are capable of determining that one or more objects are moving toward a portion of the scene that corresponds to the FOV of the image capture device, and adjusting a power setting of the image capture device to a different power state based on the determination that the one or more objects are moving toward a portion of the scene that corresponds to the FOV of the image capture device.

[0011]

[0011] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may determine that a view of the image capture device of the one or more objects is obstructed by at least one object, and based on determining that the image capture device's view of the one or more objects is obstructed by the at least one object, set a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device, wherein triggering an action is further based on determining that the image capture device's view of the one or more objects is obstructed.

[0012]

[0012] In some examples, determining that the image capture device's view of the one or more objects is obstructed by at least one object further comprises determining, based on the comparison, that the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device, and determining, based on the location of the one or more objects, that the image capture device's view of the one or more objects is obstructed by the at least one object.

[0013] In some examples, the image capture device comprises multiple image sensors. In some cases, the triggered action comprises controlling a power setting of the image capture device, and controlling the power setting of the image capture device further comprises controlling individual power settings of the multiple image sensors.

[0014]

[0014] In some examples, controlling individual power settings of the multiple image sensors further includes dedicating at least one of the multiple processors of the mobile device to a particular one of the multiple image sensors for image processing based on determining that a hand is within the FOV of the particular one of the multiple image sensors.

[0015]

[0015] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may control use of multiple image capture devices of a mobile device by multiple processors of the mobile device based on determining that one or more objects are outside the FOV of the image capture device, the multiple image capture devices including the image capture device.

[0016] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can determine that one or more objects are outside the FOV of the image capture device and moving toward a portion of the scene corresponding to the FOV of the image capture device, and in response to determining that one or more objects are outside the FOV of the image capture device and moving toward a portion of the scene corresponding to the FOV of the image capture device, switch an active camera setting from the image capture device to a different image capture device. In some cases, the switched active camera setting can trigger the mobile device to use the different image capture device to capture one or more images.

[0017] In some examples, at least one of the one or more objects comprises a hand associated with a user of the mobile device.

[0018]

[0018] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may determine a path of an RF signal comprising a direct path of a transmitted RF signal, and determine a location of one or more objects relative to a mobile device based on the path of the RF signal.

[0019] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can determine a location of one or more objects relative to the mobile device based on one or more reflected paths. In some cases, determining the location of the one or more objects further comprises determining at least one of a respective distance, a respective azimuth angle, and a respective elevation angle associated with the one or more reflected paths.

[0020] In some cases, the triggered action comprises controlling a power setting of the image capture device. In some examples, controlling the power setting of the image capture device is further based on a light level falling below a threshold. In some examples, controlling the power setting of the image capture device is further based on a privacy setting. In some examples, the privacy setting is based on user input, application data, and / or global navigation satellite system (GNSS) data.

[0021]

[0021] In some aspects, the methods, non-transitory computer-readable media, and devices described above can determine at least one of the size and shape of one or more objects based on the RF detection data and one or more reflected paths.

[0022]

[0022] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can determine a shape of an object from one or more objects based on the RF sensing data and one or more reflected paths, determine that the object comprises a hand associated with a user of the mobile device based on the shape of the object, and generate a cropped image of the hand using an image captured by the image capture device.

[0023] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can determine a location of one or more objects relative to the mobile device based on one or more reflected paths. In some cases, the cropped image is generated based on the location of the one or more objects, and the location of the one or more objects comprises a hand location.

[0024]

[0024] In some aspects, the methods, non-transitory computer-readable media, and devices described above can select at least one of the one or more reflection paths from the one or more reflection paths based on the respective distances of the associated objects being within a distance threshold.

[0025] In some aspects, the method, non-transitory computer-readable medium, and apparatus described above can send the cropped image to a destination device. In some examples, the destination device comprises at least one of a server and a mobile device. In some cases, the mobile device comprises an extended reality device.

[0026] In some examples, at least one of the one or more objects comprises a hand of a user of a mobile device. In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can determine at least one of a map of the physical space and a hand gesture associated with the user's hand based on the RF sensing data and the one or more reflected paths.

[0027] In some cases, the triggered action comprises extracting a portion of the image including the one or more objects. In some cases, the triggered action comprises determining whether to capture one or more images of the one or more objects. In some examples, the triggered action is further based on a determination that the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device.

[0028] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may use a machine learning algorithm to detect at least one of a hand associated with a user of a mobile device and an obstruction of a view of an image capture device relative to the hand. In some examples, the hand comprises at least one of one or more objects.

[0029] In some aspects, the device is or is part of a mobile device (e.g., a mobile phone or “smartphone” or other mobile device), a wearable device (e.g., a head-mounted display), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a tablet, a personal computer, a laptop computer, a server computer, a wireless access point, or any other device having an RF interface. In some aspects, the device described above may include one or more cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device described above may include one or more sensors that may be used to determine the location of the device, the orientation of the device, and / or for any other purpose.

[0030]

[0030] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0031]

[0031] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0032] [Figure 1]

[0032] FIG. 1 illustrates an example of a wireless communications network, in accordance with some examples of the present disclosure. [Diagram 2]

[0033] 1 is a block diagram illustrating an example of a computing system of a user device, in accordance with some examples of the present disclosure. [Diagram 3]

[0034] FIG. 1 illustrates an example of a wireless device that utilizes radio frequency sensing techniques to detect objects and / or object characteristics in an environment, in accordance with some examples of the present disclosure. [Figure 4]

[0035] FIG. 1 illustrates an example of an environment including a wireless device for detecting objects and / or object characteristics, in accordance with some examples of the present disclosure. [Figure 5A]

[0036] FIG. 13 is an example of a graphical representation showing size and location of objects and walls based on radio frequency sensing, according to some examples of the present disclosure. [Figure 5B]

[0037] FIG. 11 is another example of a graphical representation showing the size and location of an object determined by radio frequency sensing, in accordance with some examples of the present disclosure. [Figure 6]

[0038] FIG. 1 illustrates an example use case for using radio frequency sensing to reduce uplink traffic from an extended reality device to a destination device, in accordance with some examples of the present disclosure. [Figure 7A]

[0039] FIG. 1 illustrates an example extended reality rendering scenario, in accordance with some examples of the present disclosure. [Figure 7B] FIG. 1 illustrates an example extended reality rendering scenario, in accordance with some examples of the present disclosure. [Figure 8]

[0040] 1 is a flow diagram illustrating an example process for extended reality optimization using radio frequency sensing, in accordance with some examples of the present disclosure. [Figure 9]

[0041] 1 is a block diagram illustrating an example of a computing system in accordance with some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033]

[0042] Some aspects and embodiments of the present disclosure are provided below for illustrative purposes. Alternative aspects may be devised without departing from the scope of the present disclosure. Moreover, well-known elements of the present disclosure are not described in detail or are omitted so as not to obscure the relevant details of the present disclosure. As will be apparent to those skilled in the art, some of the aspects and embodiments described herein may be applied independently, and some of them may be applied in combination. In the following description, for illustrative purposes, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not limiting.

[0034]

[0043] The following description provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the present application, as set forth in the appended claims.

[0035]

[0044] As mentioned above, wireless electronic devices can provide a variety of wireless and other services, such as, but not limited to, geolocation, mapping, extended reality, image processing, and route finding, among others. Non-limiting examples of wireless electronic devices can include mobile phones, wearable devices, smart home assistants, televisions, connected vehicles, gaming systems, Internet of Things (IoT) devices, cameras, tablet computers, laptop computers, and the like. To implement wireless functionality, wireless electronic devices can include hardware and software components configured to transmit and receive radio frequency (RF) signals. For example, wireless electronic devices can be configured to communicate via Wi-Fi, 5G / New Radio (NR), Bluetooth, and / or Ultra Wideband (UWB), among others. In some cases, wireless electronic devices can also implement digital cameras to capture videos and / or images.

[0036]

[0045] For example, wireless devices (e.g., mobile phones, wearable devices, connected vehicles, laptop computers, tablet computers, IoT devices, gaming systems, smart home assistants, cameras, etc.) are often equipped with cameras. Cameras enable electronic devices to capture video and / or images. Video and / or images may be captured for recreational applications, professional photography, surveillance, extended reality, and automation, among other applications. Moreover, camera devices are increasingly equipped with functionality to modify and / or manipulate the video and / or images for various effects and / or applications. For example, camera devices may be equipped with video / image processing capabilities to detect objects on the captured image, generate image and / or video effects, render the video / image, etc.

[0037]

[0046] In some examples, the camera may be implemented in an electronic device for extended reality (XR). XR technologies may include augmented reality (AR), virtual reality (VR), mixed reality (MR), etc. XR technologies may combine real environments from the physical world (e.g., the real world) with virtual environments or content to provide an XR experience to a user. An XR experience allows a user to interact with a real or physical environment that is enhanced or augmented with virtual content, and vice versa. XR technologies may be implemented to provide functionality and / or enhance user experiences in a wide range of contexts, such as, for example, healthcare, retail, education, social media, entertainment, etc.

[0038]

[0047] To provide a realistic XR experience, XR technology can integrate virtual content with the physical world. In some examples, this can involve generating a map of the real-world environment and determining or calculating a specific pose of the user's XR device relative to the map of the real-world environment to anchor the virtual content in a convincing manner. The pose information can be used to match the virtual content with the user's perceived movement and spatiotemporal state of the real-world environment.

[0039]

[0048] XR devices and other electronic devices, such as, for example, smartphones, tablets, laptops, among others, may include wireless capabilities and often perform functions such as geolocation, mapping, and / or route discovery, among others. In some cases, the XR device may send tracking frames and sensor measurements (e.g., inertial measurement unit (IMU) measurements, etc.) captured at the XR device to a destination device associated with the XR device, such as a mobile device (e.g., smartphone), a server (e.g., edge server, cloud server, etc.), and / or other device. The XR device may be a wearable device configured to be worn by a user. In some aspects, the XR device may be configured to be worn on the user's head, such as a head mounted display (HMD), smart glasses, XR or AR glasses, etc. The tracking frames may include images captured by a camera in the XR device associated with the user's field of view, for example, by pointing the camera in the XR device in the general direction of the user's vision, i.e., by positioning the camera in the XR device to face in that direction. The tracking frames may be used by a destination device (e.g., a mobile device or a server) to detect hand gestures, perform hand and controller tracking, and the like.

[0040]

[0049] However, the process of capturing and transmitting tracking frames to a destination device may consume a significant amount of network and device resources. For example, the process of capturing and transmitting tracking frames may increase power consumption in the XR device. Moreover, tracking frames from the XR device may be large and may significantly increase the uplink (UL) traffic requirements / usage for (UL) transmission to the destination device (e.g., a mobile device or a server).

[0041]

[0050] In some cases, using a tracking frame to detect hand gestures and track hands and other objects may have some challenges and / or limitations. For example, the tracking frame may not enable hand tracking when the hand is outside the field of view (FOV) of the camera in the XR device, such as when the hand is behind the user's body or in the user's pocket. However, in this scenario, even a rough estimate of hand location will help predict future hand positions and implement smoother hand tracking.

[0042]

[0051] Reducing the size of the tracking frame may reduce UL airtime, XR device power consumption, contention with downlink (DL) traffic, and / or other improvements. In some examples, the size of the tracking frame may be reduced using computer vision to process the tracking frame and estimate the (coarse) location of the hand in the tracking frame. In this approach, the XR device can reduce the size of the tracking frame sent to the destination device by sending only partial frames that contain the hand(s). However, implementing computer vision in the XR device to estimate the (coarse) location of the hand(s) in the tracking frame may significantly increase power consumption in the XR device.

[0043]

[0052] Moreover, the camera device components implemented by the electronic device to capture images (also referred to as frames) may increase power consumption in the electronic device. In many cases, some electronic devices, such as mobile devices (e.g., XR devices, Internet Protocol (IP) cameras, smartphones, wearable devices, smart home assistants, tablet computers, laptop computers, IoT devices, etc.), may have more limited battery / power capabilities than other devices and may be more significantly affected by the power consumption of the camera device components. Power consumption in an electronic device may often have a greater impact on the battery life of the electronic device if the electronic device performs additional and / or more computationally intensive operations. For example, in some cases, the XR device may not send image data (e.g., tracking frames, etc.) or sensor measurements to a destination device to offload some operations previously described (e.g., hand gesture detection, object detection, hand tracking, controller tracking, etc.), but rather use the image data and sensor measurements to perform such operations in the XR device. In such cases, the additional operations performed at the XR device (e.g., for implementations in which such operations are offloaded to a destination device) may increase power consumption at the XR device and thus the impact of battery / power constraints / limitations at the XR device.

[0044]

[0053] As described further herein, in some examples, an electronic device can reduce its power consumption by reducing and / or limiting unnecessary power and / or resource usage of camera device components. Such reduction in power consumption can increase the battery life of the electronic device and can be advantageous given the limited battery / power capabilities of the electronic device, such as in scenarios where the electronic device is an XR device and does not offload operations (or offloads fewer operations) to a destination device as previously described. For example, to reduce power consumption in an XR device, the XR device can control settings of camera device components in the XR device to reduce and / or increase their power consumption levels depending on whether an object of interest (or objects) in the scene is visible by the camera device.

[0045]

[0054] To illustrate, if an object of interest is not visible to the camera device of the XR device (e.g., outside the field of view of the camera device and / or occluded by another object(s), etc.), the XR device may turn off the camera device or reduce the power mode of the camera device to avoid unnecessary power consumption by the camera device while the camera device is unable to capture an image of the object. If the object of interest is within the field of view of the camera device (or approaching the field of view of the camera device within a threshold estimated time frame, proximity, and / or trajectory) and is not occluded by another object(s), the XR device may turn on the camera device or increase the power mode of the camera device to enable (or better enable) the camera device to capture an image of the object.

[0046]

[0055] Described herein are systems and techniques for reducing the amount of network and / or device resources used by devices such as XR devices (e.g., head mounted displays, smart glasses, etc.) in several applications, such as XR, automation, image / video processing, etc. In some examples, the systems and techniques described herein can reduce the amount / size of UL traffic from an XR device to a destination device, such as a mobile device or server. For example, the systems and techniques described herein can enable hand gesture detection and / or hand and object tracking without full tracking frames from an XR device.

[0047]

[0056] In some examples, the XR device can use radio frequency (RF) sensing to detect the location of at least one hand of the user. The XR device can use the detected location of the user's hand(s) to create a partial frame, i.e., only a portion of the captured (full) tracking frame, that includes the user's hand(s) and send the partial frame to the destination device instead of sending the full tracking frame. The destination device can use the partial frame to detect hand gestures, track the user's hand(s), and / or detect any other objects in the scene, such as the XR controller. The partial frame can be smaller than the full tracking frame. Thus, by sending the partial frame instead of the full tracking frame, the XR device can reduce the size of the UL traffic to the destination device and overall network usage.

[0048]

[0057] RF sensing can use RF data to generate a map of a space, such as a space in a scene or an indoor space. In some cases, using a monostatic configuration, a wireless device (e.g., an XR device, a smartphone, etc.) can acquire RF sensing data using a wireless interface capable of performing transmit and receive functions. In some examples, a wireless device can implement a Wi-Fi radar for acquiring RF sensing data. A Wi-Fi radar can implement an RF interface capable of performing transmit and receive functions. A Wi-Fi radar (and / or a wireless device implementing a Wi-Fi radar) can use the signal strength of a radio transmission to determine the location / position of one or more objects. For example, a wireless device can utilize a wireless interface (e.g., of a Wi-Fi radar) to transmit an RF signal and capture a signal that reflects off an object in the surrounding environment. The wireless interface can also receive a leakage signal that is coupled from a transmitter antenna directly to a receiver antenna without reflecting off an object. In some examples, a wireless device can gather RF sensing data in the form of channel state information (CSI) data related to a direct path (leakage signal) of a transmitted signal and data related to a reflected path of a received signal corresponding to the transmitted signal. In some cases, a bistatic configuration may be used in which the transmitting and receiving functions are performed by different devices. For example, a first device may transmit a wireless signal that reflects off one or more objects in a scene, and a wireless interface of a second device may receive the reflected signal and / or receive the signal directly from the first device. In some cases, the signal may be an omnidirectional signal transmitted using an omnidirectional antenna, and the signal may be transmitted in a 360 degree radiation pattern.

[0049]

[0058] The CSI data can describe how a wireless signal propagates from a transmitter to a receiver. The CSI data can represent the combined effects of scattering, fading, and power attenuation over distance, and can indicate the channel properties of a communication link. The collected CSI data can reflect the varying multipath reflections induced by a moving object due to its frequency diversity. In some examples, the CSI data can include I / Q numbers for each tone in the frequency domain for a bandwidth. Changes in some CSI properties can be used to detect motion, estimate changes in location, determine changes in motion patterns, and the like.

[0050]

[0059] In some examples, the CSI data may be used to determine or calculate the distance and angle of arrival of the reflected signal. In some cases, the CSI data may be used to determine the distance, azimuth, and / or elevation of one or more paths of one or more reflected signals. The distance, azimuth, and / or elevation of one or more paths of one or more reflected signals may be used to identify the size, shape, and / or location of one or more objects in the surrounding environment. The size, shape, and / or location of the one or more objects may be used to control one or more resources (e.g., power resources, sensor resources, processor resources, etc.) of the device, generate an indoor map, track one or more objects, etc. In one example, the distance of the reflected signal may be determined by measuring the time difference from receipt of the leakage signal to receipt of the reflected signal. In another example, the reflection angle may be determined by using an antenna array to receive the reflected signal and measuring the difference in the received phase at each element of the antenna array.

[0051]

[0060] In some cases, with RF sensing, the system can use signal processing to extract reflections and focus on reflection paths that are short in distance (e.g., within a threshold distance such as 1.5 m or 1 m, as some objects of interest such as hands are generally close to the XR device) and / or shorter in distance relative to the direct path, to reduce computational complexity and / or power consumption at the XR device. RF sensing can be used to estimate the dimensions of the object producing the reflection. An object classification algorithm (e.g., signal processing, machine learning, etc.) can be used to classify the detected object. For example, the object classification algorithm can be used to classify the detected object as a hand or as not a hand (or as having one or more classifications other than a hand). When an omnidirectional signal is transmitted, RF sensing can track 360 degrees and is not limited by the camera view. Thus, RF sensing can be used to track objects (e.g., hands, etc.) outside the camera view. In some examples, RF sensing can track objects even without a tracking frame and provide an estimate of the object's location. The XR device can send RF sensing information to a destination device (e.g., a smartphone or a server) to help predict future object positions and / or achieve smooth object tracking.

[0052]

[0061] In some examples, the XR device may receive an RF detection frame including a downlink physical layer protocol data unit (DL-PPDU) from a destination device (e.g., a smartphone, a server, etc.) or a Wi-Fi radar signal. The XR device may estimate CSI information from the RF detection frame. The XR device may process the CSI information and extract a direct path and a reflected path of an RF signal associated with the RF detection frame. A direct path may be extracted even without line of sight (LOS) and may be detected even through some barriers, such as an object or a wall. In some cases, the direct path of the DL-PPDU may be from a destination device (e.g., a smartphone or a server) to the XR device. In some cases, the direct path of the Wi-Fi radar signal may be from a Tx antenna(s) at the XR device to an Rx antenna(s) at the XR device (monostatic configuration). In other cases, the direct path of the Wi-Fi radar signal may be from a Tx antenna(s) at the destination device or another device to an Rx antenna(s) at the XR device (bistatic configuration).

[0053]

[0062] In some examples, the XR device may select reflected paths that are shorter in distance (e.g., or within a threshold distance, such as 1.5 m or 1 m) relative to the direct path and estimate the range, azimuth, and elevation of each selected reflected path with respect to the RF sensing coordinate system. The XR device may use the range, azimuth, and elevation of each selected reflected path to detect an object(s) in physical space and measure dimensions of each detected object. In some examples, the range, azimuth, and elevation (and / or the angle thereof) of each selected reflected path may be determined or calculated based on an estimated time-of-flight and an estimated angle-of-arrival of a reflected signal associated with each reflected path.

[0054]

[0063] In some aspects, the XR device can use an object classification algorithm (e.g., signal processing, machine learning) to classify a detected object. For example, the XR device can use an object classification algorithm to classify the detected object(s) as a body part (e.g., a hand, a finger, a leg, a foot, etc.), an input device (e.g., a stylus, a controller, a ring, a glove, a mouse, a keyboard, a joystick, a knob, a body suit, a mat or treadmill, a ball, etc.), or any other object. In some cases, the XR device can detect one or more visual markers (e.g., visual features, patterns, codes, properties, etc.) on an object such as an input device and use the one or more visual markers to classify the object using an object classification algorithm. In some examples, the XR device can use an object classification algorithm to classify whether the detected object is a hand or not. The XR device can perform camera calibration to align the camera image coordinate system with the RF sensing coordinate system. The camera image coordinate system can be from a field of view (FOV) of the camera at the XR device based on the camera location and pose of the XR device. In some examples, the XR device can use sensor data (with or without other data), such as data from an inertial measurement unit (IMU), to calculate the pose of the XR device. In some cases, the XR device can use the pose of the XR device to calculate reference coordinates for the XR device's FOV and / or refine object positions (e.g., hand positions, input device positions, etc.) calculated using RF sensing.

[0055]

[0064] The RF sensing coordinate system may be based on the antenna location and pose of the XR device. In the case of a fixed spatial relationship between the RF sensing components (one or more), e.g., between the Rx antenna (one or more) in the XR device and the camera (both are generally mounted on the XR device), the pose of the XR device may be omitted when deriving the above-mentioned coordinate system. The XR device may determine the location of a detected object classified as a hand and use the location of the hand to generate a partial image capturing / including the detected hand. In some examples, the XR device may use the location of the hand to crop the image of the hand captured by the camera to generate a partial image (e.g., a cropped image of the hand). The XR device may send the partial image in a UL packet to a destination device. The destination device may use the partial image to detect hand gestures, track the hand, and / or track other objects, such as one or more input devices (e.g., a controller, a stylus, a ring, a glove, a mouse, a keyboard, a joystick, a knob, a body suit, a mat or treadmill, a ball, etc.).

[0056]

[0065] In some examples, the XR device can use the RF sensing data to determine whether the detected hand is occluded by one or more objects and / or whether the detected hand is outside the FOV of the camera of the XR device. For example, the XR device can determine whether the detected hand is under a table, occluding the hand from the FOV of the camera. As another example, the XR device can determine whether the hand is behind the user's back and outside the FOV of the camera. In some cases, when it is determined that the hand is occluded and / or outside the FOV of the camera, the XR device can control the power settings of the camera to reduce the power consumption of the XR device. For example, the XR device can turn off the camera or put the camera into a lower power mode while the hand is occluded or outside the FOV of the camera. If the XR device later determines that the hand is no longer occluded or outside the FOV of the camera, it can turn the camera back on or put the camera into a higher power mode. In some cases, the XR device can use RF sensing to detect movement of the hand(s) and prepare the camera to take one or more images to capture the hand gesture if the movement indicates that the hand(s) are approaching the camera's FOV. In some cases, when the hand is determined to be occluded or otherwise outside the camera's FOV, the XR device can use the RF sensing data to generate a sparse map of the physical space associated with the XR device.

[0057]

[0066] In some cases, the XR device may determine and store device position data and device orientation data. In some instances, the XR device position data and device orientation data may be used to adjust calculations for the distance and angle of reflection of reflected signals (determined using CSI data) if the device is moving. For example, the position and orientation data may be used to correlate one or more reflected signals with their corresponding transmitted signals. In some examples, device position or location data may be collected using techniques that measure round trip time (RTT), passive positioning, angle of arrival (AoA), received signal strength indicator (RSSI), using CSI data, using any other suitable technique, or any combination thereof. Device orientation data may be obtained from electronic sensors on the XR device, such as one or more gyroscopes, accelerometers, compasses, any other suitable sensors, or any combination thereof.

[0058]

[0067] Various aspects of the techniques described herein are described below with reference to the figures. Figure 1 is a block diagram of an example communication system 100. According to some aspects, the communication system 100 can include a wireless local area network (WLAN) 108, such as a Wi-Fi network. For example, the WLAN 108 can be a network that implements at least one of the IEEE 802.11 wireless communications protocol standard family (such as those defined by the IEEE 802.11-2016 specification or amendments thereof, including, but not limited to, 802.11ay, 802.11ax, 802.11az, 802.11ba, and 802.11be).

[0059]

[0068] The WLAN 108 may include a number of wireless communication devices, such as an access point (AP) 102 and user equipment (UE) 104a, 104b, 104c, and 104d (collectively "UE 104"). Although only one AP 102 is shown, the WLAN 108 may also include multiple APs 102. Generally, a UE may be any wireless communication device (e.g., a mobile phone, a router, a tablet computer, a laptop computer, a wearable device (e.g., an extended reality (XR) device such as a smart watch, glasses, a virtual reality (VR) headset, an augmented reality (AR) headset or glasses, or a mixed reality (MR) headset, etc.), a vehicle (e.g., an automobile, a motorcycle, a bicycle, etc.), an Internet of Things (IoT) device, etc.) used by a user to communicate over a wireless communication network. The UE may be mobile or (e.g., at some times) stationary and may communicate with a radio access network (RAN). The term "UE" as used herein may be referred to interchangeably as "access terminal" or "AT", "user device", "user terminal" or UT, "client device", "wireless device", "subscriber device", "subscriber terminal", "subscriber station", "mobile device", "mobile terminal", "mobile station", or variations thereof. Generally, a UE may communicate with a core network via a RAN, through which the UE may be connected with external networks, such as the Internet, and other UEs. A UE may also communicate with other UEs and / or other devices described herein.

[0060]

[0069] An associated set of APs 102 and UEs 104 may be referred to as a Basic Service Set (BSS) managed by each AP 102. The BSS may be identified to users by a Service Set Identifier (SSID) and to other devices by a Basic Service Set Identifier (BSSID), which may be the Media Access Control (MAC) address of the AP 102. The AP 102 periodically broadcasts a beacon frame ("beacon") containing the BSSID to allow any UE 104 within wireless range of the AP 102 to "associate" or reassociate with the AP 102 to establish a respective (hereinafter also referred to as a "Wi-Fi link") communication link 106 with the AP 102 or to maintain a communication link 106 with the AP 102. For example, the beacon may include an identification of a primary channel used by each AP 102, as well as a timing synchronization function to establish or maintain timing synchronization with the AP 102. The AP 102 may provide access to an external network to various UEs 104 in a WLAN via their respective communication links 106.

[0061]

[0070] To establish a communication link 106 with an AP 102, each of the UEs 104 is configured to perform passive or active scanning operations ("scans") on frequency channels in one or more frequency bands (e.g., the 2.4 GHz, 5 GHz, 6 GHz, or 60 GHz bands). To perform passive scanning, the UEs 104 listen for beacons transmitted by the respective APs 102 at periodic time intervals called target beacon transmission times (TBTTs) (measured in time units (TUs), where one TU may equal 1024 microseconds (μs)). To perform active scanning, the UEs 104 generate and transmit probe requests continuously on each channel to be scanned and listen for probe responses from the APs 102. Each UE 104 may be configured to identify or select an AP 102 to associate with based on scanning information obtained through passive or active scanning, and perform authentication and association operations to establish a communication link 106 with the selected AP 102. The AP 102 assigns an association identifier (AID) to the UE 104 at the height of the association operation, and the AP 102 uses the AID to track the UE 104 .

[0062]

[0071] Given the increasing ubiquity of wireless networks, the UE 104 may have the opportunity to select one of many BSSs within range of the UE or to select among multiple APs 102 that together form an extended service set (ESS) including multiple connected BSSs. An extended network station associated with a WLAN 108 may be connected to a wired or wireless distribution system that may allow multiple APs 102 to be connected in such an ESS. Thus, the UE 104 may be covered by more than one AP 102 and may associate with different APs 102 at different times for different transmissions. Furthermore, after association with an AP 102, the UE 104 may also be configured to periodically scan its surroundings to find a more suitable AP 102 to associate with. For example, a UE 104 moving relative to its associated AP 102 may perform a "roaming" scan to find another AP 102 with more desirable network characteristics, such as a greater received signal strength indicator (RSSI) or reduced traffic load.

[0063]

[0072] In some cases, the UE 104 may form a network without the AP 102 or other equipment other than the UE 104 itself. One example of such a network is an ad-hoc network (or wireless ad-hoc network). An ad-hoc network may alternatively be referred to as a mesh network or a peer-to-peer (P2P) network. In some cases, the ad-hoc network may be implemented within a larger wireless network, such as a WLAN 108. In such an implementation, the UEs 104 may be able to communicate with each other through the AP 102 using the communication link 106, but the UEs 104 may also communicate with each other directly via a direct wireless link 110. In addition, the two UEs 104 may communicate via one or more device-to-device (D2D) peer-to-peer (P2P) links, referred to as "sidelinks." In the example of FIG. 1, the UE 104b has a direct wireless link 110 (e.g., a D2D P2P link) with the UE 104a, which is connected to one or more base stations 160, allowing the UE 104b to indirectly obtain cellular connectivity. 1, the communications system 100 may include multiple base stations in communication with the UE 104. In one example, the direct wireless link 110 may be supported using any well-known D2D RAT, such as LTE Direct (LTE-D), Wi-Fi Direct® (WiFi-D), Bluetooth®, UWB, etc.

[0064]

[0073] The AP 102 and the UE 104 may function and communicate (via their respective communication links 106) in accordance with the IEEE 802.11 wireless communications protocol standards family (such as those defined by the IEEE 802.11-2016 specification or amendments thereof, including, but not limited to, 802.11ay, 802.11ax, 802.11az, 802.11ba, and 802.11be). These standards define WLAN radio and baseband protocols for the PHY layer and the medium access control (MAC) layer. The AP 102 and the UE 104 send and receive wireless communications (hereinafter also referred to as “Wi-Fi communications”) between each other in the form of PHY Protocol Data Units (PPDUs) (or Physical Layer Convergence Protocol (PLCP) PDUs). The APs 102 and UEs 104 in the WLAN 108 may transmit PPDUs over an unlicensed spectrum, which may be a portion of the spectrum that includes frequency bands traditionally used by Wi-Fi technology, such as the 2.4 GHz band, the 5 GHz band, the 60 GHz band, the 3.6 GHz band, and the 900 MHz band. Some implementations of the APs 102 and UEs 104 described herein may also communicate in other frequency bands, such as the 6 GHz band, which may support both licensed and unlicensed communications. The APs 102 and UEs 104 may also be configured to communicate on other frequency bands, such as shared licensed frequency bands, where multiple operators may have licenses to operate in the same or overlapping frequency band or bands.

[0065]

[0074] Each of the frequency bands may include multiple sub-bands or frequency channels. For example, PPDUs conforming to the IEEE 802.11n, 802.11ac, 802.11ax and 802.11be standard amendments may be transmitted on the 2.4, 5 GHz or 6 GHz bands, each of which is divided into multiple 20 MHz channels. Thus, these PPDUs are transmitted on physical channels with a minimum bandwidth of 20 MHz, but larger channels may be formed through channel bonding. For example, PPDUs may be transmitted on physical channels with bandwidths of 40 MHz, 80 MHz, 160 MHz or 200 MHz by bonding together multiple 20 MHz channels.

[0066]

[0075] In some examples, the communications system 100 may include one or more base stations 160. The one or more base stations 160 may include macrocell base stations (high-power cellular base stations) and / or small cell base stations (low-power cellular base stations). In an aspect, the macrocell base stations may include eNBs and / or ng-eNBs corresponding to 4G / LTE networks, or gNBs corresponding to 5G / NR networks, or a combination of both, and the small cell base stations may include femtocells, picocells, microcells, etc.

[0067]

[0076] One or more base stations 160 collectively form a RAN and may interface with a core network 170 (e.g., Evolved Packet Core (EPC) or 5G Core (5GC)) through backhaul links 122 and through the core network 170 to one or more servers 172 (which may be part of the core network 170 or external to the core network 170). In addition to other functions, the one or more base stations 160 may perform functions related to one or more of forwarding user data, radio channel encryption and decryption, integrity protection, header compression, mobility control functions (e.g., handover, dual connectivity), inter-cell interference coordination, connection setup and release, load balancing, distribution for non-access stratum (NAS) messages, NAS node selection, synchronization, RAN sharing, Multimedia Broadcast Multicast Services (MBMS), subscriber and equipment tracing, RAN Information Management (RIM), paging, positioning, and delivery of alert messages.

[0068]

[0077] One or more base stations 160 may wirelessly communicate with UEs, such as UE 104a, via communication links 120. The communication links 120 between the one or more base stations 160 and the UEs 104 may include uplink transmissions (also referred to as reverse links) from the UEs (e.g., UEs 104a, 104b, 104c, and / or 104d) to the base stations 160, and / or downlink transmissions (also referred to as forward links) from the base stations 160 to one or more of the UEs 104. The communication links 120 may use MIMO antenna techniques, including spatial multiplexing, beamforming, and / or transmit diversity. The communication links 120 may be over one or more carrier frequencies.

[0069]

[0078] Each of the UEs 104 in the communication system 100 may be configured to perform an RF sensing function for generating an indoor map. The RF sensing function may be implemented using any of the RF interfaces present in the UEs 104 that are capable of simultaneously transmitting and receiving RF signals. The UEs 104 may transfer data related to indoor mapping (e.g., RF sensing data, partial map data, location data, orientation data, etc.) by utilizing the communication system 100.

[0070]

[0079] In some examples, the UE 104 may communicate with one or more servers, such as the server 172, as part of one or more services and / or functions. For example, the UE 104 may communicate with the server 172 as part of an XR experience. The server 172 may assist with one or more functions, such as, for example, tracking, mapping, rendering, etc. Communication with the server 172 may occur via a core network 170 that may be accessed by the UE 104 by utilizing a communication link with a base station 160 or an AP 102. The AP 102 may access the core network, including the server 172, via the communication link 112.

[0071]

[0080] 2 is a diagram illustrating an example computing system 220 of a user device 210. In some examples, the user device 210 may be an example UE. For example, the user device 210 may include a mobile phone, a router, a tablet computer, a laptop computer, a wearable device (e.g., a smart watch, glasses, an XR device, etc.), an Internet of Things (IoT) device, and / or another device used by a user to communicate over a wireless communication network.

[0072]

[0081] The computing system 220 includes software and hardware components that may be electrically or communicatively coupled (or in other communication, as appropriate) via a bus 238. For example, the computing system 220 may include one or more processors 234. The one or more processors 234 may include one or more central processing units (CPUs), image signal processors (ISPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), application processors (APs), graphics processing units (GPUs), digital signal processors (DSPs), visual processing units (VPUs), neural network signal processors (NSPs), microcontrollers, special purpose hardware, any combination thereof, and / or other processing devices or systems. The bus 238 may be used by the one or more processors 234 to communicate between cores and / or with one or more memory devices 236.

[0073]

[0082] The computing system 220 may also include one or more memory devices 236, one or more digital signal processors (DSPs) 232, one or more subscriber identity modules (SIMs) 224, one or more modems 226, one or more wireless transceivers 228, one or more antennas 240, one or more input devices 222 (e.g., a camera, a mouse, a keyboard, a touch-sensitive screen, a touchpad, a keypad, a microphone, etc.), and / or one or more output devices 230 (e.g., a display, a speaker, a printer, etc.).

[0074]

[0083] The one or more wireless transceivers 228 can receive wireless signals (e.g., signals 242) via antenna 240 from one or more other devices, such as other user devices, network devices (e.g., base stations such as eNBs and / or gNBs, Wi-Fi access points (APs) such as routers, range extenders, etc.), cloud networks, etc. In some examples, the computing system 220 can include multiple antennas or antenna arrays that can facilitate simultaneous transmission and reception capabilities. In some cases, the antenna 240 can be an omnidirectional antenna such that RF signals can be received from all directions and transmitted in all directions. The wireless signals 242 can be transmitted over a wireless network. The wireless network can be any wireless network, such as a cellular or telecommunications network (e.g., 3G, 4G, 5G, etc.), a wireless local area network (e.g., a Wi-Fi network), a Bluetooth® network, and / or other network.

[0075]

[0084] In some examples, the one or more wireless transceivers 228 may include an RF front end that includes one or more components such as amplifiers, mixers (also called signal multipliers) for signal downconversion, frequency synthesizers (also called oscillators) that provide signals to the mixers, baseband filters, analog-to-digital converters (ADCs), one or more power amplifiers, among other components. The RF front end can generally handle the selection and conversion of the wireless signal 242 to baseband or an intermediate frequency, and can convert the RF signal to the digital domain.

[0076]

[0085] In some cases, the computing system 220 may include a coding-decoding device (or codec) configured to encode and / or decode data transmitted and / or received using the one or more wireless transceivers 228. In some cases, the computing system 220 may include an encryption-decryption device or component configured to encrypt and / or decrypt (e.g., according to the AES and / or DES standards) data transmitted and / or received by the one or more wireless transceivers 228.

[0077]

[0086] The one or more SIMs 224 may each securely store an International Mobile Subscriber Identity (IMSI) number and associated keys assigned to a user of the user device 210. The IMSI and keys may be used to identify and authenticate a subscriber when accessing a network provided by a network service provider or operator associated with the one or more SIMs 224. The one or more modems 226 may modulate one or more signals to encode information for transmission using the one or more wireless transceivers 228. The one or more modems 226 may also demodulate signals received by the one or more wireless transceivers 228 to decode the transmitted information. In some examples, the one or more modems 226 may include a Wi-Fi modem, a 4G (or LTE) modem, a 5G (or NR) modem, and / or other types of modems. The one or more modems 226 and the one or more wireless transceivers 228 may be used to communicate data for the one or more SIMs 224.

[0078]

[0087] The computing system 220 may also include (and / or may be in communication with) one or more non-transitory machine-readable storage media or storage devices (e.g., one or more memory devices 236), which may include, but are not limited to, local and / or network accessible storage, disk drives, drive arrays, optical storage devices, solid-state storage devices such as RAM and / or ROM that may be programmable, flash updatable, etc. Such storage devices may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, etc.

[0079]

[0088] In some examples, the functionality may be stored as one or more computer program products (e.g., instructions or code) in memory device(s) 236 and executed by one or more processors 234 and / or one or more DSPs 232. Computing system 220 may also include software elements (e.g., located in one or more memory devices 236) including, for example, an operating system, device drivers, executable libraries, and / or other code, such as one or more application programs that may include computer programs that implement functionality provided by various embodiments and / or that may be designed to implement methods and / or configure systems as described herein.

[0080]

[0089] As mentioned above, systems and techniques for XR optimization (e.g., for optimizing XR operations / features, devices / resources, settings, capabilities, etc.) using radio frequency (RF) sensing are described herein. FIG. 3 is a diagram illustrating an example of a wireless device 300 that utilizes RF sensing techniques to detect an object 302 to perform one or more XR optimizations as described herein. In some examples, the wireless device 300 may be an XR device (e.g., an HMD, smart glasses, etc.), a mobile phone, a wireless access point, or some other device that includes at least one RF interface.

[0081]

[0090] In some aspects, the wireless device 300 may include one or more components for transmitting an RF signal. The wireless device 300 may include a digital-to-analog converter (DAC) 304 capable of receiving a digital signal or waveform (e.g., from a microprocessor, not shown) and converting the digital signal to an analog waveform. The analog signal that is the output of the DAC 304 may be provided to an RF transmitter 306. The RF transmitter 306 may be a Wi-Fi transmitter, a 5G / NR transmitter, a Bluetooth transmitter, or any other transmitter capable of transmitting an RF signal.

[0082]

[0091] The RF transmitter 306 may be coupled to one or more transmit antennas, such as a TX antenna 312. In some examples, the TX antenna 312 may be an omnidirectional antenna capable of transmitting RF signals in all directions. For example, the TX antenna 312 may be an omnidirectional Wi-Fi antenna capable of radiating Wi-Fi signals (e.g., 2.4 GHz, 5 GHz, 6 GHz, etc.) in a 360-degree radiation pattern. In another example, the TX antenna 312 may be a directional antenna that transmits RF signals in a particular direction.

[0083]

[0092] In some examples, the wireless device 300 may also include one or more components for receiving RF signals. For example, the receiver lineup in the wireless device 300 may include one or more receive antennas, such as the RX antenna 314. In some examples, the RX antenna 314 may be an omni-directional antenna capable of receiving RF signals from multiple directions. In other examples, the RX antenna 314 may be a directional antenna configured to receive signals from a particular direction. In further examples, both the TX antenna 312 and the RX antenna 314 may include multiple antennas (e.g., elements) configured as an antenna array.

[0084]

[0093] The wireless device 300 may also include an RF receiver 310 coupled to an RX antenna 314. The RF receiver 310 may include one or more hardware components for receiving an RF waveform, such as a Wi-Fi signal, a Bluetooth signal, a 5G / NR signal, or any other RF signal. An output of the RF receiver 310 may be coupled to an analog-to-digital converter (ADC) 308. The ADC 308 may be configured to convert the received analog RF waveform to a digital waveform that may be provided to a processor, such as a digital signal processor (not shown).

[0085]

[0094] In one example, the wireless device 300 may implement an RF sensing technique by transmitting a TX waveform 316 from the TX antenna 312. Although the TX waveform 316 is shown as a single line, in some examples, the TX waveform 316 may be transmitted in all directions (e.g., 360 degrees) or multiple directions by the TX antenna 312 (e.g., via beamforming). In one example, the TX waveform 316 may be a Wi-Fi waveform transmitted by a Wi-Fi transmitter in the wireless device 300. In some examples, the TX waveform 316 may be implemented to have a sequence with perfect or near perfect autocorrelation properties. For example, the TX waveform 316 may include a single carrier Zadoff sequence or may include symbols that are similar to orthogonal frequency division multiplexing (OFDM) long training field (LTF) symbols.

[0086]

[0095] In some cases, the wireless device 300 may implement RF sensing techniques by performing transmit and receive functions concurrently. For example, the wireless device 300 may enable its RF receiver 310 to receive at or near the same time that the wireless device 300 enables the RF transmitter 306 to transmit the TX waveform 316. In some examples, the transmission of a sequence or pattern included in the TX waveform 316 may be repeated continuously such that the sequence is transmitted a certain number of times or for a certain duration. In some examples, repeating a pattern in the transmission of the TX waveform 316 may be used to avoid missing reception of a reflected signal when the RF receiver 310 is enabled after the RF transmitter 306. In some examples, the TX waveform 316 may include a sequence with a sequence length L that is transmitted two or more times, which may enable the RF receiver 310 to be enabled in a time less than or equal to L to receive a reflection corresponding to the entire sequence without missing any information.

[0087]

[0096] By implementing simultaneous transmission and reception capabilities, the wireless device 300 can receive a signal corresponding to the TX waveform 316. For example, the wireless device 300 can receive a signal reflected from an object within range of the TX waveform 316, such as the RX waveform 318 reflected from the object 302. The wireless device 300 can receive a leakage signal (e.g., the TX leakage signal 320) coupled directly from the TX antenna 312 to the RX antenna 314 without reflecting from an object. In some cases, the RX waveform 318 can include multiple sequences corresponding to multiple copies of a sequence included in the TX waveform 316. In some examples, the wireless device 300 can combine multiple sequences received by the RF receiver 310 to improve the signal-to-noise ratio (SNR).

[0088]

[0097] 3 as a monostatic configuration, the present disclosure is not limited to a monostatic configuration. In some examples, the TX waveform 316 may be transmitted by a corresponding transmit chain (DAC, RF TX, antenna(s)) provided at a spatially separated transmitting device, such as a destination device. For example, information regarding the relative position and / or orientation of the wireless device 300 and a separate transmitting device from a positioning process of the wireless device 300 may be determined at and / or communicated to the wireless device 300 to implement RF sensing techniques according to the present disclosure.

[0089]

[0098] In some examples, the wireless device 300 may implement an RF sensing technique by obtaining RF sensing data associated with each received signal corresponding to the TX waveform 316. In some examples, the RF sensing data may include channel state information (CSI) data related to a direct path (e.g., the leakage signal 320 or a line-of-sight path) of the TX waveform 316 and data related to a reflected path (e.g., the RX waveform 318) corresponding to the TX waveform 316.

[0090]

[0099] In some cases, the RF sensing data (e.g., CSI data) may include information that may be used to determine how an RF signal (e.g., TX waveform 316) propagates from the RF transmitter 306 to the RF receiver 310. The RF sensing data may include data corresponding to effects on a transmitted RF signal due to scattering, fading, and power attenuation over distance, or any combination thereof. In some examples, the RF sensing data may include imaginary and real data (e.g., I / Q components) corresponding to each tone in a frequency domain over a particular bandwidth.

[0091]

[0100] In some examples, the RF sensing data may be used to determine (e.g., calculate) a distance and an angle of arrival corresponding to a reflected waveform, such as the RX waveform 318. In some examples, the RF sensing data may be used to detect motion, determine a location, detect a change in location or motion patterns, obtain a channel estimate, determine characteristics (e.g., elevation angle, azimuth angle, distance, etc.) of the reflected path of the reflected waveform, or any combination thereof. In some cases, the distance and angle of arrival of the reflected signal may be used to identify a size, shape, and / or location of an object (e.g., object 302) in the surrounding environment. In some cases, the distance, azimuth angle, and / or elevation angle of the reflected path of the reflected signal with respect to the RF sensing coordinate system may be used to determine a dimension (e.g., shape, size, etc.) and / or location of an object (e.g., object 302) in the surrounding environment. In some cases, the wireless device 300 may perform a camera calibration to align a camera image coordinate system of the camera device(s) with the RF sensing coordinate system. In some examples, the camera image coordinate system may be from the FOV of the camera device(s) in the wireless device 300 and may be based on the location of the camera device(s) and the pose of the wireless device 300. In some examples, the wireless device 300 may use sensor data such as IMU data (with or without other data) to calculate the pose of the wireless device 300. In some examples, the wireless device 300 may use the pose of the XR device to calculate a reference coordinate of the FOV of the wireless device 300 and / or refine the position of an object (e.g., hand position, input device position, etc.) calculated using RF sensing.

[0092]

[0101] In some examples, the dimensions and / or location of the object may be used to determine whether the object is visible (e.g., within the FOV, unobstructed, etc.) to an image capture device (e.g., a camera device, an image sensor, etc.) of the wireless device 300. In some cases, if the object is not visible to the image capture device, the wireless device 300 may turn off or power down (e.g., to a lower power mode). The wireless device 300 may keep the image capture device turned off or powered down until the object (or another object of interest) is / becomes visible to the image capture device. In this way, the image capture device does not consume additional power (or consumes less power) during the period(s) during which the image capture device is unable to capture an image of the object(s) of interest. The object of interest may be, for example, a user's hand, an input device (e.g., a stylus, a controller, a glove, etc.), etc. In other examples, if the wireless device 300 includes more image sensors than ISPs (or any other processor or device resources) and an object is not visible to the image capture devices, the wireless device 300 can intelligently determine which image sensors to use (e.g., activate, utilize, etc.) and / or which image sensors to dedicated or share with a particular ISP (or any other processor or device resource). In some cases, this can reduce power and / or other computing resources by intelligently using a subset of the image sensors as opposed to using all image sensors all the time (e.g., leaving all image sensors on / active or powered up all the time).

[0093]

[0102] In some cases, the wireless device 300 may use signal processing, machine learning algorithms, any other suitable techniques, or any combination thereof to determine (e.g., calculate) the distance and angle of arrival corresponding to the reflected waveform (e.g., the distance and angle of arrival corresponding to the RX waveform 318). In other examples, the wireless device 300 may send the RF sensing data to another computing device, such as a server (e.g., server 172), which may determine (e.g., perform calculations to determine) the distance and angle of arrival corresponding to the RX waveform 318 or other reflected waveform.

[0094]

[0103] In some examples, the distance of the RX waveform 318 may be determined by measuring the time difference from receipt of the leakage signal 320 to receipt of the reflected signal. For example, the wireless device 300 may determine a baseline distance of zero, which is based on the difference (e.g., propagation delay) from the time the wireless device 300 transmits the TX waveform 316 to the time it receives the leakage signal 320. The wireless device 300 may then determine a distance associated with the RX waveform 318 based on the difference from the time the wireless device 300 transmits the TX waveform 316 to the time it receives the RX waveform 318, which may then be adjusted according to the propagation delay associated with the leakage signal 320. In doing so, the wireless device 300 may determine a distance traveled by the RX waveform 318, which may be used to determine the distance of the object (e.g., object 302) that caused the reflection.

[0095]

[0104] In some examples, the angle of arrival of the RX waveform 318 may be determined by measuring the time difference of arrival of the RX waveform 318 between individual elements of a receive antenna array, such as antenna 314. In some examples, the time difference of arrival may be determined by measuring the difference in the received phase at each element in the receive antenna array.

[0096]

[0105] In some cases, the distance and angle of arrival of the RX waveform 318 may be used to determine the distance between the wireless device 300 and the object 302 as well as the position of the object 302 relative to the wireless device 300. The distance and angle of arrival of the RX waveform 318 may also be used to determine the size and shape of the object 302 causing the reflection. For example, the wireless device 300 may utilize the determined distance and angle of arrival corresponding to the RX waveform 318 to determine the point at which the TX waveform 316 reflected off the object 302. The wireless device 300 may aggregate the reflection points for the various reflected signals to determine the size and shape of the object 302.

[0097]

[0106] For purposes of illustration and explanation, the object 302 is described throughout this disclosure as a human hand. However, one of ordinary skill in the art will recognize from this disclosure that the object 302 can include any other type of object. For example, the object 302 can include a different body part, a human, an animal, a device, a structure, or any other object or objects.

[0098]

[0107] As mentioned above, the wireless device 300 may include a wearable device such as a head-mounted device (e.g., an XR device, etc.), a mobile device such as a smartphone, a laptop, a tablet, etc. In some examples, the wireless device 300 may be configured to obtain device location data and device orientation data along with the RF sensing data. In some instances, the device location data and device orientation data may be used to determine or adjust the distance and angle of arrival of a reflected signal such as the RX waveform 318. For example, a user may be holding the wireless device 300 and walking around a room during an RF sensing process. In this instance, the wireless device 300 may have a first location and a first orientation when it transmits the TX waveform 316 and may have a second location and a second orientation when it receives the RX waveform 318. The wireless device 300 may take into account the change in location and the change in orientation when it processes the RF sensing data to determine the distance and angle of arrival. For example, the location data, orientation data, and RF sensing data may be correlated based on a timestamp associated with each element of data. In some techniques, a combination of location data, orientation data, and RF sensing data may be used to determine the size and location of the object 302.

[0099]

[0108] In some examples, device location data may be gathered by the wireless device 300 using techniques including round trip time (RTT) measurements, passive positioning, angle of arrival, received signal strength indicator (RSSI), CSI data, using any other suitable techniques, or any combination thereof. In some examples, device orientation data may be obtained from electronic sensors on the wireless device 300, such as a gyroscope, an accelerometer, a compass, a magnetometer, a barometer, any other suitable sensor, or any combination thereof. For example, a gyroscope on the wireless device 300 may be used to detect or measure changes in the orientation (e.g., relative orientation) of the wireless device 300, and a compass may be used to detect or measure the absolute orientation of the wireless device 300.

[0100]

[0109] 4 is a diagram illustrating an indoor environment 400 that may include one or more wireless devices configured to perform RF sensing. In some examples, the indoor environment 400 may include one or more mobile wireless devices (e.g., mobile device 402) that may be configured to perform RF sensing to optimize XR operations / functionality and / or resource usage as described further herein. In some cases, the indoor environment 400 may include one or more fixed wireless devices (e.g., access points (APs) 404) that may be configured to perform RF sensing.

[0101]

[0110] In some aspects, the AP 404 may be a Wi-Fi access point having a static or fixed location within the indoor environment 400. Although the indoor environment 400 is shown as having an access point (e.g., the AP 404), any type of fixed wireless device (e.g., a desktop computer, a wireless printer, a camera, a smart television, a smart appliance, etc.) may be configured to perform the techniques described herein. In some examples, the AP 404 may include hardware and software components that may be configured to simultaneously transmit and receive RF signals, such as the components described herein with respect to the wireless device 300. For example, the AP 404 may include one or more antennas (e.g., the TX antenna 406) that may be configured to transmit RF signals and one or more antennas (e.g., the RX antenna 408) that may be configured to receive RF signals. As discussed with respect to the wireless device 300, the AP 404 may include an omni-directional antenna and / or an antenna array configured to transmit and receive signals from any direction.

[0102]

[0111] In one aspect, the AP 404 may transmit an RF signal 410 that may reflect off one or more objects located in the indoor environment 400 (e.g., one or more objects located in the scene, a wall or other barrier, a device, a human, a body part, a structure, and / or other objects). For example, the RF signal 410 may reflect off a wall 422, causing a reflected signal 412 to be received by the AP 404 via the RX antenna 408. As another example, the RF signal may reflect off the hand of a user of a mobile device (e.g., the mobile device 402), causing a reflected signal to be received by the AP 404 and / or the mobile device via the respective RX antennas. When transmitting the RF signal 410, the AP 404 may also receive a leakage signal 414 corresponding to a direct path from the TX antenna 406 to the RX antenna 408.

[0103]

[0112] In some examples, the AP 404 can obtain RF sensing data associated with the reflected signal 412. For example, the RF sensing data can include CSI data corresponding to the reflected signal 412. In a further aspect, the AP 404 can obtain a distance D corresponding to the reflected signal 412. 1 and the angle of arrival θ 1 For example, the AP 404 may use the RF sensing data to determine the distance D by calculating the time of flight for the reflected signal 412 based on the difference between the leakage signal 414 and the reflected signal 412. 1 In a further example, the AP 404 can determine the angle of arrival θ by utilizing an antenna array (e.g., antenna 408) to receive the reflected signal and measuring the difference in the received phase at each element of the antenna array. 1 can be determined.

[0104]

[0113] In some examples, the AP 404 may use a distance D corresponding to the reflected signal 412 to identify the wall 422. 1 and the angle of arrival θ 1 In some aspects, the AP 404 may utilize the distance D corresponding to the reflected signal 412. 1 and the angle of arrival θ 1 Based on the received signal, the location, shape, and / or size of the wall 422 may be identified. In some aspects, the AP 404 may communicate with a server (e.g., the server 172) to provide data regarding the location, shape, and / or size of the wall 422. In some examples, the AP 404 may collect RF sensing data and provide the RF sensing data to a server for processing to calculate time-of-flight and angle-of-arrival for reflected signals.

[0105]

[0114] In some examples, the indoor environment 400 may include a mobile device 402. Although shown as a head-mounted XR device, the mobile device 402 may include any type of mobile device, such as a smartphone, a tablet, a laptop, a smartwatch, etc. According to some examples, the mobile device 402 may be configured to perform RF sensing to identify a position, shape, and / or size of one or more objects in the indoor environment 400.

[0106]

[0115] In some cases, the mobile device 402 can cause an RF waveform 416a to be transmitted via one of its RF transmitters, such as the RF transmitter 306. As shown, the RF waveform 416a is transmitted at time t=0. In some instances, the mobile device 402 can cause it to transmit a waveform 416a ... 1 At a later time, shown as , it may move while performing RF sensing so as to be at a different location.

[0107]

[0116] In some examples, the RF waveform 416a reflects off of an object 420 and the reflected waveform 418a is generated at time t=0+Δt 1 t, which can cause the RF waveform 416a to be received by the mobile device 402 at t=0+Δt. In some cases, the wavelength of the RF waveform 416a can be configured to allow the RF waveform 416a to penetrate and / or traverse the object 420 (shown as RF waveform 416b after penetrating the object 420) and reflect off a wall 424. The reflection 418b from the wall 424 traverses the object 420 and a second reflected waveform 418c is received at a later time, e.g., t=0+Δt 2 The received signal may be received by the mobile device 402 at

[0108]

[0117] In some examples, the mobile device 402 can collect RF sensing data corresponding to the reflected waveforms 418a and 418c. The mobile device 402 can also collect the RF sensing data corresponding to the time when the RF waveform 416a was transmitted (e.g., t=0) and also the time when the reflected waveform 418a was received (e.g., t=0+Δt 1 ) and the time at which the reflected waveform 418c was received (e.g., t=0+Δt 2 ) device location data and device orientation data corresponding to the

[0109]

[0118] In some aspects, the mobile device 402 can utilize the RF sensing data to determine the time of flight and angle of arrival for each reflected waveform 418a and 418c. In some examples, the mobile device 402 can utilize the location and orientation data to account for device movement during the RF sensing process. For example, the time of flight of the reflected waveforms 418a and 418c can be adjusted based on the movement of the device toward the object 420 and / or the wall 424, respectively. In another example, the angle of arrival of the reflected waveforms 418a and 418c can be adjusted based on the movement and orientation of the mobile device at the time the mobile device transmitted the RF waveform 416a versus the time the mobile device 402 received the reflected waveforms 418a and 418c.

[0110]

[0119] In some cases, the mobile device 402 can utilize time of flight, angle of arrival, location data, and / or orientation data to determine the size, shape, and / or position of the object 420 and / or wall 424. FIG. 5A is an example of a graphical representation 500 showing the width and distance of the object 420 and wall 424 based on RF sensing that may be performed by the mobile device 402.

[0111]

[0120] As shown, the graphical representation 500 may include an azimuth angle in degrees on the x-axis and a distance in centimeters on the y-axis. The graphical representation 500 may further include a reference to the object 420 and the wall 424 based on the azimuth angle and time of flight of the reflected signal. The graphical representation 500 illustrates that RF sensing techniques may be used to detect reflections from objects or walls behind one another. In this example, the RF waveform 416a generates a first reflection from the object 420 and a second reflection from the wall 424, which are received by the mobile device 402.

[0112]

[0121] The mobile device 402 can utilize the distance and azimuth data to identify the distance and width of the object 420 and the wall 424. In some techniques, the mobile device 402 can use the distance, azimuth, and elevation data to create a map of the indoor environment 400 that includes references to the object 420 and the wall 424. In other techniques, the mobile device 402 can use the RF sensing data to correct a partial map that it receives from a server, such as the server 172. In other aspects, the mobile device 402 can send the RF sensing data to a server for processing and creation of an indoor map of the indoor environment 400.

[0113]

[0122] In some examples, the AP 404 and the mobile device 402 may be configured to implement a bistatic configuration, in which the transmit and receive functions are performed by different devices. For example, the AP 404 (and / or other devices in the indoor environment 400, which may be static or fixed) may transmit an omnidirectional RF signal that may include signals 415a and 415b. As shown, the signal 415a may travel directly (e.g., without reflection) from the AP 404 to the mobile device 402. The signal 415b may reflect off a wall 426, causing a corresponding reflected signal 415c to be received by the mobile device 402.

[0114]

[0123] In some cases, the mobile device 402 can utilize RF sensing data associated with the direct signal path (e.g., signal 415a) and the reflected signal path (e.g., signal 415c) to identify the size and shape of the reflector (e.g., wall 426). For example, the mobile device 402 can obtain, retrieve, and / or estimate location data associated with the AP 404. In some aspects, the mobile device 402 can use the location data associated with the AP 404 and the RF sensing data (e.g., CSI data) to determine the time of flight, distance, and / or angle of arrival associated with the signals transmitted by the AP 404 (e.g., direct path signals such as signal 415a and reflected path signals such as signal 415c). In some cases, the mobile device 402 and the AP 404 can further send and / or receive communications that can include data associated with the RF signal 415a and / or the reflected signal 415c (e.g., transmission time, sequence / pattern, time of arrival, time of flight (TOF), angle of arrival, etc.).

[0115]

[0124] In some examples, the mobile device 402 and / or the AP 404 can obtain RF sensing data in the form of CSI data that can be used to construct a matrix based on a number of frequencies (e.g., tones), represented as “K,” and a number of antenna array elements, represented as “N.”

[0116]

[0125] Upon constructing the CSI matrix, the mobile device 402 and / or AP 404 can calculate the range, azimuth, and / or elevation angles for the direct and reflected signal paths by utilizing a two-dimensional Fourier transform.

[0117]

[0126] In some examples, the mobile device 402 and the AP 404 can perform RF sensing techniques regardless of their association with each other or with a Wi-Fi network. For example, the mobile device 402 can utilize its Wi-Fi transmitter and Wi-Fi receiver to perform RF sensing as described herein when it is not associated with an access point or Wi-Fi network. In a further example, the AP 404 can perform RF sensing techniques regardless of whether the AP 404 has a wireless device associated with it.

[0118]

[0127] 5B is another example of a graphical representation 520 illustrating the size (e.g., width, height, etc.) and location of objects 522, 524, and 526 determined by RF sensing as described herein. In some examples, the size and location of objects 522, 524, and 526 may be determined by mobile device 402 using RF sensing. In other examples, the size and location of objects 522, 524, and 526 may be determined by another device using RF sensing, such as server 172 or any other device including at least one RF interface.

[0119]

[0128] As shown, the graphical representation 520 may include angle of arrival (AoA) and elevation on the x-axis (and y-axis) and TOF / range on the z-axis. The graphical representation 520 may include references to objects 522, 524, and 526 based on the azimuth, elevation, and TOF / range of the signal. The graphical representation 520 illustrates that RF sensing techniques may be used to detect reflections from such objects to determine the size, shape, and / or location of the objects. In this example, the objects have different sizes and different locations relative to the mobile device 402.

[0120]

[0129] The size, shape, and / or location of the object 522 may be represented based on the azimuth, elevation, and TOF / range calculated for the direct path 530 of the transmitted or leaked signal. In some examples of bistatic configurations, the object 522 may represent the mobile device 402 and / or one or more components of the mobile device 402. Similarly, the size, shape, and / or location of the object 524 may be determined based on the azimuth, elevation, and TOF / range calculated for a selected reflected path 532 associated with one or more reflected signals. The selected reflected path 532 may include one or more reflected signal paths selected based on the distance of the one or more reflected signal paths relative to the direct path 530. For example, the selected path 532 may include a reflected path having a distance that is within a threshold beyond the distance associated with the direct path 530.

[0121]

[0130] In some cases, the distance and / or threshold distance used to select the selected path 532 may depend on the object of interest. For example, if the object of interest is the hand of a user wearing an XR device (e.g., mobile device 402), the distance of the selected path 532 may be shorter than the distance of one or more other reflected paths corresponding to one or more signals reflected from one or more other objects (e.g., walls, structures, etc.) not attached to the user, since the user's hand may be expected to be within a certain distance of the user (and thus the XR device worn by the user), such as 1.5m or 1m, and the one or more other objects may be somewhere within a larger range of distances to the user (and thus the XR device worn by the user), including distances that exceed the typical distance between the user's hand and the user (and / or the XR device worn by the user, the device held by the user, etc.), such as 1.5m or 1m.

[0122]

[0131] Moreover, the size, shape, and / or location of object 526 may be determined based on the azimuth, elevation, and TOF / range calculated for reflected path 534 associated with one or more signals reflected from object 526. In this example, object 526 is at a greater relative distance than objects 522 and 524. The greater relative distance in this example exceeds a threshold(s) determined for selecting the selected reflected path 532. In some examples, the threshold(s) may be expressed in terms of the difference between the TOF for the direct path and the TOF for each reflected path.

[0123]

[0132] In some examples, to detect the size, shape, and / or location of the object 524, the mobile device 402 (or another device) may select one or more reflected paths (e.g., paths of the reflected signal) that are within a threshold distance relative to the direct path 530 and estimate the azimuth and elevation angles of each selected reflected path to measure the dimensions of the object 524. For example, in an illustrative example, if the object of interest is a hand and the threshold distance is one meter, the mobile device 402 may select one or more reflected paths whose distance relative to the direct path 530 is less than or equal to one meter.

[0124]

[0133] In some cases, an object classification algorithm (e.g., signal processing, machine learning, etc.) may be implemented to classify the object 524. For example, if the object of interest is a hand, an object classification algorithm may be implemented to classify the object 524 as a hand or not a hand.

[0125]

[0134] In some examples, RF sensing techniques can implement signal processing to extract reflections and focus on reflections that are shorter in distance (e.g., within a distance threshold) to detect objects close to the mobile device 402, such as a user's hand, controller, etc., to reduce computational complexity and / or to reduce power consumption in the mobile device 402. In some cases, RF sensing can be used to estimate the location and / or dimensions of the object producing the reflection. In some examples, object classification algorithms (e.g., signal processing, machine learning, etc.) can be used to perform a binary classification of the object to indicate whether or not the object is a particular type of object. For example, the object classification algorithm can perform a binary classification to indicate whether or not the object is a hand.

[0126]

[0135] RF sensing can track in 360 degrees and is not limited by the camera view. In some examples, RF sensing can track objects outside of the camera view. For example, RF sensing can track a user's hand when the user's hand is outside of or obstructed from the camera view. In some cases, RF sensing can provide an estimate of the object's location. An electronic device, such as the mobile device 402, can use the estimate to predict the object's future position and / or achieve smooth object tracking. In some cases, an electronic device, such as the mobile device 402, can provide the estimate to another device, such as a server or another electronic device, to help predict the object's future position and / or achieve smooth object tracking. In some examples, the electronic device can signal the object's location (e.g., calculated using RF sensing) to a destination device (e.g., a server or any other device) regardless of whether the object is outside of (or obstructed from) the camera view. For example, as previously described, the electronic device can use RF sensing to calculate the position of a user's hand(s) even if the user's hand(s) are outside the FOV of the camera device(s) on the electronic device. The electronic device can signal the calculated position of the user's hand(s) to a destination device, such as a server or a mobile phone, regardless of whether the user's hand(s) are within the FOV of the camera device(s).

[0127]

[0136] In some examples, the size, shape, and / or location of an object may be determined using RF detection to determine whether such object is visible to an image capture device of a mobile device (e.g., mobile device 402). For example, the size, shape, and / or location of an object may be determined using RF detection to determine whether the object is within the FOV of the image capture device and / or whether the object is occluded (e.g., whether the image capture device's view of the object is obstructed). In some cases, if the location of the object of interest does not correspond to the FOV of the image capture device (e.g., the object is not within the FOV of the image capture device), or if the location of the object corresponds to the FOV of the image capture device but the object is possibly occluded / obstructed from the view / visibility of the image capture device, the mobile device associated with the image capture device may turn off the image capture device or set the power setting of the image capture device to a lower power mode. In some examples, when a mobile device turns off an image capture device, the mobile device can use RF sensing to monitor hand movement and proactively turn on the image capture device when the hand is approaching the FOV of the image capture device (e.g., approaching the FOV within a threshold time frame, distance, etc.) such that the image capture device can capture an image(s) of the hand when the hand is in the FOV of the image capture device. The mobile device can use the image(s) once the hand is within the FOV of the image capture device to determine hand gestures.

[0128]

[0137] Because the image capture device cannot capture images of the object of interest while the object is occluded or outside the FOV of the image capture device, the mobile device may turn off or power down the image capture device to conserve power. If the object of interest subsequently comes within the FOV of the image capture device or if the object is no longer occluded / obstructed, the object is now visible to the image capture device, thus allowing the image capture device to capture an image of the object, and the mobile device may turn on the image capture device or set the power setting of the image capture device to a higher power mode to allow the image capture device to capture the image(s) of the object and / or a higher quality image(s).

[0129]

[0138] In some examples, to reduce power consumption and / or optimize resource usage in a mobile device, when an object of interest is not within or is occluded from one or more image sensors in the mobile device, the mobile device can control which image sensors in the mobile device are used by which processors in the mobile device when the object of interest is not within the FOV of the one or more image sensors in the mobile device, such that the one or more image sensors are unable to capture an image of the object. For example, in some cases, a mobile device can include multiple image sensors and multiple ISPs for processing image data from the image sensors. In some cases, when the number of image sensors exceeds the number of ISPs, the multiple image sensors can share the same ISP for processing image data from such image sensors.

[0130]

[0139] However, if an object of interest is within the FOV of an image sensor and is not otherwise obstructed from the view of the image sensor, the mobile device may dedicate an ISP to that image sensor for processing image data from that image sensor. If instead, an object of interest is obstructed or outside the FOV of some image sensors, the mobile device may instead allow those image sensors to share the same ISP. In some cases, the mobile device may intelligently allocate image sensors to ISPs (or vice versa) to increase or decrease image processing capabilities for some image sensors (or for image data from some image sensors) depending on whether the object of interest is within the FOV of one or more image sensors or obstructed from the view of one or more image sensors. The mobile device may use RF sensing as described herein to determine whether an object is within the FOV, outside the FOV, obstructed and therefore not visible to the image sensor(s), etc.

[0131]

[0140] In other examples, the size, shape, and / or location of an object may be determined using RF sensing to reduce the size of an uplink frame sent to a particular device, such as a server, for tracking operations. For example, in some cases, the mobile device 402 may capture an image of the object(s) in a scene and send the captured image to a destination device that uses such image to track the object(s) in the scene. To reduce bandwidth usage, latency, etc., the mobile device 402 may reduce the size of the image sent to the destination device by cropping the image to include the object(s) in the image and exclude other portions of the image. The mobile device 402 may use RF sensing to detect the object(s) and determine the location of the object(s). The mobile device 402 may use the determined location of the object(s) to determine how / where to crop the image capturing the object(s). The cropped image may be smaller and thus may enable the mobile device 402 to reduce the size of the uplink traffic including the cropped image sent to the destination device.

[0132]

[0141] For example, FIG. 6 illustrates an example use case 600 for using RF sensing to reduce uplink traffic from an XR device 620 to a destination device 610. The uplink traffic can include tracking frames captured by an image capture device including one or more image sensors on the XR device 620, and the destination device 610 can use those tracking frames to track one or more objects (e.g., hands, devices, etc.) in a scene. In some cases, the destination device 610 can additionally or instead use the tracking frames for other operations, such as, for example, mapping the scene and / or features in the scene. In some cases, the destination device 610 can be a server, such as, for example, an edge server on a cloud network. In other cases, the destination device 610 can be a user electronic device, such as, for example, a smartphone, tablet, laptop, game console, etc.

[0133]

[0142] As shown, the destination device 610 can send downlink (DL) frames 622, 624, and 626 to the XR device 620 according to a wireless communication protocol standard as described above. The XR device 620 can use the DL frames 622-626 to detect nearby reflectors (e.g., objects that cause reflections of signals transmitted by the destination device 610) using RF sensing. The XR device 620 can use RF sensing to detect the location, shape, and / or size of nearby reflectors. In some examples, the XR device 620 can use the CSI captured in the DL frames 622-626 to extract / identify reflection information, such as azimuth, elevation, and / or distance of the reflection path / signal.

[0134]

[0143] In block 630, the XR device 620 can estimate the location of the nearby reflector (and optionally the size and / or shape of the nearby reflector) based on the reflection information. For example, the XR device 620 can determine the azimuth, elevation and TOF / range of the reflection path associated with the nearby reflector to calculate the three-dimensional (3D) position of the nearby reflector in physical space.

[0135]

[0144] In block 632, the XR device 620 can use the determined location of the reflector of interest to extract a partial image that captures the reflector of interest. For example, the XR device 620 can capture an image of the reflector of interest (e.g., via an image capture device of the XR device 620). The XR device 620 can use the determined location of the reflector to determine how to crop the image to generate a smaller / partial image that includes the reflector but excludes other portions of the captured image. The XR device 620 can then crop the image to generate a smaller / partial image that includes the reflector.

[0136]

[0145] The XR device 620 can send a smaller / partial image including the reflector to the destination device 610 in the UL frame 640. Because the image including the reflector has been cropped / reduced, the UL frame 640 can be smaller than if the XR device 620 had instead sent the captured image without first cropping it as previously described. As a result, the XR device 620 can reduce the size of the UL traffic to the destination device 610, the latency of the UL traffic, etc.

[0137]

[0146] In some implementations, if the XR device 620 supports multi-link operation (MLO), the XR device 620 can use a Wi-Fi radar to capture CSI and perform RF sensing. For example, the XR device 620 can send a Wi-Fi sounding signal to the destination device 610 using one link while maintaining DL / UL traffic on another link to avoid creating extra on-time for the XR device 620. In other cases, if the XR device 620 does not support MLO, the XR device 620 can slightly increase the Wi-Fi on-time to cover the Wi-Fi radar airtime. For example, the XR device 620 can increase the Wi-Fi on-time at the beginning or end of the device data TX / RX window.

[0138]

[0147] In some cases, the XR device 620 can perform RF sensing with a single received frame and can use the single received frame to learn the surrounding environment. In other cases, the XR device 620 can optionally use multiple received frames for RF sensing to improve detection accuracy (e.g., using available existing DL data frames).

[0139]

[0148] 7A and 7B are diagrams illustrating example rendering scenarios for XR involving communication between a destination device 702, a mobile device 704, and an XR device 706. FIG. 7A illustrates a rendering scenario 700 for XR in which virtual content associated with an XR application is rendered by a mobile device 704.

[0140]

[0149] In this example, the destination device 702 can send DL data 710 to the mobile device 704 and can receive UL data 720 from the mobile device 704. In some cases, the link between the destination device 702 and the mobile device 704 can be separated from the link between the mobile device 704 and the XR device 706. The DL data 710 can include, for example, but not limited to, virtual content associated with the XR experience / application, tracking information, mapping information, and / or any other XR data. The UL data 720 can include, for example, but not limited to, tracking frames, location information, data requests, etc.

[0141]

[0150] In some cases, the destination device 702 may be a server, such as, for example, an edge server on a cloud network. In other cases, the destination device 702 may be any electronic device, such as, for example, a laptop, a desktop computer, a tablet, a game console, etc. The mobile device 704 may include any electronic device, such as, for example, a smartphone, a tablet, a laptop, an IoT device, a game console, etc. The XR device 706 may include, for example, an XR (e.g., AR, VR, etc.) wearable device, such as, for example, an HMD, a smart glass, etc.

[0142]

[0151] The mobile device 704 can receive DL data 710 and send DL data 712 to the XR device 706, which can use the DL data 712 for the XR presentation / experience. In some examples, the DL data 712 can include rendered virtual content. For example, the mobile device 704 can render virtual content from the destination device 702 and send the rendered virtual content to the XR device 706. In some cases, the DL data 712 can include XR-related data, such as, for example, a depth map, an eye buffer (e.g., render texture, eye buffer resolution, etc.), tracking information, and / or any other XR data.

[0143]

[0152] The XR device 706 can obtain sensor data, such as inertial measurements, image data, etc., and provide such data to the mobile device 704. For example, the XR device 706 can send UL data 722 including data from one or more inertial measurement units (IMUs) and tracking frames from an image capture device in the XR device 706 to the mobile device 704. In some cases, the mobile device 704 can use the UL data 722 to render virtual content according to the position of the XR device 706 and / or one or more objects in a scene.

[0144]

[0153] 7B illustrates another example rendering scenario 725 in which virtual content is rendered at a destination device 702. In this example, the destination device 702 can send DL data 730 to a mobile device 704 and receive UL data 740 from the mobile device 704. The mobile device 704 can send DL data 732 to an XR device 706 and receive UL data 742 from the XR device 706. In this example, the mobile device 704 can be used as a pass-through with minimal or lower processing for the XR data between the destination device 702 and the XR device 706.

[0145]

[0154] In some cases, the UL data 722 and UL data 742 from the XR device 706 may include partial frames generated as previously described with respect to FIG. 6. For example, the UL data 722 and UL data 742 may include tracking frames generated by cropping an image of the object of interest. The image may be cropped based on a location of the object of interest estimated using RF sensing. For example, the XR device 706 may capture an image of the object, estimate the location of the object using RF sensing, and use the location of the object to crop the image of the object to reduce the size of the object of interest while still capturing it. The XR device 706 may provide the cropped image in the UL data 722 or UL data 742. The destination device 702 may use the cropped image to help track the object and / or predict the future location of the object.

[0146]

[0155] 8 is a flow chart illustrating an example process 800 for performing XR optimization using RF sensing. At block 802, the process 800 can include obtaining RF sensing data. In some examples, the sensing data can include channel state information (CSI). In some cases, the RF sensing data can include a set of data associated with a received waveform that is a reflection of a transmitted waveform reflected from an object. In some examples, the transmitted waveform can include a signal (e.g., a Wi-Fi signal) transmitted by an antenna from a wireless device such as an XR device (e.g., XR device 620, XR device 706).

[0147]

[0156] In some cases, the RF sensing data may include CSI data corresponding to reflections received in response to transmitting a signal. In one illustrative example, the RF sensing data may include Wi-Fi CSI data corresponding to reflections received in response to transmitting a Wi-Fi signal. In other examples, the RF sensing data may include CSI data obtained using 5G NR, Bluetooth, UWB, 60 GHz mmWave, any combination thereof, or other type(s) of signal(s).

[0148]

[0157] In some examples, the RF sensing data may include data associated with a received leakage waveform corresponding to the transmitted waveform and / or associated with one or more reflected waveforms corresponding to the one or more reflected RF signals.

[0149]

[0158] At block 804, the process 800 may include determining one or more reflection paths of the one or more reflected RF signals based on the RF sensing data. In some examples, each reflected RF signal may include a reflection of the transmitted RF signal from one or more objects in the physical space. In some cases, at least one of the one or more objects may include a hand associated with a user of a mobile device, such as an XR device, a smartphone, or the like.

[0150]

[0159] In some examples, determining one or more reflected paths of the one or more reflected RF signals may include determining a path of the RF signal that includes a direct path of the transmitted RF signal and determining a location of the one or more objects relative to the mobile device based on the path of the RF signal.

[0151]

[0160] At block 806, process 800 may include comparing the one or more reflected paths to an FOV of an image capture device associated with the mobile device. In some cases, comparing the one or more reflected paths to the FOV of the image capture device may include determining, based on the one or more reflected paths, whether the one or more objects are within a portion of a scene that corresponds to the FOV of the image capture device associated with the mobile device. In some aspects, process 800 may include determining a location of the one or more objects relative to the mobile device. In some cases, the location of the one or more objects may be determined based on a determined path of the RF signal, including a direct path of the transmitted RF signal and / or one or more reflected paths of the one or more reflected RF signals.

[0152]

[0161] In some examples, determining the location of the one or more objects may include determining a respective distance, a respective angle of reflection, and / or a respective angle of elevation associated with the one or more reflected paths. In some cases, the location of the one or more objects may be determined based on the respective distance, the respective angle of reflection, and / or the respective angle of elevation.

[0153]

[0162] In some cases, determining the location of the one or more objects may include determining a respective distance, azimuth, and / or elevation angle of a path between the mobile device and each of the one or more objects. In some cases, the location of the one or more objects may be determined based on the respective distance, azimuth, and / or elevation angle.

[0154]

[0163] At block 808, process 800 may include triggering an action by the mobile device and / or an image capture device associated with the mobile device based on the comparison. In some cases, triggering the action may be based on a determination of whether one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device. In some examples, the determination of whether one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device may be based on the comparison.

[0155]

[0164] In some examples, the triggered action may include controlling a power setting of the image capture device. In some cases, controlling the power setting of the image capture device may be further based on a light source level falling below a threshold and / or a privacy setting. In some examples, the privacy setting may be based on user input, application data, and / or Global Navigation Satellite System (GNSS) data.

[0156]

[0165] In some examples, the triggered action may include extracting a portion of the image that includes the one or more objects. In some cases, the triggered action may include determining whether to capture one or more images of the one or more objects. In some examples, the triggered action may be further based on a determination that the one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device. In some examples, the determination that the one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device may be based on a comparison.

[0157]

[0166] In some cases, determining whether one or more objects are within a portion of a scene corresponding to the FOV of the image capture device may be based on respective distances, respective azimuth angles, and / or respective elevation angles associated with one or more reflected paths and / or one or more reflected RF signals.

[0158]

[0167] In some examples, determining whether the hand is within the portion of the scene corresponding to the FOV of the image capture device may include determining, based on the comparison, that the one or more objects are outside the FOV of the image capture device. In some cases, the process 800 may include setting a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within the portion of the scene corresponding to the FOV of the image capture device, based on the determination that the one or more objects are outside the FOV of the image capture device. In some cases, the adjusted power state may be an off state. In other cases, the adjusted power state may be an on state associated with a lower power mode than a power mode of the image capture device when the one or more objects are within the portion of the scene corresponding to the FOV of the image capture device.

[0159]

[0168] In some examples, process 800 may include determining, based on the comparison, whether the one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device. In some cases, determining whether the one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device may include determining that the image capture device's view of the one or more objects is obstructed (e.g., blocked from the image capture device's view) by at least one object.

[0160]

[0169] In some aspects, process 800 may include determining that one or more objects are moving toward a portion of the scene that corresponds to the FOV of the image capture device, and adjusting a power setting of the image capture device to a different power state based on determining that the one or more objects are moving toward the portion of the scene that corresponds to the FOV of the image capture device.

[0161]

[0170] In some embodiments, process 800 may include determining a size and / or shape of one or more objects based on one or more reflected paths.

[0162]

[0171] In some aspects, process 800 may include determining that an image capture device's view of the one or more objects is obstructed by at least one object. In some cases, determining that the image capture device's view of the one or more objects is obstructed by at least one object includes determining that the one or more objects are within a portion of the scene that corresponds to the FOV of the image capture device and determining that the image capture device's view of the one or more objects is obstructed by the at least one object based on a location of the one or more objects.

[0163]

[0172] In some cases, the image capture device may include multiple image sensors. In some examples, the triggered action may include controlling a power setting of the image capture device. In some cases, controlling the power setting of the image capture device may include controlling individual power settings of the multiple image sensors. In some cases, controlling individual power settings of the multiple image sensors may include dedicating at least one of the multiple processors of the device to a particular one of the multiple image sensors for image processing based on determining that the hand is within the FOV of the particular one of the multiple image sensors.

[0164]

[0173] In some aspects, process 800 may include determining, based on the comparison, that one or more objects are within a portion of the scene corresponding to the FOV of the image capture device, and determining, based on the location of the one or more objects, that a view of the image capture device of the hand is obstructed by at least one object. In some aspects, process 800 may include controlling use of the multiple image capture devices of the mobile device by multiple processors (e.g., an ISP, etc.) of the mobile device based on determining that the hand is outside the FOV of the image capture device. In some cases, controlling use of the multiple image capture devices by the multiple processors may include dedicating, to one of the multiple processors, a particular one of the multiple image capture devices that was previously shared by two or more of the multiple processors.

[0165]

[0174] In some aspects, the process 800 may include controlling use of a plurality of image capture devices of the mobile device by a plurality of processors of the mobile device based on determining that one or more objects are outside the FOV of the image capture device. In some examples, the plurality of image capture devices may include the image capture devices described above. In some cases, controlling use of the plurality of image capture devices by the plurality of processors may include dedicating to one of the plurality of processors a particular one of the plurality of image capture devices previously shared by two or more of the plurality of processors. In some examples, each of the plurality of processors may include an image signal processor and each of the plurality of image capture devices may include an image sensor.

[0166]

[0175] In some aspects, process 800 may include determining that one or more objects are outside the FOV of the image capture device and moving toward a portion of the scene corresponding to the FOV of the image capture device, and in response to determining that one or more objects are outside the FOV of the image capture device and moving toward the portion of the scene corresponding to the FOV of the image capture device, switching an active camera setting from the image capture device to a different image capture device, and the switched active camera setting triggering the apparatus to use the different image capture device to capture one or more images.

[0167]

[0176] In some aspects, the process 800 may include determining a shape of an object from one or more objects based on one or more reflection paths, and determining that the object is a hand associated with the user based on the shape of the object. In some aspects, the process 800 may include generating a cropped image of the hand using an image captured by an image capture device, and sending the cropped image to a destination device (e.g., destination device 702, mobile device 704). In some cases, the process 800 may include sending to the destination device the cropped image and a request for a tracked pose of the hand and / or a hand gesture within a map of the physical space. In some examples, the cropped image may be generated based on the location of one or more objects. In some examples, the location of the one or more objects includes a hand location. In some cases, the destination device may include a server (e.g., server 172) and / or a mobile device (e.g., UE 104, user device 210, wireless device 300, mobile device 402, mobile device 704).

[0168]

[0177] In some aspects, the process 800 may include determining a location of the one or more objects relative to the mobile device based on the one or more reflected paths. In some cases, a cropped image is generated based on the location of the one or more objects. In some examples, the location of the one or more objects may include a hand location. In some aspects, the process 800 may include selecting at least one of the one or more reflected paths based on the distance of each of the associated objects being within a distance threshold.

[0169]

[0178] In some examples, the mobile device may be an XR device (e.g., XR device 620, XR device 706). In some cases, the mobile device may include a head mounted display.

[0170]

[0179] In some aspects, the process 800 may include using a machine learning algorithm to detect a hand associated with a user of the mobile device and / or an obstruction of the view of the image capture device relative to the hand. In some cases, the hand may include at least one of the one or more objects.

[0171]

[0180] In some aspects, process 800 may include determining a map of the physical space and / or a hand gesture associated with the user's hand based on one or more reflex paths.

[0172]

[0181] In some examples, the processes described herein (e.g., process 800 and / or other processes described herein) may be performed by a computing device or apparatus (e.g., a UE, an XR device, etc.). In one example, process 800 may be performed by user device 210 of FIG. 2. In another example, process 800 may be performed by a computing device having a computing system 900 shown in FIG. 9. For example, a computing device having the computing architecture shown in FIG. 9 may include components of user device 210 of FIG. 2 and may implement the operations of FIG. 8.

[0173]

[0182] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) configured to perform steps of processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to communicate and / or receive data, any combination thereof, and / or other component(s). The one or more network interfaces may be configured to communicate and / or receive wired and / or wireless data, including data according to 3G, 4G, 5G, and / or other cellular standards, data according to the Wi-Fi (802.11x) standard, data according to the Bluetooth standard, data according to the Internet Protocol (IP) standard, and / or other types of data.

[0174]

[0183] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0175]

[0184] Process 800 is illustrated as a logical flow diagram, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0176]

[0185] Additionally, process 800 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored in a computer-readable or machine-readable storage medium, for example in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0177]

[0186] 9 is a diagram illustrating an example of a system for implementing some aspects of the present technology. In particular, FIG. 9 illustrates an example of a computing system 900, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device that constitutes any of the components thereof, in which the components of the system communicate with each other using a connection 905. The connection 905 may be a physical connection to a processor 910 using a bus, or a direct connection to the processor 910, such as in a chipset architecture. The connection 905 may also be a virtual connection, a networked connection, or a logical connection.

[0178]

[0187] In some embodiments, computing system 900 is a distributed system in which the functions described in this disclosure may be distributed in a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the system components described represent many such components, each of which performs some or all of the functions described for it. In some embodiments, a component may be a physical or virtual device.

[0179]

[0188] The exemplary system 900 includes at least one processing unit (CPU or processor) 910 and a connection 905 that communicatively couples various system components, including system memory 915, such as read only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910. The computing system 900 may include a cache 912 of high-speed memory directly connected to the processor 910, in close proximity to the processor 910, or integrated as part of the processor 910.

[0180]

[0189] The processor 910 may include any general purpose processor and hardware or software services, such as services 932, 934, and 936 stored in a storage device 930, configured to control the processor 910 as well as special purpose processors, where software instructions are incorporated into the actual processor design. The processor 910 may essentially be a fully self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0181]

[0190] To enable user interaction, computing system 900 includes input device(s) 945, which may represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 900 may also include output device(s) 935, which may be one or more of several output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input and output for communicating with computing system 900.

[0182]

[0191] The computing system 900 may include a communications interface 940 that may generally govern and manage user input and system output. The communications interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet® port / plug, a fiber optic port / plug, a proprietary wired port / plug, 3G, 4G, 5G and / or other cellular data network wireless signal transmission, Bluetooth® wireless signal transmission, Bluetooth® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 902 .11 Reception and / or transmission of wired or wireless communications may be performed or facilitated using wired transceivers and / or wireless transceivers, including those that utilize Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX®), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.

[0183]

[0192] The communication interface 940 may also include one or more GNSS receivers or transceivers used to determine the location of the computing system 900 based on reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite System (GNSS) systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operating on any particular hardware configuration, and thus the basic features herein may be readily substituted with improved hardware or firmware configurations as they are developed.

[0184]

[0193] The storage device 930 may be a non-volatile and / or non-transitory and / or computer readable memory device, and may be a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a Stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (e.g., a level 1 (L1) cache, a level 2 (L2) cache, a level 3 (L3) cache, a level 4 (L4) cache, a level 5 (L5) cache, or other (L#) cache), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0185]

[0194] Storage devices 930 may include software services, servers, services, etc., which, when code defining such software is executed by processor 910, cause the system to perform a function. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium in relation to the necessary hardware components, such as processor 910, connections 905, output devices 935, etc., to perform that function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data.

[0186]

[0195] A computer-readable medium may include non-transitory media on which data may be stored, which does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0187]

[0196] Those skilled in the art will recognize that, although specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein, the present application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in various ways, except as limited by the prior art, and the appended claims are to be construed to include such variations. Various features and aspects of the applications described above may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0188]

[0197] For clarity of explanation, in some cases, the technology may be presented as including individual functional blocks comprising devices, device components, steps or routines in a method implemented in software, or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0189]

[0198] Moreover, those skilled in the art will appreciate that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0190]

[0199] Individual embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0191]

[0200] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.

[0192]

[0201] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0193]

[0202] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may in some cases be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof, depending in part on the particular application, in part on the desired design, in part on the corresponding technology, etc.

[0194]

[0203] The various example logic blocks, modules, and circuits described with respect to the aspects disclosed herein may be implemented or performed using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, as further examples.

[0195]

[0204] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0196]

[0205] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0197]

[0206] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0198]

[0207] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0199]

[0208] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0200]

[0209] The phrases "coupled to" or "communicatively coupled to" refer to any component that is physically connected to another component, either directly or indirectly, and / or that is in communication with another component, either directly or indirectly (e.g., connected to another component via a wired or wireless connection and / or other suitable communications interface).

[0201]

[0210] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or more than one member of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0202]

[0211] Illustrative examples of the present disclosure include the following:

[0203]

[0212] Aspect 1: An apparatus comprising at least one memory and one or more processors coupled to the at least one memory, the one or more processors configured to acquire radio frequency (RF) detection data; determine one or more reflection paths of one or more reflected RF signals based on the RF detection data, where each reflected RF signal comprises a reflection of a transmitted RF signal from one or more objects in physical space; compare the one or more reflection paths to a field of view (FOV) of an image capture device associated with the apparatus; and trigger an action by at least one of the apparatus and the image capture device based on the comparison.

[0204]

[0213] Aspect 2: The apparatus of aspect 1, wherein the one or more processors are configured to determine, based on the comparison, that the one or more objects are outside the FOV of the image capture device, and, based on determining that the one or more objects are outside the FOV of the image capture device, set a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device.

[0205]

[0214] Aspect 3: The apparatus of aspect 2, wherein the one or more processors are configured to determine that one or more objects are moving toward a portion of the scene that corresponds to the FOV of the image capture device, and adjust a power setting of the image capture device to a different power state based on the determination that the one or more objects are moving toward the portion of the scene that corresponds to the FOV of the image capture device.

[0206]

[0215] Aspect 4: The apparatus of any of aspects 1 to 3, wherein the one or more processors are configured to: determine that the image capture device's view of the one or more objects is obstructed by at least one object; and, based on determining that the image capture device's view of the one or more objects is obstructed by the at least one object, set a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device, wherein triggering an action is further based on determining that the image capture device's view of the one or more objects is obstructed.

[0207]

[0216] Aspect 5: The apparatus of aspect 4, wherein to determine that the image capture device's view of the one or more objects is obstructed by at least one object, the one or more processors are further configured to determine, based on the comparison, that the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device, and to determine, based on the location of the one or more objects, that the image capture device's view of the one or more objects is obstructed by the at least one object.

[0208]

[0217] Aspect 6: An apparatus as described in any of aspects 1 to 5, wherein the image capture device comprises multiple image sensors, and wherein the triggered action comprises controlling power settings of the image capture device, and wherein to control the power settings of the image capture device, the one or more processors are configured to control individual power settings of the multiple image sensors.

[0209]

[0218] Aspect 7: The device of aspect 6, wherein to control individual power settings of the multiple image sensors, the one or more processors are configured to dedicate at least one of the device's multiple processors to a particular one of the multiple image sensors for image processing based on determining that the hand is within the FOV of the particular one of the multiple image sensors.

[0210]

[0219] Aspect 8: An apparatus as described in any of aspects 1 to 7, wherein the one or more processors are configured to control use of multiple image capture devices of the apparatus by multiple processors of the apparatus based on determining that one or more objects are outside the FOV of the image capture device, the multiple image capture devices including the image capture device.

[0211]

[0220] Aspect 9: The apparatus of aspect 7, wherein to control use of the multiple image capture devices by the multiple processors, one or more processors are configured to dedicate to one of the multiple processors a particular one of the multiple image capture devices that was previously shared by two or more of the multiple processors.

[0212]

[0221] Example 10: The apparatus of example 7, wherein each of the multiple processors comprises an image signal processor and each of the multiple image capture devices comprises an image sensor.

[0213]

[0222] Aspect 11: An apparatus as described in any of aspects 1 to 10, wherein the one or more processors are configured to determine, based on the comparison, that one or more objects are outside the FOV of the image capture device and are moving toward a portion of the scene corresponding to the FOV of the image capture device, and in response to determining that one or more objects are outside the FOV of the image capture device and are moving toward a portion of the scene corresponding to the FOV of the image capture device, switch an active camera setting from the image capture device to a different image capture device, and the switched active camera setting triggers the apparatus to use the different image capture device to capture one or more images.

[0214]

[0223] Example 12: An apparatus described in any of examples 1 to 11, wherein at least one of the one or more objects comprises a hand associated with a user of the apparatus.

[0215]

[0224] Aspect 13: An apparatus described in any of aspects 1 to 12, wherein one or more processors are configured to determine a path of an RF signal comprising a direct path of a transmitted RF signal, and to determine a location of one or more objects relative to the apparatus based on the path of the RF signal.

[0216]

[0225] Aspect 14: An apparatus described in any of aspects 1 to 13, wherein one or more processors are configured to determine at least one of respective distances, respective azimuth angles, and respective elevation angles associated with one or more reflected paths, and to determine locations of one or more objects relative to the apparatus based on at least one of the respective distances, respective azimuth angles, and respective elevation angles.

[0217]

[0226] Aspect 15: An apparatus described in any of aspects 1 to 14, wherein the triggered action comprises controlling a power setting of an image capture device, wherein controlling the power setting of the image capture device is further based on the light source level falling below a threshold.

[0218]

[0227] Aspect 16: The apparatus of any of aspects 1 to 14, wherein the triggered action comprises controlling a power setting of the image capture device, wherein controlling the power setting of the image capture device is further based on a privacy setting, wherein the privacy setting is based on at least one of user input, application data, and global navigation satellite system (GNSS) data.

[0219]

[0228] Aspect 17: An apparatus described in any of aspects 1 to 16, wherein the one or more processors are configured to determine at least one of a size and a shape of the one or more objects based on the RF detection data and one or more reflection paths.

[0220]

[0229] Aspect 18: An apparatus as described in any of aspects 1 to 17, wherein the one or more processors are configured to determine a shape of an object from one or more objects based on the RF sensing data and one or more reflected paths, determine that the object comprises a hand associated with a user of the apparatus based on the shape of the object, and generate a cropped image of the hand using an image captured by the image capture device.

[0221]

[0230] Aspect 19: The device of aspect 18, wherein the one or more processors are configured to determine a location of one or more objects relative to the device based on one or more reflected paths, and wherein a cropped image is generated based on the locations of the one or more objects, and wherein the locations of the one or more objects comprise a hand location.

[0222]

[0231] Aspect 20: The apparatus of aspect 18, wherein the one or more processors are configured to select at least one of the one or more reflection paths from the one or more reflection paths based on the respective distances of the associated objects being within a distance threshold.

[0223]

[0232] Example 21: The apparatus of example 18, wherein the one or more processors are configured to send the cropped image to a destination device.

[0224]

[0233] Aspect 22: The apparatus of aspect 21, wherein the destination device comprises at least one of a server and a mobile device, wherein the apparatus comprises an extended reality device.

[0225]

[0234] Aspect 23: An apparatus as described in any of aspects 1 to 22, wherein at least one of the one or more objects comprises a hand of a user of the apparatus, and wherein the one or more processors are configured to determine at least one of a map of the physical space and a hand gesture associated with the user's hand based on the RF detection data and the one or more reflected paths.

[0226]

[0235] Example 24: An apparatus as described in any of examples 1 to 23, wherein the triggered action comprises extracting a portion of the image including one or more objects.

[0227]

[0236] Aspect 25: An apparatus as described in any of aspects 1 to 24, wherein the triggered action comprises determining whether to capture one or more images of the one or more objects, the triggered action being further based on a determination that the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device.

[0228]

[0237] Example 26: An apparatus described in any of examples 1 to 25, wherein the apparatus comprises an augmented reality device.

[0229]

[0238] Aspect 27: The apparatus of aspect 26, wherein the augmented reality device comprises a head-mounted display.

[0230]

[0239] Aspect 28: An apparatus described in any of aspects 1 to 27, wherein the one or more processors are configured to use a machine learning algorithm to detect at least one of a hand associated with a user of the apparatus and an obstruction of a view of the image capture device relative to the hand, where the hand comprises at least one of one or more objects.

[0231]

[0240] Example 29: The apparatus of any of examples 1 to 28, wherein the RF sensing data comprises channel state information (CSI) data.

[0232]

[0241] Example 30: An apparatus according to any of examples 1 to 29, wherein the apparatus comprises a mobile device.

[0233]

[0242] Example 31: The apparatus of any of examples 1 to 30, wherein the mobile device comprises a wearable device.

[0234]

[0243] Aspect 32: The apparatus of any of aspects 1 to 31, wherein the transmitted RF signal comprises a downlink physical layer protocol data unit (DL-PPDU) from a destination device.

[0235]

[0244] Aspect 33: An apparatus as described in any of aspects 1 to 32, wherein the transmitted RF signal comprises a Wi-Fi radar signal.

[0236]

[0245] Aspect 34: A method comprising acquiring radio frequency (RF) detection data; determining one or more reflection paths of one or more reflected RF signals based on the RF detection data, where each reflected RF signal comprises a reflection of a transmitted RF signal from one or more objects in physical space; comparing the one or more reflection paths to a field of view (FOV) of an image capture device associated with a mobile device; and triggering an action by at least one of the image capture device and the mobile device based on the comparison.

[0237]

[0246] Aspect 35: The method of aspect 34, further comprising: determining, based on the comparison, that the one or more objects are outside the FOV of the image capture device; and, based on determining that the one or more objects are outside the FOV of the image capture device, setting a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device.

[0238]

[0247] Aspect 36: The method of aspect 35, further comprising determining that one or more objects are moving toward a portion of the scene that corresponds to the FOV of the image capture device, and adjusting a power setting of the image capture device to a different power state based on determining that one or more objects are moving toward a portion of the scene that corresponds to the FOV of the image capture device.

[0239]

[0248] Aspect 37: A method as described in any of aspects 34 to 36, further comprising determining that the image capture device's view of the one or more objects is obstructed by at least one object, and setting a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device based on determining that the image capture device's view of the one or more objects is obstructed by the at least one object, wherein triggering an action is further based on determining that the image capture device's view of the one or more objects is obstructed.

[0240]

[0249] Aspect 38: The method of aspect 37, wherein determining that the image capture device's view of the one or more objects is obstructed by at least one object further comprises: determining, based on the comparison, that the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device; and determining, based on the location of the one or more objects, that the image capture device's view of the one or more objects is obstructed by the at least one object.

[0241]

[0250] Aspect 39: A method as described in any of aspects 34 to 38, wherein the image capture device comprises multiple image sensors, and wherein the triggered action comprises controlling power settings of the image capture device, and wherein controlling the power settings of the image capture device further comprises controlling individual power settings of the multiple image sensors.

[0242]

[0251] Aspect 40: The method of any of aspects 34 to 39, wherein controlling individual power settings of the multiple image sensors further comprises dedicating at least one of the multiple processors of the mobile device to a particular one of the multiple image sensors for image processing based on determining that the hand is within the FOV of the particular one of the multiple image sensors.

[0243]

[0252] Aspect 41: A method as described in any of aspects 34 to 40, further comprising controlling use of multiple image capture devices of the mobile device by multiple processors of the mobile device based on a determination that one or more objects are outside the FOV of the image capture device, the multiple image capture devices including an image capture device.

[0244]

[0253] Aspect 42: The apparatus of aspect 41, wherein to control use of the multiple image capture devices by the multiple processors, one or more processors are configured to dedicate to one of the multiple processors a particular one of the multiple image capture devices that was previously shared by two or more of the multiple processors.

[0245]

[0254] Aspect 43: The apparatus of aspect 41, wherein each of the multiple processors comprises an image signal processor and each of the multiple image capture devices comprises an image sensor.

[0246]

[0255] Aspect 44: A method as described in any of aspects 34 to 43, further comprising: determining that one or more objects are outside the FOV of the image capture device and moving toward a portion of the scene corresponding to the FOV of the image capture device; and in response to determining that one or more objects are outside the FOV of the image capture device and moving toward a portion of the scene corresponding to the FOV of the image capture device, switching the active camera setting from the image capture device to a different image capture device, and the switched active camera setting triggers the mobile device to use the different image capture device to capture one or more images.

[0247]

[0256] Aspect 45: A method as described in any of aspects 34 to 44, wherein at least one of the one or more objects comprises a hand associated with a user of the mobile device.

[0248]

[0257] Aspect 46: The method of any of aspects 34 to 45, further comprising determining a path of the RF signal comprising a direct path of the transmitted RF signal, and determining a location of the one or more objects relative to the mobile device based on the path of the RF signal.

[0249]

[0258] Aspect 47: A method as described in any of aspects 34 to 46, further comprising determining at least one of a respective distance, a respective azimuth angle, and a respective elevation angle associated with one or more reflected paths, and determining a location of the one or more objects relative to the mobile device based on at least one of the respective distances, the respective azimuth angles, and the respective elevation angles.

[0250]

[0259] Aspect 48: A method as described in any of aspects 34 to 47, wherein the triggered action comprises controlling a power setting of an image capture device, wherein controlling the power setting of the image capture device is further based on the light source level falling below a threshold.

[0251]

[0260] Aspect 49: A method as described in any of aspects 34 to 47, wherein the triggered action comprises controlling a power setting of the image capture device, wherein controlling the power setting of the image capture device is further based on a privacy setting, wherein the privacy setting is based on at least one of user input, application data, and global navigation satellite system (GNSS) data.

[0252]

[0261] Example 50: A method according to any of examples 34 to 49, further comprising determining at least one of a size and a shape of the one or more objects based on the RF sensing data and the one or more reflection paths.

[0253]

[0262] Aspect 51: A method as described in any of aspects 34 to 50, further comprising determining a shape of an object from one or more objects based on the RF sensing data and one or more reflected paths, determining that the object comprises a hand associated with a user of the mobile device based on the shape of the object, and generating a cropped image of the hand using an image captured by the image capture device.

[0254]

[0263] Aspect 52: The method of aspect 51, further comprising determining a location of one or more objects relative to the mobile device based on one or more reflected paths, wherein a cropped image is generated based on the location of the one or more objects, wherein the location of the one or more objects comprises a hand location.

[0255]

[0264] Aspect 53: The method of aspect 51, further comprising selecting at least one of the one or more reflection paths from the one or more reflection paths based on the respective distances of the associated objects being within a distance threshold.

[0256]

[0265] Example 54: The method of example 51, further comprising sending the cropped image to a destination device.

[0257]

[0266] Aspect 55: The method of aspect 54, wherein the destination device comprises at least one of a server and a mobile device, wherein the mobile device comprises an extended reality device.

[0258]

[0267] Aspect 56: A method as described in any of aspects 34 to 55, wherein at least one of the one or more objects comprises a hand of a user of the mobile device, and the method further comprises determining at least one of a map of the physical space and a hand gesture associated with the user's hand based on the RF sensing data and the one or more reflected paths.

[0259]

[0268] Example 57: A method according to any of examples 34 to 55, wherein the triggered action comprises extracting a portion of the image containing one or more objects.

[0260]

[0269] Aspect 58: A method as described in any of aspects 34 to 57, wherein the triggered action comprises determining whether to capture one or more images of the one or more objects, the triggered action being further based on a determination that the one or more objects are within a portion of the scene corresponding to the FOV of the image capture device.

[0261]

[0270] Aspect 59: A method as described in any of aspects 34 to 58, further comprising using a machine learning algorithm to detect at least one of a hand associated with a user of the mobile device and an obstruction of a view of the image capture device relative to the hand, wherein the hand comprises at least one of one or more objects.

[0262]

[0271] Aspect 60: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method as described in any of aspects 34 to 58.

[0263]

[0272] Embodiment 61: An apparatus comprising means for carrying out the method according to any of embodiments 34 to 58.

[0264]

[0273] Aspect 62: The apparatus of aspect 61, wherein the apparatus comprises an augmented reality device.

[0265]

[0274] Aspect 63: The apparatus described in aspect 62, wherein the augmented reality device comprises a head-mounted display.

[0266]

[0275] Aspect 64: The apparatus of aspect 61, wherein the apparatus comprises a mobile device.

[0267]

[0276] Example 65: The apparatus of example 61, wherein the apparatus comprises a wearable device.

Claims

1. At least one memory; one or more processors coupled to the at least one memory; an apparatus comprising: acquiring radio frequency (RF) sensing data; determining one or more reflection paths of one or more reflected RF signals based on the RF sensing data, where each reflected RF signal comprises a reflection of the transmitted RF signal from one or more objects in a physical space; comparing the one or more reflected paths with a field of view (FOV) of an image capture device associated with the apparatus; triggering an action by at least one of the apparatus and the image capture device based on the comparison; and An apparatus configured to:

2. The one or more processors: determining, based on the comparison, that the one or more objects are outside the FOV of the image capture device; based on determining that the one or more objects are outside the FOV of the image capture device, setting a power setting of the image capture device to an off state or an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of a scene corresponding to the FOV of the image capture device; The apparatus of claim 1 configured to:

3. The one or more processors: determining, based on the comparison, that the one or more objects are moving toward the FOV of the image capture device; adjusting the power setting of the image capture device to the different power state based on determining that the one or more objects are moving toward the FOV of the image capture device; The apparatus of claim 2 configured to:

4. The one or more processors: determining that a view of the image capture device of the one or more objects is obstructed by at least one further object; setting a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of a scene corresponding to the FOV of the image capture device based on determining that the view of the image capture device of the one or more objects is obstructed by at least one object, wherein triggering the action is further based on the determining that the view of the image capture device of the one or more objects is obstructed. The apparatus of claim 1 configured to:

5. To determine that the view of the image capture device of the one or more objects is obstructed by at least one object, the one or more processors: determining, based on the comparison, that the one or more objects are within the portion of the scene that corresponds to the FOV of the image capture device; The apparatus of claim 4 , further configured to:

6. The one or more processors: determining, based on the comparison, that the one or more objects are outside the FOV of the image capture device and are moving towards a portion of the scene corresponding to the FOV of the image capture device; in response to determining that the one or more objects are outside the FOV of the image capture device and moving toward the portion of the scene corresponding to the FOV of the image capture device, switching an active camera setting from the image capture device to a different image capture device, the switched active camera setting triggering the apparatus to use the different image capture device to capture one or more images. The apparatus of claim 1 configured to:

7. The device of claim 1 , wherein at least one of the one or more objects comprises a hand associated with a user of the device.

8. the one or more processors: determining an RF signal path comprising a direct path of the transmitted RF signal; determining a location of the one or more objects relative to the device based on the path of the RF signal; The apparatus of claim 1 configured to:

9. acquiring radio frequency (RF) sensing data; determining one or more reflection paths of one or more reflected RF signals based on the RF sensing data, where each reflected RF signal comprises a reflection of the transmitted RF signal from one or more objects in a physical space; comparing the one or more reflected paths with a field of view (FOV) of an image capture device associated with the mobile device; triggering an action by at least one of the image capture device and the mobile device based on the comparison; and A method comprising:

10. determining that a view of the image capture device of the one or more objects is obstructed by at least one further object; setting a power setting of the image capture device to an adjusted power state that is lower than a different power state associated with the image capture device when the one or more objects are within a portion of a scene corresponding to the FOV of the image capture device based on determining that the view of the image capture device of the one or more objects is obstructed or blocked by at least one object, wherein triggering the action is further based on the determining that the view of the image capture device of the one or more objects is obstructed. The method of claim 9 further comprising:

11. Determining that the view of the image capture device of the one or more objects is obstructed by at least one object includes: determining, based on the comparison, that the one or more objects are within the portion of the scene that corresponds to the FOV of the image capture device; The method of claim 10 further comprising:

12. determining that the one or more objects are outside the FOV of the image capture device and are moving towards a portion of the scene corresponding to the FOV of the image capture device; in response to determining that the one or more objects are outside the FOV of the image capture device and moving toward the portion of the scene corresponding to the FOV of the image capture device, switching an active camera setting from the image capture device to a different image capture device, the switched active camera setting triggering the mobile device to use the different image capture device to capture one or more images. The method of claim 9 further comprising:

13. The method of claim 9 , wherein at least one of the one or more objects comprises a hand associated with a user of the mobile device.

14. determining an RF signal path comprising a direct path of the transmitted RF signal; determining a location of the one or more objects relative to the mobile device based on the path of the RF signal; The method of claim 9 further comprising:

15. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: acquiring radio frequency (RF) sensing data; determining one or more reflection paths of one or more reflected RF signals based on the RF sensing data, where each reflected RF signal comprises a reflection of the transmitted RF signal from one or more objects in a physical space; comparing the one or more reflected paths with a field of view (FOV) of an image capture device; triggering an action by at least one of the image capture device and an electronic device associated with the image capture device based on the comparison; and A non-transitory computer-readable medium for causing

Images