Adjusting wireless communications based on network strength

Abstract

A head-mounted device may exchange wireless communications with external electronic equipment. Weak network strength of wireless communications may cause a delay, which results in a sub-optimal user experience. To improve operation of a head-mounted device, the head-mounted device may predict a change in network strength associated with wireless communications. The head-mounted device may predict changes in network strength based on historical network strength data and / or scene understanding data of the physical environment. In response to predicting a change in network strength, the head-mounted device may change a characteristic of the wireless communication. The head-mounted device may vary forward error correction applied to the wireless communication, vary a bit rate of the wireless communication, and / or vary the number of retries during the wireless communication.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 432,718, filed February 5, 2024, and U.S. Provisional Patent Application No. 63 / 493,445, filed March 31, 2023, which are hereby incorporated by reference in their entirety. BACKGROUND

[0002] The present disclosure relates generally to electronic devices, and more particularly to electronic devices with displays.

[0003] Some electronic devices, such as head-mounted devices, include displays that are positioned close to a user’s eyes during operation (sometimes referred to as near-eye displays). The displays can present three-dimensional content to the user. Delays, if not careful, can cause artifacts and / or discomfort for a user viewing images on a head-mounted device. SUMMARY

[0004] A method of operating an electronic device can include exchanging wireless communications with an external electronic device, predicting a change in network strength associated with the wireless communications, and changing a characteristic of the wireless communications in response to predicting the change in network strength.

[0005] A method of operating an electronic device configured to wirelessly communicate with a head-mounted device can include rendering display data and wirelessly transmitting the display data to the head-mounted device, predicting a change in network strength associated with a wireless connection between the electronic device and the head-mounted device, and changing a characteristic of rendering the display data and wirelessly transmitting the display data to the head-mounted device in response to predicting the change in network strength.

[0006] A method of operating an electronic device can include exchanging wireless communications with a head-mounted device in response to a prediction of a change in network strength associated with the wireless communications, thereby changing a characteristic of the wireless communications. Exchanging the wireless communications can include wirelessly transmitting rendered display data to the head-mounted device. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a schematic diagram of an illustrative system with an electronic device and external electronic equipment, in accordance with some embodiments.

[0008] Figure 2 is a flow diagram illustrating exemplary method steps performed by a head-mounted device and a paired electronic device in a remote rendering arrangement, in accordance with some embodiments.

[0009] Figure 3 is a top-down view of an illustrative extended reality environment including a physical object and a head-mounted device, in accordance with some embodiments.

[0010] Figure 4is a flow diagram of an exemplary method performed by a head-mounted device that changes a characteristic of wireless communication in response to predicting a change in network strength, in accordance with some embodiments.

[0011] Figure 5 is a flow diagram of an exemplary method performed by an electronic device paired with a head-mounted device, in accordance with some embodiments. DETAILED DESCRIPTION

[0012] An exemplary system with an electronic device is shown in Figure 1 An exemplary system with an electronic device is shown in

[0013] The electronic device 10 can be in wireless communication with the external electronic equipment 30. The external electronic equipment 30 can be a computing device such as a laptop computer, a computer monitor including an embedded computer, a tablet computer, a cellular telephone, a media player or other hand-held or portable electronic device, a smaller device such as a wrist-watch device, a pendant device, a headphone or earpiece device, a device embedded in eyeglasses or other equipment worn on a user's head, or other wearable or miniature device, a display, a computer display including an embedded computer, a computer display not including an embedded computer, a gaming device, a navigation device, an embedded system such as a system having electronic equipment installed in a kiosk or automobile in which a display is included, or other electronic equipment. The external electronic equipment 30 can have the shape of a pair of eyeglasses (e.g., a supporting frame), can form a housing having a helmet shape, or can have other configurations for helping to mount and secure components of one or more displays on a user's head or near a user's eyes.

[0014] External electronic device 30 may include one or more servers. These servers may be implemented on one or more stand-alone data processing devices or a distributed network. The servers may provide information such as web page content to electronic device 10 in response to requests from electronic device 10 (e.g., via a network). The network through which electronic device 10 communicates with external electronic device 30 may include a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet).

[0015] like Figure 1 As shown, electronic device 10 (sometimes referred to as head-mounted device 10, system 10, head-mounted display 10, etc.) may have control circuitry 14. Control circuitry 14 may be configured to perform operations within electronic device 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code and other data used to perform operations within electronic device 10 are stored on a non-transitory computer-readable storage medium (e.g., a tangible computer-readable storage medium) within control circuitry 14. Software code may sometimes be referred to as software, data, program instructions, commands, or code. Non-transitory computer-readable storage medium (sometimes commonly referred to as memory) may include non-volatile memory such as non-volatile random access memory (NVRAM), one or more hard disk drives (e.g., disk drives or solid-state drives), one or more removable flash drives, or other removable media, etc. Software stored on non-transitory computer-readable storage medium may be executed on processing circuitry of control circuitry 14. Processing circuitry may include application-specific integrated circuits (ASICs) with processing circuitry, one or more microprocessors, digital signal processors, graphics processing units, central processing units (CPUs), or other processing circuitry.

[0016] Electronic device 10 may include input-output circuitry 20. Input-output circuitry 20 can be used to allow a user to provide user input to electronic device 10 and / or collect information about the environment in which electronic device 10 is operating. Output components in circuitry 20 can allow electronic device 10 to provide output to the user.

[0017] like Figure 1As shown, input-output circuitry 20 can include a display, such as display 16. Display 16 can be used to display images for a user of electronic device 10. Display 16 can be a transparent display, such that a user can view physical objects through the display while overlaying computer-generated content on the physical objects by presenting computer-generated images on the display. A transparent display can be formed from an array of transparent pixels (e.g., a transparent organic light-emitting diode display panel), or can be formed from a display device that provides images to a user through a beam splitter, holographic coupler, or other optical coupler (e.g., a display device such as a liquid crystal on silicon display). Alternatively, display 16 can be an opaque display that blocks light from physical objects when a user is operating electronic device 10. In this type of arrangement, a see-through camera can be used to display physical objects to the user. The see-through camera can capture images of the physical environment, and the physical environment images can be displayed on the display for the user to view. Additional computer-generated content (e.g., text, game content, other visual content, etc.) can optionally be overlaid on the physical environment images to provide an augmented reality environment for the user. When display 16 is opaque, the display can also optionally display entirely computer-generated content (e.g., without displaying images of the physical environment).

[0018] Display 16 can include one or more optical systems (e.g., lenses) (sometimes referred to as optical components) that allow a viewer to view images on display 16. A single display 16 can produce images for both eyes, or a pair of displays 16 can be used to display images. In configurations with multiple displays (e.g., a left-eye display and a right-eye display), the focal length and positioning of the lenses can be selected such that any gap that exists between the displays will be invisible to the user (e.g., so that the images of the left and right displays seamlessly overlap or merge). A display module (sometimes referred to as a display component) that generates different images for the left and right eyes of a user can be referred to as a stereoscopic display. A stereoscopic display can be capable of presenting two-dimensional content (e.g., a user notification with text) and three-dimensional content (e.g., a simulation of a physical object such as a cube).

[0019] Input-output circuitry 20 can include various other input-output devices. For example, input-output circuitry 20 can include one or more cameras 18. Camera 18 can include one or more outward-facing cameras (as one example, when electronic device is mounted on a user's head, the one or more outward-facing cameras face the physical environment around the user). Camera 18 can capture visible light images, infrared images, or any other desired type of image. If desired, the camera can be a stereoscopic camera. The outward-facing camera(s) can capture see-through video for device 10. Camera 18 can include one or more inward-facing cameras (e.g., that obtain gaze detection information).

[0020] like Figure 1 As shown, the input-output circuitry 20 may include a positioning and motion sensor 22 (e.g., a compass, gyroscope, accelerometer, and / or other devices for monitoring the position, orientation, and movement of the electronic device 10, satellite navigation system circuitry such as a Global Positioning System circuit for monitoring the user's position, etc.). For example, the control circuitry 14 may use the sensor 22 to monitor the current orientation of the user's head relative to the surrounding environment (e.g., the user's head posture). A camera 18 may also be considered part of the positioning and motion sensor 22. The camera may be used for face tracking (e.g., by capturing images of the user's chin, mouth, etc., when the device is worn on the user's head), body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc., when the device is worn on the user's head), and / or for positioning (e.g., using visual ranging, visual-inertial ranging, or other simultaneous localization and mapping (SLAM) techniques).

[0021] Input-output circuitry 20 may include one or more depth sensors 24. Each depth sensor may be a pixelated depth sensor (e.g., configured to measure multiple depths across a physical environment) or a point sensor (configured to measure a single depth within a physical environment). Each depth sensor (whether pixelated or point-based) may use phase detection (e.g., phase detection autofocus pixels) or light detection and ranging (LIDAR) to measure depth. Any combination of depth sensors may be used to determine the depth of a physical object within the physical environment.

[0022] If needed, the input-output circuit 20 may also include other sensors and input-output components (e.g., gaze tracking sensors, ambient light sensors, force sensors, temperature sensors, touch sensors, image sensors for detecting hand gestures or body postures, buttons, capacitive proximity sensors, light-based proximity sensors, other proximity sensors, strain gauges, gas sensors, pressure sensors, humidity sensors, magnetic sensors, microphones, speakers, audio components, tactile output devices such as actuators, light-emitting diodes, other light sources, etc.).

[0023] The head-mounted device 10 may also include communication circuitry 26 to allow the head-mounted device to communicate with external devices such as electronic devices 30 (e.g., tethered computers, portable devices such as handheld devices, watches, or laptops, or other electrical equipment). Communication circuitry 26 can be used for both wired and wireless communication with external devices.

[0024] The communication circuitry 26 can include radio-frequency (RF) transceiver circuitry formed from one or more integrated circuits, power amplifier circuitry, low-noise input amplifiers, passive RF components, one or more antennas, transmission lines, and other circuitry for handling RF wireless signals. Wireless signals can also be sent using light (e.g., using infrared communication).

[0025] The radio-frequency transceiver circuitry in the wireless communication circuitry 26 can handle wireless local area network (WLAN) communication bands such as 2.4 GHz and 5 GHz (IEEE 802.11) bands; wireless personal area network (WPAN) communication bands such as 2.4 GHz communication bands (e.g., 3G bands, 4G LTE bands, 5G New Radio Frequency Range 1 (FR1) bands below 10 GHz, etc.); near-field communication (NFC) bands (e.g., 13.56 MHz); satellite navigation bands (e.g., L1 Global Positioning System (GPS) band of 1575 MHz, L5 GPS band of 1176 MHz, GLONASS bands, BeiDou Navigation Satellite System (BDS) bands, etc.); ultra-wideband (UWB) communication bands supported by IEEE 802.15.4 protocols and / or other UWB communication protocols (e.g., a first UWB communication band of 6.5 GHz and / or a second UWB communication band of 8.0 GHz); and / or any other desired communication bands.

[0026] The radio-frequency transceiver circuitry can include millimeter wave / centimeter wave transceiver circuitry supporting communications at frequencies between about 10 GHz and 300 GHz. For example, the millimeter wave / centimeter wave transceiver circuitry can support communications in extremely high frequency (EHF) or millimeter wave communication bands between about 30 GHz and 300 GHz and / or in centimeter wave communication bands (sometimes referred to as super high frequency (SHF) bands) between about 10 GHz and 30 GHz. For example, the millimeter wave / centimeter wave transceiver circuitry can support communications in the following communication bands: an IEEE K communication band between about 18 GHz and 27 GHz, a K aCommunication Band, K-band, between about 12 GHz and 18 GHz u Communication Band, V-band, between about 40 GHz and 75 GHz, W-band, between about 75 GHz and 110 GHz, or any other desired band of frequencies between about 10 GHz and 300 GHz. If desired, the millimeter / cenimeter wave transceiver circuitry can support IEEE 802.11 ad communications at 60 GHz (e.g., WiGig or 60 GHz Wi-Fi bands of about 57 GHz to 61 GHz) and / or 5thGeneration mobile networks or 5thGeneration wireless systems (5G) New Radio (NR) Frequency Range 2 (FR2) communication bands between about 24 GHz and 90 GHz.

[0027] The antennas in the wireless communication circuitry 26 can include antennas with resonant elements formed from loop antenna structures, patch antenna structures, inverted-F antenna structures, slot antenna structures, planar inverted-F antenna structures, spiral antenna structures, dipole antenna structures, monopole antenna structures, combinations of these designs, and the like. Different types of antennas can be used for different bands and combinations of bands. For example, one type of antenna can be used in forming the local wireless link and another type of antenna can be used in forming the remote wireless link antenna.

[0028] One application example to be described herein is an example in which the electronic device 10 is a head-mounted device and the external electronic equipment 30 is a paired non-head-mounted device (e.g., a cellular phone, a laptop computer, a tablet computer, a watch, etc.). The external electronic equipment 30 can sometimes be referred to as the electronic device 30 or the paired electronic device 30.

[0029] The electronic device 10 can be paired with the electronic device 30. In other words, a wireless link can be established between the electronic devices 10 and 30 to allow for fast and efficient communication between the devices 10 and 30. The electronic devices 10 and 30 can be associated with the same user (e.g., logged into a cloud service using the same user ID), can exchange wireless communications, and the like. Each of the electronic devices 10 and 30 can include a battery. In one embodiment, the head-mounted device 10 can have a battery with a lower total capacity than the electronic device 30.

[0030] If desired, content for display 16 on head-mounted device 10 can be rendered by electronic device 30 and wirelessly transmitted from electronic device 30 to head-mounted device 10 for subsequent display. Rendering display content for a first electronic device using a second electronic device can sometimes be referred to herein as remote rendering. Remote rendering can be used to reduce power consumption on one electronic device. For example, it can be desirable to reduce power consumption of head-mounted device 10. Remote rendering shifts some of the processing burden (and thus power consumption) for operating display 16 on electronic device 10 to electronic device 30.

[0031] In some cases, a graphics processing unit (GPU) in electronic device 30 can be capable of more complex rendering operations than a graphics processing unit in electronic device 10 (e.g., in control circuit 14). In such cases, remote rendering can allow for rendering of more complex display data than if head-mounted device 10 used only its local GPU.

[0032] Figure 2 is a flowchart of illustrative method steps performed by a head-mounted device and a paired electronic device in a remote rendering arrangement. As shown, the steps on the left (e.g., steps 102, 112, 116, 118, and 120) are performed by a first electronic device such as head-mounted device 10 in Figure 1 The steps on the right (e.g., steps 104, 106, 108, 110, and 114) are performed by a second electronic device such as electronic device 30 in Figure 1 The two devices in can be paired and can exchange wireless communications. Figure 2

[0033] At step 102, head-mounted device 10 can transmit head pose information to paired electronic device 30. As one example, head-mounted device 10 can use positioning and motion sensors 22 to obtain head pose information. Head-mounted device 10 can wirelessly transmit the head pose information (e.g., using Bluetooth communications). Head-mounted device 10 can transmit a single head pose for a single point in time, or multiple head poses associated with different points in time. In other words, head-mounted device 10 can transmit one or more head poses, where each head pose has a corresponding timestamp. In general, the transmitted head pose information can include any desired additional sensor information, contextual information, historical data, etc.

[0034] ​At step 104, the electronic device 30 can receive head pose information from the paired head mounted device 10. At step 106, the electronic device 30 then uses the received head pose information to estimate the head pose for a given display frame. The electronic device 30 can use historical head pose information for the head mounted device 10 stored at the electronic device 30 to estimate the head pose for a given display frame. For example, the head pose information from the head mounted device 104 can identify a first head pose at a first time (ti), a second head pose at a second time (t2), and a third head pose at a third time (t3). The estimated time for displaying a given display frame is t4. The electronic device uses the head poses over time at ti, t2, and t3 to predict the head pose at t4. The electronic device 30 can use any desired number of head poses (e.g., one or more) to predict the head pose at t4.

[0035] At step 108, the electronic device 30 can render display data for a given display frame using the estimated head pose from step 106. The electronic device 30 can use a graphics processing unit or any other desired computing resource to render the display data. The rendered display data can then be compressed at step 110. The compressed display data is then wirelessly transmitted to the paired head mounted device at step 114.

[0036] At step 112, the head mounted device 10 can receive the rendered display data from the paired electronic device. The received rendered display data can have an associated target head pose (e.g., the fourth head pose for time t4 using the example described above). At step 116, the head mounted device 10 can decompress the rendered display data. Generally, any desired encoding / decoding scheme can be used for the compression and decompression of the display data.

[0037] At step 118, the head mounted device can adjust the display data based on updated head pose information. As previously mentioned, the rendered display data can be rendered by the electronic device 30 for a predicted head pose. The head mounted device 10 can modify the head pose estimate for a given display frame based on head pose data that has been obtained between step 102 (when the head pose information was transmitted to device 30) and step 118.

[0038] In the previous example, the electronic device 30 renders display data for a given display frame at an estimated fourth head pose at the target time t4. At step 118, the head-mounted device 10 can estimate a fifth head pose at the target time t4 using updated head pose information collected with the positioning and motion sensors 22. At step 118, the head-mounted device 10 thus adjusts the display data to compensate for the difference between the originally predicted head pose (e.g., the fourth head pose) and the newly predicted head pose (e.g., the fifth head pose). Adjusting the display data to compensate for the difference between the originally predicted head pose and the newly predicted head pose can include re-projecting the display data to the newly predicted head pose.

[0039] Finally, at step 120, the head-mounted device 10 can display the display data for the given frame using the display 16. If desired, steps 118 and 120 (e.g., adjusting the display data based on the head pose information and displaying the display data) can be performed at a higher frequency than step 108 (e.g., rendering the display data). For example, steps 118 and 120 can be performed at twice the frequency of step 108. As one example, steps 118 and 120 can be performed at a frequency of 120 Hz, while step 108 can be performed at a frequency of 60 Hz.

[0040] Using the remote rendering operation of Figure 2 , the head-mounted device 10 saves power relative to an arrangement in which all display data is rendered locally. However, the remote rendering operation relies on wireless communication between the head-mounted device 10 and the electronic device 30. Thus, an unreliable network strength between the head-mounted device 10 and the electronic device 30 can cause undesirable latency in the content displayed to the user of the head-mounted device.

[0041] In the example of Figure 2 , the head-mounted device 10 relies on wireless communication for the remote rendering arrangement with the external electronic device 30. However, this example is merely illustrative. In general, the head-mounted device 10 can use wireless communication in many applications (e.g., video conferencing, online gaming, entertainment streaming, etc.). For all of these wireless communications, a weak network strength for the wireless communication can cause latency, which results in a suboptimal user experience.

[0042] To improve operation of the head-worn device 10, the head-worn device can predict changes in network strength associated with wireless communications. In response to predicting a change in network strength, the head-worn device can change a characteristic of the wireless communications. For example, consider a scenario in which a user is engaged in a video conference call. If the head-worn device predicts a decrease in network strength for the wireless communications for the video conference call, the head-worn device can take a proactive mitigation action, such as changing forward error correction applied to the wireless communications, changing a bit rate for the wireless communications, changing a number of retries during the wireless communications, and so on.

[0043] The head-worn device 10 can predict changes in network strength based on historical network strength data and / or scene understanding data for the physical environment.

[0044] Figure 3 is a top view of an illustrative three-dimensional environment that includes the electronic device 10. In Figure 3 In the depicted example, the three-dimensional environment 36 is an extended reality (XR) environment that includes a physical environment (with various physical objects) surrounding the electronic device 10. The three-dimensional environment 36 can optionally include one or more virtual objects. The XR environment 36 can include a plurality of physical walls 32 that define several rooms (e.g., rooms in a house or apartment). In the depicted example XR environment, there are five rooms. The physical environment surrounding the electronic device 10 can include physical objects, such as physical objects 34-1, 34-2, and 34-3. Figure 3 The physical objects depicted in the example can be any desired type of physical object (e.g., a table, a bed, a sofa, a chair, a refrigerator, etc.). Figure 3

[0045] During operation of the electronic device 10, the electronic device 10 can move throughout the three-dimensional environment 36. In other words, a user of the electronic device 10 can repeatedly carry the electronic device 10 between different rooms in the three-dimensional environment 36. While operating in the three-dimensional environment 36, the electronic device 10 can use one or more sensors (e.g., the camera 18, the position and motion sensor 22, the depth sensor 24, etc.) to collect sensor data about the three-dimensional environment 36. The electronic device 10 can construct a scene understanding dataset for the three-dimensional environment.

[0046] To construct the scene understanding dataset, the electronic device can use input from sensors such as the camera 18, the position and motion sensor 22, and the depth sensor 24. As one example, data from the depth sensor 24 and / or the position and motion sensor 22 can be used to construct a spatial mesh that represents the physical environment. The spatial mesh can include a polygonal model of the physical environment and / or a series of vertices that represent the physical environment. The spatial mesh (sometimes referred to as spatial data, etc.) can define dimensions, locations, and orientations of planes within the physical environment. The spatial mesh represents the physical environment surrounding the electronic device. ​

[0047] Other data, such as data from the camera 18, can be used to construct the scene understanding dataset. For example, the camera 18 can capture images of the physical environment. The electronic device can analyze the images to identify characteristics of the planes in the spatial grid (e.g., the color of the plane or the material forming the plane). This characteristic can be included in the scene understanding dataset.

[0048] The scene understanding dataset can include the identities of various physical objects in the extended reality environment. For example, the electronic device 10 can analyze images from the camera 18 and / or the depth sensor 24 to identify physical objects. The electronic device 10 can identify physical objects such as a bed, a sofa, a chair, a table, a refrigerator, etc. This information identifying the physical objects can be included in the scene understanding dataset.

[0049] The scene understanding dataset can also include information about various virtual objects in the extended reality environment. The electronic device 10 can be used to display virtual objects, and thus knows the identity, size, shape, color, etc. of the virtual objects in the extended reality environment. This information about the virtual objects can be included in the scene understanding dataset.

[0050] The scene understanding dataset can be constructed over time on the electronic device 10 as the electronic device moves throughout the extended reality environment. For example, consider an example in which the electronic device 10 starts in a room that includes the physical objects 34-1 and 34-2. Figure 3 The electronic device can use the depth sensor to obtain depth information (and develop a spatial grid) of the currently occupied room (with objects 34-1 / 34-2). The electronic device can develop a scene understanding dataset for the currently occupied room (including the spatial grid, physical object information, virtual object information, etc.). At this point, the electronic device does not have information for the unoccupied room.

[0051] Next, the electronic device 10 can be transported into a room that has the physical object 34-3. While in this new room, the electronic device can use the depth sensor to obtain depth information (and develop a spatial grid) of the currently occupied room (with object 34-3). The electronic device can develop a scene understanding dataset for the currently occupied room (including the spatial grid, physical object information, virtual object information, etc.). The electronic device now has a scene understanding dataset that includes data about both the currently occupied room (with object 34-3) and the previously occupied room (with objects 34-1 and 34-2). In other words, as the electronic device enters a new portion of the three-dimensional environment, data can be added to the scene understanding dataset. Thus, over time (as the electronic device is transported into each room in the three-dimensional environment), the scene understanding dataset includes data about the entire three-dimensional environment 36.

[0052] The electronic device 10 can maintain a scene understanding dataset that includes all of the scene understanding data associated with the extended reality environment 36 (e.g., including both the room that is currently occupied and the room that is currently unoccupied).

[0053] In summary, the head-mounted device 10 can maintain a scene understanding dataset that includes a spatial grid representing the physical environment, characteristics of the planes in the spatial grid, identities of various physical objects in the extended reality environment, and / or information about various virtual objects in the extended reality environment. In addition to the scene understanding dataset, the head-mounted device can also maintain historical network strength information. The historical network strength information can be considered to be part of the scene understanding dataset, or can be considered to be separate from the scene understanding dataset.

[0054] The historical network strength information can be accumulated over time during operation of the head-mounted device. The network strength can be characterized in any desired manner. For example, the head-mounted device can characterize the network strength associated with wireless communication using a measured bit rate of outgoing transmissions, a measured bit rate of incoming transmissions, and / or an expected bit rate associated with the type of wireless communication being used. As an additional example, the network strength can be measured by the head-mounted device 10 using milliwatts (mW), decibels / milliwatts (dBm), or a received signal strength indicator (RSSI). Any subset of the above factors can be used to characterize the network strength.

[0055] Figure 3 Three different locations in the three-dimensional environment are identified: location A, location B, and location C. As the head-mounted device 10 spends time at each of these locations, the network strength associated with these locations can be stored on the head-mounted device 10. In addition to being associated with a location, the stored network strength information can also have an associated timestamp (to identify trends in network strength over the time of day), a network type (e.g., Wi-Fi vs. cellular), and so on.

[0056] In one example, the historical network strength information stored on the electronic device 10 can all be obtained using the head-mounted device 10. In another possible example, at least some of the historical network strength information can be obtained by one or more external electronic devices. Multiple devices can contribute to a common network strength database that is then accessible by the head-mounted device 10. This can enable the head-mounted device 10 to use network strength information for locations that it has not yet traveled to.

[0057] Consider a scenario in which the historical network strength information indicates a first network strength at location A, a second network strength at location B, and a third network strength at location C. The second network strength can be weaker than the first network strength. The third network strength can be weaker than the second network strength. The head-mounted device can detect that the user is traveling from location A toward location B. In this scenario, the head-mounted device can predict that the network strength will decrease (e.g., once the user reaches location B, which has a weaker network strength than location A). In an alternative scenario, the head-mounted device can detect that the user is traveling from location C toward location B. In this scenario, the head-mounted device can predict that the network strength will increase (e.g., once the user reaches location B, which has a stronger network strength than location C).

[0058] In examples in which the historical network strength information stored on the electronic device 10 was all obtained using the head-mounted device 10, the head-mounted device 10 needs to travel to each of locations A, B, and C to obtain the network strength information before network strength predictions associated with those locations can be made. In examples in which at least some of the historical network strength information can be obtained by one or more external electronic devices, the head-mounted device 10 can use network strength information obtained with one or more external electronic devices to make network strength predictions associated with locations A, B, and C without needing to travel to locations A, B, and C.

[0059] In response to a predicted change in network strength of a wireless communication, the head-mounted device can make various adjustments to the wireless communication. The adjustments can include changing forward error correction applied to the wireless communication, changing a bit rate of the wireless communication, changing a number of retries during the wireless communication, or other desired adjustments.

[0060] Examples in which the head-mounted device 10 uses historical network strength information (which is associated with locations, times of day, etc.) to predict network strength have been described herein. This example is merely illustrative. Alternatively or additionally, network strength predictions can be made using a scene understanding dataset obtained by the head-mounted device 10.

[0061] A scene understanding dataset can include a spatial grid of a room currently occupied by the head-mounted device. The spatial grid can be used to determine dimensions of the room that include the head-mounted device. The dimensions of the room can be used to make predictions about network strength.

[0062] As another example, a scene understanding dataset can include information about materials in the physical environment. For example, the scene understanding dataset can identify materials (e.g., metal or concrete) that are associated with poor network strength. Accordingly, the head-mounted device can use the materials identified in the scene understanding data to predict changes in network strength.

[0063] As another example, the scene understanding dataset can include information about physical objects in the physical environment. For example, the scene understanding dataset can identify physical objects that are associated with poor network strength. Accordingly, the head-mounted device can use the physical objects identified in the scene understanding data to predict changes in network strength.

[0064] Consider an example in which a user wearing the head-mounted device 10 enters a parking lot. Scene understanding data obtained by the head-mounted device 10 can identify a concrete wall around the head-mounted device. The presence of the concrete wall can be associated with poor network strength. Accordingly, the head-mounted device 10 can predict a decrease in network strength as the user enters the parking lot.

[0065] As another example, scene understanding data obtained by the head-mounted device 10 can identify that the user is in a parking lot. Being in a parking lot can be associated with poor network strength. Accordingly, the head-mounted device 10 can predict a decrease in network strength in response to detecting that the user is in the parking lot.

[0066] Figure 4 is a flowchart of illustrative method steps for operating a head-mounted device. As shown, at step 202, the head-mounted device 10 can exchange wireless communications with an external electronic equipment 30. The external electronic equipment can be a paired electronic device (e.g., a cellular phone, a laptop computer, a tablet computer, a watch, etc.), an external server, or other external electronic equipment. The wireless communications can include cellular communications, Bluetooth communications, Wi-Fi communications, or any other desired type of communications.

[0067] At step 204, the head-mounted device 10 can predict a change in network strength associated with the wireless communications. The head-mounted device 10 can use location data (e.g., from the positioning and motion sensors 22), historical network strength data (obtained using the head-mounted device 10 and / or one or more external electronic devices), and / or scene understanding data to predict the change in network strength.

[0068] For example, the head-mounted device 10 can predict a change in location, and based on historical network strength data for the current location and the predicted new location, predict a change in network strength. The head-mounted device 10 can also predict a change in network strength based on historical network strength data at the current location of the head-mounted device.

[0069] In some cases, interpolation can be used to determine a predicted network strength using historical network strength information. For example, the historical network strength information can include a first network strength for a first location and a second network strength for a second location. When the head-mounted device 10 is at (or predicted to be at) a third location between the first location and the second location, an interpolation between the first network strength and the second network strength can be used to estimate a third network strength for the third location.

[0070] The head-mounted device can use at least the depth sensor 24 to determine a depth map of the physical environment. The head-mounted device can use the depth map to predict changes in network strength. For example, the head-mounted device can use a room layout determined using the depth map to predict changes in network strength. The depth map can be considered part of a scene understanding dataset maintained by the head-mounted device.

[0071] The head-mounted device can use the camera 18, the position and motion sensor 22, and / or the depth sensor 24 to obtain scene understanding data. The head-mounted device can use the scene understanding data to predict changes in network strength. For example, the head-mounted device can use an identity of a physical object included in the scene understanding data to predict changes in network strength. As another example, the head-mounted device can use a material (e.g., metal, concrete, drywall, or another desired material) included in the scene understanding data to predict changes in network strength.

[0072] Next, at step 206, the head-mounted device 10 can change a characteristic of the wireless communication in response to predicting a change in network strength. Examples of adjustments to the wireless communication include changing forward error correction applied to the wireless communication, changing a bit rate of the wireless communication, changing a number of retries during the wireless communication, and the like.

[0073] Consider an example in which the wireless communication is exchanged at step 202 without forward error correction. Then, at step 204, it is predicted that the network strength associated with the wireless communication will decrease. In this case, at step 206, forward error correction can be applied to the wireless communication.

[0074] Consider another example in which the wireless communication is transmitted at step 202 using a first bit rate. Then, at step 204, it is predicted that the network strength associated with the wireless communication will decrease. In this case, at step 206, the bit rate of the transmission can be decreased.

[0075] As discussed in connection with Figure 2The type of wireless communication performed by the head-mounted device 10 with the paired electronic device is in a remote rendering arrangement. In the remote rendering arrangement, the head-mounted device 10 receives rendered display data from the paired electronic device and displays the rendered display data on the display 16. The paired electronic device can optionally render display data for multiple viewpoints (e.g., a first frame of image data for a primary intended viewpoint and one or more secondary frames of image data for secondary viewpoints different from the primary viewpoint). At step 206, the remote rendering communication can be adjusted in response to a predicted change in network strength of the wireless connection between the head-mounted device 10 and the paired electronic device. For example, the head-mounted device 10 can send prediction information and / or instructions regarding the predicted change in network strength to the paired electronic device. In response, the paired electronic device can render display data for a different number of viewpoints with different complexity (e.g., with a different number of polygons, three-dimensional voxel resolution, number of objects in the content, etc.), with different resolution, and / or with different field of view. The paired electronic device can render display data for a higher number of viewpoints (e.g., three discrete frames of image data for three respective viewpoints) in response to a predicted increase in network strength, or a lower number of viewpoints (e.g., one frame of image data for one respective viewpoint) in response to a predicted decrease in network strength. The paired electronic device can render display data with higher complexity in response to a predicted increase in network strength, or with lower complexity in response to a predicted decrease in network strength. The paired electronic device can render display data with higher resolution in response to a predicted increase in network strength, or with lower resolution in response to a predicted decrease in network strength. The paired electronic device can render display data with a larger field of view in response to a predicted increase in network strength, or with a smaller field of view in response to a predicted decrease in network strength. As yet another example, at step 206, the head-mounted device 10 can (in response to a predicted weakening in network strength) cease the remote rendering arrangement and instead render display data locally using a local graphics processing unit (GPU) (e.g., a GPU in the control circuit 14).

[0076] Figure 5 is a flowchart of exemplary method steps performed by an electronic device that remotely renders display data for a paired head-mounted device. At step 302, the electronic device exchanges communications with the head-mounted device. The communications can include receiving head pose information from the head-mounted device and transmitting rendered display data to the head-mounted device.

[0077] At step 304, in response to the prediction of a change in network strength associated with the wireless communication, the electronic device can change a characteristic of the wireless communication. The electronic device 30 can make the prediction of the change in network strength locally using the location information and / or historical network strength information. Alternatively, the electronic device 30 can receive the prediction information from the head-mounted device 10. As yet another alternative, the electronic device 30 can receive instructions from the head-mounted device 10 to adjust the wireless communication with the head-mounted device 10 (without including an explicit prediction in the instructions).

[0078] At step 304, the electronic device can render display data for the head-mounted device 10 at different resolutions and / or with different fields of view. For example, the electronic device 30 can render display data at a higher resolution in response to a predicted increase in network strength, or render display data at a lower resolution in response to a predicted decrease in network strength. The electronic device 30 can render display data with a larger field of view in response to a predicted increase in network strength, or render display data with a smaller field of view in response to a predicted decrease in network strength.

[0079] Alternatively or additionally, at step 304, the electronic device 30 can change forward error correction applied to the wireless communication, change a bit rate of the wireless communication, change a number of retries during the wireless communication, etc.

[0080] Examples are described herein with respect to head pose prediction and rendering display data based on head pose prediction. It should be noted that head pose prediction can depend on various factors. These include the nature of the content being rendered, the user's position relative to their environment, the user's current motion (e.g., walking versus standing or sitting), whether the user is in the vicinity of or interacting with other users, and historical data that has been accumulated both with respect to this particular user and with respect to a more general population of users. With respect to the nature of the content, characteristics such as the complexity of the content, whether the content is head-locked, body-locked, or world-locked, and placement of the content relative to both physical and virtual environments can also play a role in determining how to render additional views or how to adjust content sent over the link.

[0081] To this end, in addition to accelerometer data from the positioning and motion sensors 22, the head pose information transmitted from the head-mounted device 10 to the electronic device 30 (e.g., at step 102 in Figure 2 may include additional information that can impact head pose prediction. The additional information can include information about the user's position relative to their environment, the user's current motion (e.g., walking versus standing or sitting), whether the user is in the vicinity of or interacting with other users, etc.

[0082] As described above, one aspect of the present technology is the collection and use of information, such as sensor information. The present disclosure contemplates that in some instances, data collected including personal information data will be maintained, processed and used on a very limited basis for improving the functionality of the present technology and to confirm to applicable law. For example, personal information data can be used to deliver targeted content to users. Thus, the use of certain personal information data can be beneficial to the user, for instance, by being used to deliver targeted content that is of greater interest to the user based on their personal information data.

[0083] The present disclosure recognizes that the use of such personal information in the present technology can be employed to the benefit of users. For example, the personal information data can be used to deliver targeted content that is of greater interest to the user. Accordingly, use of such personal information data is contemplated to improve the quality of the content being delivered, which can benefit users. Further, other uses for personal information data that benefit the user are also contemplated. For instance, health and fitness data can be used to provide insights into a user's general wellness, or can be used as positive feedback to an individual using the technology to pursue a healthy goal.

[0084] The present disclosure contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining privacy and security, including, for example, principles published by well-established privacy and security organizations, any privacy mechanisms required by an applicable jurisdiction, any privacy mechanisms deemed necessary by the entities' business practices (e.g., consumer review and response mechanisms for handling complaints).

[0085] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware and / or software elements can be provided to prevent or block access to such personal information data. For example, the present technology can be configured to allow users to opt-in or opt-out of participation in the collection of personal information data during registration for services or anytime thereafter. In another example, users can be provided with an option to not provide certain types of personal information data. In yet another example, users can be provided with an option to limit the length of time personal information data is maintained. In addition to providing “opt-in” and “opt-out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user can be notified upon download of an application (an “app”) that their personal information data will be accessed and then reminded again just prior to the app accessing the personal information data.

[0086] Moreover, it is the intent of the present disclosure that personal information data should be managed and processed at a level of granularity that minimizes unintended or unauthorized access or use. Data can be minimized by limiting collection and deleting data that is no longer needed. In addition, and when applicable, data de-identification can be used to protect a user’s privacy. De-identification can be facilitated, when appropriate, by removing specific identifiers (e.g., birth date, etc.), controlling the amount or characteristics of data stored, controlling data storage (e.g., aggregating data across users), and / or other methods.

[0087] Thus, while the present disclosure broadly covers technologies that use information including personal information data to implement various disclosed embodiments, the present disclosure also contemplates embodiments that can be implemented without the need for access to such personal information data. That is, the various embodiments of the present technology can not operate in systems that do not collect or otherwise have access to personal information data.

[0088] According to one embodiment, a method of operating an electronic device is provided, the method comprising: exchanging wireless communications with an external electronic device; predicting a change in network strength associated with the wireless communications; and in response to predicting the change in the network strength, changing a characteristic of the wireless communications.

[0089] According to another embodiment, changing the characteristic of the wireless communications comprises changing a forward error correction applied to the wireless communications.

[0090] According to another embodiment, changing the characteristic of the wireless communications comprises changing a bit rate of the wireless communications.

[0091] According to another implementation, changing the characteristic of the wireless communication includes changing a number of retries during the wireless communication.

[0092] According to another implementation, predicting the change in the network strength includes predicting the change in the network strength based on historical network strength information.

[0093] According to another implementation, the historical network strength information includes historical network strength information at different locations, and predicting the change in the network strength includes predicting a change in location from a first location having a first historical network strength to a second location having a second historical network strength different from the first historical network strength.

[0094] According to another implementation, predicting the change in the network strength based on the historical network strength information includes interpolating between a first historical network strength at a first location and a second historical network strength at a second location based on a third location associated with the electronic device.

[0095] According to another implementation, the historical network strength information includes network strength information obtained by one or more additional electronic devices.

[0096] According to another implementation, the electronic device includes one or more depth sensors, and the method further includes obtaining a depth map of a physical environment using the one or more depth sensors, predicting the change in the network strength includes predicting the change in the network strength based on the depth map.

[0097] According to another implementation, predicting the change in the network strength based on the depth map includes predicting the change in the network strength based on a room layout determined using the depth map.

[0098] According to another implementation, the electronic device includes one or more sensors, and the method further includes obtaining scene understanding data of a physical environment using the one or more sensors, predicting the change in the network strength includes predicting the change in the network strength based on the scene understanding data.

[0099] According to another implementation, predicting the change in the network strength based on the scene understanding data includes predicting the change in the network strength based on an identity of a physical object in the physical environment.

[0100] According to another implementation, predicting the change in the network strength based on the scene understanding data includes predicting the change in the network strength based on a material identified in the scene understanding data.

[0101] According to another implementation, the electronic device is a head mounted device and the external electronic equipment is a cellular telephone.

[0102] According to another implementation, exchanging the wireless communication with the external electronic equipment includes wirelessly transmitting head pose information to the cellular telephone and wirelessly receiving display data from the cellular telephone.

[0103] According to one implementation, a method of operating an electronic device configured to wirelessly communicate with a head mounted device is provided, the method comprising: rendering display data and wirelessly transmitting the display data to the head mounted device; predicting a change in network strength associated with a wireless connection between the electronic device and the head mounted device; and in response to predicting the change in network strength, changing a characteristic of the rendering the display data and wirelessly transmitting the display data to the head mounted device.

[0104] According to another implementation, the characteristic includes a characteristic selected from the group consisting of forward error correction, bit rate, and number of retries.

[0105] According to another implementation, changing the characteristic of the rendering display data and wirelessly transmitting the display data to the head mounted device includes changing a resolution of the display data during the rendering of the display data.

[0106] According to another implementation, changing the characteristic of the rendering the display data and wirelessly transmitting the display data to the head mounted device includes changing a field of view of the display data during the rendering of the display data.

[0107] According to one implementation, a method of operating an electronic device is provided, the method comprising exchanging wireless communications with a head mounted device, exchanging the wireless communications including wirelessly transmitting rendered display data to the head mounted device, and in response to a prediction of a change in network strength associated with the wireless communications, changing a characteristic of the wireless communications.

[0108] The foregoing is merely illustrative and various modifications can be made to the described embodiments. Such embodiments can be implemented individually or in any combination. CLAIM (MODIFIED PURSUANT TO ARTICLE 19 OF THE TREATY) 1. A method of operating an electronic device, the electronic device comprising one or more sensors, the method comprising: exchanging wireless communications with external electronic equipment; obtaining a three-dimensional model of a physical environment using the one or more sensors; predicting, based at least in part on the three-dimensional model of the physical environment, a change in network strength associated with the wireless communication; and in response to predicting the change in network strength, changing a characteristic of the wireless communication. 2. The method of claim 1, wherein changing the characteristic of the wireless communication comprises changing forward error correction applied to the wireless communication. 3. The method of claim 1, wherein changing the characteristic of the wireless communication comprises changing a bit rate of the wireless communication. 4. The method of claim 1, wherein changing the characteristic of the wireless communication comprises changing a number of retries during the wireless communication. 5. The method of claim 1, wherein predicting the change in network strength comprises predicting the change in network strength based on historical network strength information. 6. The method of claim 5, wherein the historical network strength information comprises historical network strength information at different locations, and wherein predicting the change in network strength comprises predicting a change in location from a first location having a first historical network strength to a second location having a second historical network strength different from the first historical network strength. 7. The method of claim 5, wherein predicting the change in network strength based on the historical network strength information comprises interpolating between a first historical network strength at a first location and a second historical network strength at a second location based on a third location associated with the electronic device. 8. The method of claim 5, wherein the historical network strength information comprises network strength information obtained by one or more additional electronic devices. 9. The method of claim 1, wherein the one or more sensors comprise one or more depth sensors, wherein the three-dimensional model of the physical environment comprises a depth map of the physical environment, and wherein predicting the change in network strength comprises predicting the change in network strength based on a room layout determined using the depth map. 10. The method of claim 1, wherein the three-dimensional model comprises a spatial mesh or a polygonal model. 11. The method of claim 1, wherein the method further comprises: obtaining, using the one or more sensors, scene understanding data of the physical environment, wherein predicting the change in network strength comprises predicting the change in network strength based on the scene understanding data. 12. The method of claim 11, wherein predicting the change in the network strength based on the scene understanding data comprises predicting the change in the network strength based on an identity of a physical object in the physical environment. 13. The method of claim 11, wherein predicting the change in the network strength based on the scene understanding data comprises predicting the change in the network strength based on a material identified in the scene understanding data. 14. The method of claim 1, wherein the electronic device is a head-mounted device, and wherein the external electronic equipment is a cellular telephone. 15. The method of claim 14, wherein exchanging the wireless communication with the external electronic equipment comprises wirelessly transmitting head pose information to the cellular telephone and wirelessly receiving display data from the cellular telephone. 16. A method of operating an electronic device configured to wirelessly communicate with a head-mounted device, the method comprising: rendering display data and wirelessly transmitting the display data to the head-mounted device; predicting a change in a network strength associated with a wireless connection between the electronic device and the head-mounted device; and in response to predicting the change in the network strength, changing a characteristic of the rendering the display data and wirelessly transmitting the display data to the head-mounted device. 17. The method of claim 16, wherein the characteristic comprises a characteristic selected from the group consisting of: forward error correction, bit rate, and number of retries. 18. The method of claim 16, wherein changing the characteristic of the rendering the display data and wirelessly transmitting the display data to the head-mounted device comprises changing a resolution of the display data during the rendering of the display data. 19. The method of claim 16, wherein changing the characteristic of the rendering the display data and wirelessly transmitting the display data to the head-mounted device comprises changing a field of view of the display data during the rendering of the display data. 20. A method of operating an electronic device, the method comprising: exchanging wireless communications with a head-mounted device, wherein exchanging the wireless communications comprises wirelessly transmitting rendered display data to the head-mounted device; and in response to a prediction of a change in a network strength associated with the wireless communications, changing a characteristic of the wireless communications.

Claims

1. A method of operating an electronic device, the method comprising: exchanging wireless communications with an external electronic device; predicting a change in network strength associated with the wireless communications; and in response to predicting the change in network strength, changing a characteristic of the wireless communications.

2. The method of claim 1, wherein changing the characteristic of the wireless communications comprises changing forward error correction applied to the wireless communications.

3. The method of claim 1, wherein changing the characteristic of the wireless communications comprises changing a bit rate of the wireless communications.

4. The method of claim 1, wherein changing the characteristic of the wireless communications comprises changing a number of retries during the wireless communications.

5. The method of claim 1, wherein predicting the change in network strength comprises predicting the change in network strength based on historical network strength information.

6. The method of claim 5, wherein the historical network strength information comprises historical network strength information at different locations, and wherein predicting the change in network strength comprises predicting a change in location from a first location having a first historical network strength to a second location having a second historical network strength different from the first historical network strength.

7. The method of claim 5, wherein predicting the change in network strength based on the historical network strength information comprises interpolating between a first historical network strength at a first location and a second historical network strength at a second location based on a third location associated with the electronic device.

8. The method of claim 5, wherein the historical network strength information comprises network strength information obtained by one or more additional electronic devices.

9. The method of claim 1, wherein the electronic device comprises one or more depth sensors, and wherein the method further comprises: obtaining a depth map of a physical environment using the one or more depth sensors, wherein predicting the change in network strength comprises predicting the change in network strength based on the depth map.

10. The method of claim 9, wherein predicting the change in network strength based on the depth map comprises predicting the change in network strength based on a room layout determined using the depth map.

11. The method of claim 1, wherein the electronic device comprises one or more sensors, and wherein the method further comprises: obtaining scene understanding data of a physical environment using the one or more sensors, wherein predicting the change in network strength comprises predicting the change in network strength based on the scene understanding data.

12. The method of claim 11, wherein predicting the change in network strength based on the scene understanding data comprises predicting the change in network strength based on an identity of a physical object in the physical environment. ​ 13. The method of claim 11, wherein predicting the change in the network strength based on the scene understanding data comprises predicting the change in the network strength based on materials identified in the scene understanding data.

14. The method of claim 1, wherein the electronic device is a head-mounted device, and wherein the external electronic equipment is a cellular telephone.

15. The method of claim 14, wherein exchanging the wireless communication with the external electronic equipment comprises wirelessly transmitting head pose information to the cellular telephone and wirelessly receiving display data from the cellular telephone.

16. A method of operating an electronic device configured to wirelessly communicate with a head-mounted device, the method comprising: rendering display data and wirelessly transmitting the display data to the head-mounted device; predicting a change in a network strength associated with a wireless connection between the electronic device and the head-mounted device; and in response to predicting the change in the network strength, changing a characteristic of the rendering the display data and wirelessly transmitting the display data to the head-mounted device.

17. The method of claim 16, wherein the characteristic comprises a characteristic selected from the group consisting of: forward error correction, bit rate, and number of retries.

18. The method of claim 16, wherein changing the characteristic of the rendering the display data and wirelessly transmitting the display data to the head-mounted device comprises changing a resolution of the display data during the rendering of the display data.

19. The method of claim 16, wherein changing the characteristic of the rendering the display data and wirelessly transmitting the display data to the head-mounted device comprises changing a field of view of the display data during the rendering of the display data.

20. A method of operating an electronic device, the method comprising: exchanging wireless communications with a head-mounted device, wherein exchanging the wireless communications comprises wirelessly transmitting rendered display data to the head-mounted device; and in response to a prediction of a change in a network strength associated with the wireless communications, changing a characteristic of the wireless communications.

Images