Wireless communication adjustment based on network strength

Abstract

A head-mounted device can replace external electronic devices with wireless communication. Weak network strength for wireless communication can cause latency, resulting in a suboptimal user experience. To improve the operation of the head-mounted device, it can predict changes in network strength related to wireless communication. The head-mounted device can predict changes in network strength based on historical network strength data and / or scene understanding data about the physical environment. In response to predicting changes in network strength, the head-mounted device can modify the characteristics of wireless communication. The head-mounted device can change the forward error correction applied to wireless communication, change the bitrate of wireless communication, and / or change the number of retries during wireless communication.

Description

Background Art

[0001] This application claims the benefit of U.S. patent application Ser. No. 18 / 432,718, filed Feb. 5, 2024, and U.S. Provisional Patent Application No. 63 / 493,445, filed Mar. 31, 2023, the entire contents of which are hereby incorporated by reference. This application relates generally to electronic devices, and more particularly to electronic devices having a display.

[0002] Some electronic devices, such as head-mounted devices, include a display (sometimes referred to as a near-eye display) positioned near the user's eyes during operation. The display may present three-dimensional content to the user. If not attended to, latency can cause artifacts and / or discomfort to a user viewing an image on a head-mounted device.

Summary of the Invention

[0003] A method of operating an electronic device can include exchanging wireless communications with an external electronic device, predicting a change in network strength associated with the wireless communications, and changing a characteristic of the wireless communications in response to predicting the change in network strength.

[0004] A method of operating an electronic device configured to wirelessly communicate with a head-mounted device can include rendering display data for wireless transmission to the head-mounted device, predicting a change in network strength associated with a wireless connection between the electronic device and the head-mounted device, and changing a characteristic of rendering display data for wireless transmission to the head-mounted device in response to predicting the change in network strength.

[0005] A method for operating an electronic device may include switching the head-mounted device and the wireless communication, and modifying the characteristics of the wireless communication, in response to predictions of changes in network strength related to the wireless communication. Switching the wireless communication may include wirelessly transmitting rendered display data to the head-mounted device. [Brief explanation of the drawing]

[0006] [Figure 1] This is a schematic diagram of an exemplary system having electronic devices and external electronic equipment according to several embodiments.

[0007] [Figure 2] This flowchart shows exemplary method steps performed by a head-mounted device and a paired electronic device in a remote rendering configuration according to several embodiments.

[0008] [Figure 3] This is a top view of an exemplary augmented reality environment, including physical objects and a head-mounted device, according to several embodiments.

[0009] [Figure 4] This is a flowchart of an exemplary method performed by a head-mounted device to change the characteristics of wireless communication in response to predicting changes in network strength, according to several embodiments.

[0010] [Figure 5] This is a flowchart of an exemplary method performed by an electronic device paired with a head-mounted device, according to several embodiments. [Modes for carrying out the invention]

[0011] Figure 1 shows an exemplary system with an electronic device. System 8 includes an electronic device 10 and an external electronic device 30. The electronic device 10 can be a computing device such as a laptop computer, a computer monitor including an embedded computer, a tablet computer, a cellular telephone, a media player, or other handheld or portable electronic device; a wristwatch-type device, a pendant-type device, a headphone-type or earphone-type device, a device embedded in glasses or other equipment worn on the user's head, or a smaller device such as other wearable or small devices; a display, a computer display including an embedded computer, a computer display not including an embedded computer, a gaming device, a navigation device, an embedded system such as a system in which an electronic device having a display is installed in a kiosk or automobile; or other electronic device. The electronic device 10 may have the shape of a pair of glasses (e.g., a support frame), may form a helmet-shaped housing, or may have other configurations that help to mount and secure one or more display components on the user's head or near the eyes.

[0012] The electronic device 10 may communicate wirelessly with the external electronic device 30. The external electronic device 30 may be a computing device such as a laptop computer, a computer monitor including an embedded computer, a tablet computer, a cellular telephone, a media player, or other handheld or portable electronic device; a wristwatch-type device, a pendant-type device, a headphone-type or earphone-type device, a device embedded in glasses or other equipment worn on the user's head, or a smaller device such as other wearable or small devices; a display, a computer display including an embedded computer, a computer display not including an embedded computer, a gaming device, a navigation device, an embedded system such as a system in which an electronic device having a display is installed in a kiosk or automobile; or other electronic device. The external electronic device 30 may have the shape of a pair of glasses (e.g., a support frame), may form a helmet-shaped housing, or may have other configurations that help to attach and secure one or more display components on the user's head or near their eyes.

[0013] The external electronic device 30 may include one or more servers. These one or more servers may be implemented on a distributed network of one or more standalone data processing units or computers. The one or more servers may, in response to requests from the electronic device 10, provide information such as web page content to the electronic device 10 (e.g., via a network). The network through which the electronic device 10 communicates with the external electronic device 30 may include a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet).

[0014] As shown in Figure 1, the electronic device 10 (sometimes called a head-mounted device 10, system 10, head-mounted display 10, etc.) may have a control circuit 14. The control circuit 14 may be configured to perform operations in the electronic device 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code and other data for performing operations in the electronic device 10 are stored in a non-temporary computer-readable storage medium (e.g., a tangible computer-readable storage medium) within the control circuit 14. Software code may be referred to as software, data, program instructions, instructions, or code. The non-temporary computer-readable storage medium (sometimes commonly referred to as memory) may include non-volatile memory such as non-volatile random-access memory (NVRAM), one or more hard drives (e.g., magnetic drives or solid-state drives), one or more removable flash drives, or other removable media. The software stored in the non-temporary computer-readable storage medium may be executed on the processing circuitry of the control circuit 14. Processing circuitry may include application-specific integrated circuits equipped with processing circuits, one or more microprocessors, digital signal processors, graphics processing units, central processing units (CPUs), or other processing circuits.

[0015] The electronic device 10 may include an input / output circuit 20. The input / output circuit 20 may be used to allow a user to provide user input to the electronic device 10 and / or to collect information about the environment in which the electronic device 10 is operating. Output components within the circuit 20 may allow the electronic device 10 to provide output to the user.

[0016] As shown in Figure 1, the input / output circuit 20 may include a display such as a display 16. The display 16 can be used to display images to the user of the electronic device 10. The display 16 may be a see-through (transparent) display, so that the user can observe physical objects through the display, while computer-generated content is overlaid on the physical objects by presenting computer-generated images on the display. The transparent display may be formed from a transparent pixel array (e.g., a transparent organic light-emitting diode display panel) or from a display device (e.g., a display device such as a liquid crystal on silicon display) that provides images to the user through a beam splitter, holographic coupler, or other optical coupler. Alternatively, the display 16 may be an opaque display that blocks light from physical objects when the user operates the electronic device 10. In this type of configuration, a pass-through camera can be used to display physical objects to the user. The pass-through camera can capture images of the physical environment, and the images of the physical environment may be displayed on the display for the user to view. Additional computer-generated content (e.g., text, game content, other visual content) may optionally be overlaid on the physical environment image to provide the user with an augmented reality environment. If the display 16 is opaque, the display may also optionally display the computer-generated content entirely (e.g., without displaying the physical environment image).

[0017] Display 16 may include one or more optical systems (e.g., lenses) (sometimes called optical assemblies) that enable a viewer to view images on Display 16. A single Display 16 may produce images for both eyes, or a pair of Displays 16 may be used to display images. In a configuration with multiple displays (e.g., a left-eye display and a right-eye display), the focal length and position of the lenses may be selected so that no gaps between the displays are visible to the user (i.e., the images on the left and right displays overlap or merge seamlessly). Display modules (sometimes called display assemblies) that produce different images for the user's left and right eyes are sometimes called stereoscopic displays. Stereoscopic displays may be capable of presenting two-dimensional content (e.g., user notifications with text) and three-dimensional content (e.g., simulations of physical objects such as cubes).

[0018] The input / output circuit 20 may include various other input / output devices. For example, the input / output circuit 20 may include one or more cameras 18. The cameras 18 may include one or more outward-facing cameras (for example, facing the physical environment around the user when the electronic device is mounted on the user's head). The cameras 18 may capture visible light images, infrared images, or any other desired type of image. The cameras may be stereo cameras, if desired. The outward-facing cameras may capture pass-through video of the device 10. The cameras 18 may include one or more inward-facing cameras (for example, to acquire gaze detection information).

[0019] As shown in Figure 1, the input / output circuit 20 may include position and motion sensors 22 (e.g., a compass, gyroscope, accelerometer, and / or other devices for monitoring the location, orientation, and movement of the electronic device 10, a satellite navigation system circuit such as a global positioning system circuit for monitoring the user's location). For example, using the sensors 22, the control circuit 14 may monitor the current orientation of the user's head relative to the surrounding environment (e.g., the user's head posture). The camera 18 may also be considered part of the position and motion sensors 22. The camera may be used for face tracking (e.g., by capturing images of the user's jaw, mouth, etc. while the device is mounted on the user's head), for body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. while the device is mounted on the user's head), and / or for localization (e.g., using visual odometry, visual inertial odometry, or other simultaneous localization and mapping (SLAM) techniques).

[0020] The input / output circuit 20 may include one or more depth sensors 24. Each depth sensor may be a pixelated depth sensor (configured to measure multiple depths across a physical environment, for example) or a point sensor (configured to measure a single depth within a physical environment). Each depth sensor (whether pixelated or point) may use phase detection (e.g., phase detection autofocus pixels) or light detection and ranging (LIDAR) to measure depth. Any combination of depth sensors can be used to determine the depth of a physical object in the physical environment.

[0021] The input / output circuit 20 may also include other sensors and input / output components as needed (e.g., a gaze tracking sensor, an ambient light sensor, a force sensor, a temperature sensor, a touch sensor, an image sensor for detecting hand gestures or body postures, a button, a capacitive proximity sensor, an optical-based proximity sensor, other proximity sensors, a strain gauge, a gas sensor, a pressure sensor, a humidity sensor, a magnetic sensor, a microphone, a speaker, an audio component, a tactile output device such as an actuator, a light-emitting diode, other light sources, etc.).

[0022] The head-mounted device 10 may also include a communication circuit 26 that enables the head-mounted device to communicate with an external device such as an electronic device 30 (e.g., a portable device such as a tether computer, a handheld device, a wristwatch, or a laptop computer, or other electrical equipment). The communication circuit 26 can be used for both wired and wireless communication with the external device.

[0023] The communication circuit 26 can include a radio frequency (RF) transceiver circuit formed from one or more integrated circuits, a power amplifier circuit, a low-noise input amplifier, passive RF components, one or more antennas, transmission lines, and other circuits for processing RF wireless signals. The wireless signal can also be transmitted using light (e.g., using infrared communication).

[0024] The radio frequency transceiver circuit in the wireless communication circuit 26 transmits signals in the following frequency bands: wireless local area network (WLAN) communication bands such as the 2.4GHz and 5GHz Wi-Fi® (IEEE802.11) bands, wireless personal area network (WPAN) communication bands such as the 2.4GHz Bluetooth® communication band, cellular low band (LB) (e.g., 600~960MHz), cellular low-midband (LMB) (e.g., 1400~1550MHz), cellular midband (MB) (e.g., 1700~2200MHz), cellular high band (HB) (e.g., 2300~2700MHz), cellular ultra-high band (UHB) (e.g., 3300~5000MHz), or other cellular communication bands of approximately 600MHz to approximately 5000MHz (e.g., 3G band, 4G band). It can handle cellular telephone communication bands such as LTE bands and 5G New Radio Frequency Range 1 (FR1) bands below 10 GHz, near-field communications (NFC) bands (e.g., at 13.56 MHz), satellite navigation bands (e.g., L1 global positioning system (GPS) band at 1575 MHz, L5 GPS band at 1176 MHz, Global Navigation Satellite System (GLONASS) band, BeiDou Navigation Satellite System (BDS) band, etc.), ultra-wideband (UWB) communication bands supported by the IEEE 802.15.4 protocol, and / or other UWB communication protocols (e.g., a first UWB communication band at 6.5 GHz and / or a second UWB communication band at 8.0 GHz), and / or any other desired communication band.

[0025] The radio frequency transceiver circuit may include a millimeter wave / centimeter wave transceiver circuit that supports communication at frequencies from approximately 10 GHz to 300 GHz. For example, the millimeter wave / centimeter wave transceiver circuit can support communication in the extremely high frequency (EHF) or millimeter wave communication band from approximately 30 GHz to 300 GHz and / or in the centimeter wave communication band from approximately 10 GHz to 30 GHz (which may also be referred to as the super high frequency (SHF) band). As an example, the millimeter wave / centimeter wave transceiver circuit can support communication in the IEEE K communication band from approximately 18 GHz to 27 GHz, the K a communication band from approximately 26.5 GHz to 40 GHz, the K u communication band from approximately 12 GHz to 18 GHz, the V communication band from approximately 40 GHz to 75 GHz, the W communication band from approximately 75 GHz to 110 GHz, or communication in any other desired frequency band from approximately 10 GHz to 300 GHz. If desired, the millimeter wave / centimeter wave transceiver circuit can support IEEE802.11ad communication at 60 GHz (e.g., WiGig near 57 - 61 GHz or the 60 GHz Wi-Fi band) and / or communication in the 5th generation mobile network or 5th generation wireless system (5G) new radio (NR) frequency range 2 (FR2) communication band from approximately 24 GHz to 90 GHz.

[0026] The antenna in the wireless communication circuit 26 can include an antenna having a resonant element formed from a loop antenna structure, a patch antenna structure, an inverted F antenna structure, a slot antenna structure, a planar inverted F antenna structure, a helical antenna structure, a dipole antenna structure, a monopole antenna structure, a hybrid of these designs, etc. Different types of antennas may be used for different bands and combinations of bands. For example, one type of antenna may be used to form a local wireless link and another type of antenna may be used to form a remote wireless link antenna.

[0027] One use case described herein is one in which the electronic device 10 is a head-mounted device and the external electronic device 30 is a paired non-head-mounted device (e.g., a cellular phone, laptop computer, tablet, or wristwatch). The external electronic device 30 may also be referred to as the electronic device 30 or the paired electronic device 30.

[0028] Electronic device 10 may be paired with electronic device 30. In other words, a wireless link can be established between electronic devices 10 and 30 to enable high-speed and efficient communication between devices 10 and 30. Electronic devices 10 and 30 may be associated with the same user (for example, they may sign in to a cloud service using the same user ID) and may exchange wireless communications. Each of electronic devices 10 and 30 may include a battery. In one embodiment, the head-mounted device 10 may have a battery with a lower total capacity than electronic device 30.

[0029] If desired, content for the display 16 on the head-mounted device 10 may be rendered by the electronic device 30, wirelessly transmitted from the electronic device 30 to the head-mounted device 10, and then displayed. Rendering the display content of a first electronic device using a second electronic device is sometimes referred to herein as remote rendering. Remote rendering can be useful in reducing power consumption on one electronic device. For example, it may be desirable to reduce the power consumption of the head-mounted device 10. Remote rendering shifts some of the processing burden (and therefore power consumption) for operating the display 16 on the electronic device 10 to the electronic device 30.

[0030] In some cases, the graphics processing unit (GPU) within the electronic device 30 may be capable of more complex rendering operations than the graphics processing unit within the electronic device 10 (for example, within the control circuit 14). In this case, remote rendering may enable the rendering of more complex display data than would be possible if the head-mounted device 10 were using only its local GPU.

[0031] Figure 2 is a flowchart of exemplary method steps performed by a head-mounted device and paired electronic devices in a remote rendering configuration. As shown, the steps on the left (e.g., steps 102, 112, 116, 118, and 120) are performed by a first electronic device, such as the head-mounted device 10 in Figure 1. The steps on the right (e.g., steps 104, 106, 108, 110, and 114) are performed by a second electronic device, such as the electronic device 30 in Figure 1. The two devices in Figure 2 may be paired and may exchange wireless communications.

[0032] In step 102, the head-mounted device 10 can transmit head posture information to the paired electronic device 30. The head-mounted device 10 may, for example, acquire head posture information using position and motion sensors 22. The head-mounted device 10 may wirelessly transmit head posture information (for example, using Bluetooth communication). The head-mounted device 10 may transmit a single head posture for a single point in time, or multiple head postures associated with different point in time. In other words, the head-mounted device 10 may transmit one or more head postures, each head posture having a corresponding timestamp. Generally, the transmitted head posture information may include any desired additional sensor information, context information, historical data, etc.

[0033] In step 104, the electronic device 30 can receive head pose information from the paired head-mounted device 10. The electronic device 30 then uses the received head pose information to estimate the head pose for a given display frame in step 106. The electronic device 30 may use the historical head pose information of the head-mounted device 10 stored in the electronic device 30 to estimate the head pose for a given display frame. For example, the head pose information from the head-mounted device 10 may identify the first head pose at a first time (t1), the second head pose at a second time (t2), and the third head pose at a third time (t3). The estimated time for displaying a given display frame is t4. The electronic device uses the head poses over time at t1, t2, and t3 to predict the head pose at t4. The electronic device 30 may use any desired number of head poses (e.g., one or more) to predict the head pose at t4.

[0034] In step 108, the electronic device 30 can render display data for a given display frame using the estimated head pose from step 106. The electronic device 30 can render the display data using a graphics processing unit or any other desired computing resource. The rendered display data can then be compressed in step 110. The compressed display data is then wirelessly transmitted to a paired head-mounted device in step 114.

[0035] In step 112, the head-mounted device 10 can receive rendering display data from the paired electronic device. The received rendering display data may have an associated target head pose (for example, the fourth head pose for time t4, using the example above). In step 116, the head-mounted device 10 can decompress the rendering display data. In general, any desired encoding / decoding scheme can be used to compress and decompress the display data.

[0036] In step 118, the head-mounted device can adjust the display data based on the updated head pose information. As previously mentioned, the rendered display data can be rendered against the head pose predicted by the electronic device 30. The head-mounted device 10 can modify the head pose estimation for a given display frame based on the head pose data acquired between step 102 (when the head pose information is transmitted to the device 30) and step 118.

[0037] In the previous example, the electronic device 30 renders display data for a given display frame regarding the estimated fourth head pose at target time t4. In step 118, the head-mounted device 10 can estimate the fifth head pose at target time t4 using the updated head pose information collected using the position and motion sensors 22. Therefore, in step 118, the head-mounted device 10 adjusts the display data to compensate for the difference between the initially predicted head pose (e.g., the fourth head pose) and the newly predicted head pose (e.g., the fifth head pose). Adjusting the display data to compensate for the difference between the initially predicted head pose and the newly predicted head pose may include reprojecting the display data onto the newly predicted head pose.

[0038] Finally, in step 120, the head-mounted device 10 can use the display 16 to display the display data for a given frame. If desired, steps 118 and 120 (e.g., adjusting the display data based on head pose information and displaying the data) may be performed at a higher frequency than step 108 (e.g., rendering the display data). For example, steps 118 and 120 may be performed at twice the frequency of step 108. As an example, steps 118 and 120 may be performed at a frequency of 120 Hz, and step 108 may be performed at a frequency of 60 Hz.

[0039] Using the remote rendering operation shown in Figure 2, the head-mounted device 10 conserves power compared to a configuration where all display data is rendered locally. However, the remote rendering operation relies on wireless communication between the head-mounted device 10 and the electronic device 30. Therefore, unreliable network strength between the head-mounted device 10 and the electronic device 30 can cause undesirable latency in the content displayed to the user of the head-mounted device.

[0040] In the example shown in Figure 2, the head-mounted device 10 relies on wireless communication for remote rendering configuration with an external electronic device 30. However, this example is merely illustrative. In general, the head-mounted device 10 may use wireless communication in a number of applications (e.g., video conferencing, online games, entertainment streaming, etc.). For all of these wireless communications, weak network strength for wireless communication can cause latency that results in a suboptimal user experience.

[0041] To improve the operation of the head-mounted device 10, the head-mounted device may predict changes in network strength related to wireless communication. Depending on the prediction of changes in network strength, the head-mounted device may modify the characteristics of the wireless communication. For example, consider a scenario in which a user is participating in a video conference call. If the head-mounted device predicts a decrease in the network strength of the wireless communication used for the video conference call, the head-mounted device may take preemptive mitigation measures such as changing the forward error correction applied to the wireless communication, changing the bitrate of the wireless communication, or changing the number of retries during wireless communication.

[0042] The head-mounted device 10 can predict changes in network strength based on historical network strength data and / or scene understanding data about the physical environment.

[0043] Figure 3 is a top view of an exemplary three-dimensional environment including the electronic device 10. In Figure 3, the three-dimensional environment 36 is an augmented reality (XR) environment that includes the physical environment around the electronic device 10 (which has various physical objects). The three-dimensional environment 36 may optionally include one or more virtual objects. The XR environment 36 may include multiple physical walls 32 that define several rooms (e.g., rooms in a house or apartment). The exemplary XR environment in Figure 3 has five rooms. The physical environment around the electronic device 10 may include physical objects such as physical objects 34-1, 34-2, and 34-3. The physical objects shown in Figure 3 may be any desired type of physical object (e.g., a table, bed, sofa, chair, refrigerator, etc.).

[0044] While the electronic device 10 is operating, it can move throughout the entire three-dimensional environment 36. In other words, a user of the electronic device 10 can repeatedly move the electronic device 10 between different rooms within the three-dimensional environment 36. While operating in the three-dimensional environment 36, the electronic device 10 can collect sensor data about the three-dimensional environment 36 using one or more sensors (e.g., a camera 18, position and motion sensors 22, a depth sensor 24, etc.). The electronic device 10 can build a scene understanding dataset for the three-dimensional environment.

[0045] To construct a scene understanding dataset, the electronic device may use input from sensors such as a camera 18, position and motion sensors 22, and a depth sensor 24. For example, data from the depth sensor 24 and / or the position and motion sensors 22 may be used to construct a spatial mesh representing the physical environment. The spatial mesh may contain a polygonal model of the physical environment and / or a set of vertices representing the physical environment. The spatial mesh (sometimes called spatial data, etc.) can define the size, location, and orientation of planes within the physical environment. The spatial mesh represents the physical environment around the electronic device.

[0046] Other data, such as data from camera 18, can be used to construct a scene understanding dataset. For example, camera 18 may capture images of the physical environment. Electronic devices can analyze these images to identify properties of planes within a spatial mesh (e.g., the color of the plane or the material that forms the plane). These properties may be included in the scene understanding dataset.

[0047] A scene understanding dataset may include the identities of various physical objects within an augmented reality environment. For example, electronic device 10 may identify physical objects by analyzing images from camera 18 and / or depth sensor 24. Electronic device 10 can identify physical objects such as beds, sofas, chairs, tables, and refrigerators. This information identifying physical objects may be included in the scene understanding dataset.

[0048] The scene understanding dataset may also contain information about various virtual objects within the augmented reality environment. The electronic device 10 may be used to display virtual objects and therefore knows the identity, size, shape, color, etc., of the virtual objects in the augmented reality environment. This information about the virtual objects may be included in the scene understanding dataset.

[0049] A scene understanding dataset can be built on the electronic device 10 over time as the electronic device moves throughout the augmented reality environment. For example, consider an example where the electronic device 10 starts in the room in Figure 3, which contains physical objects 34-1 and 34-2. The electronic device can use a depth sensor to obtain depth information for the currently occupied room (which contains objects 34-1 / 34-2) (and unfold the spatial mesh). The electronic device can then unfold a scene understanding dataset for the currently occupied room (including the spatial mesh, physical object information, virtual object information, etc.). At this point, the electronic device has no information about unoccupied rooms.

[0050] Next, the electronic device 10 may be transported to a room along with the physical object 34-3. While in this new room, the electronic device can use a depth sensor to acquire depth information for the currently occupied room (containing object 34-3) (and unfold the spatial mesh). The electronic device can unfold a scene understanding dataset for the currently occupied room (including the spatial mesh, physical object information, virtual object information, etc.). The electronic device has a scene understanding dataset that includes data for both the currently occupied room (containing object 34-3) and previously occupied rooms (containing objects 34-1 and 34-2). In other words, as the electronic device enters a new part of the three-dimensional environment, data may be added to the scene understanding dataset. Thus, over time (as the electronic device is transported to all rooms in the three-dimensional environment), the scene understanding dataset contains data for the entire three-dimensional environment 36.

[0051] The electronic device 10 may maintain a scene understanding dataset that includes all scene understanding data associated with the augmented reality environment 36 (for example, including both currently occupied and currently unoccupied rooms).

[0052] In summary, the head-mounted device 10 can maintain a scene understanding dataset containing a spatial mesh representing the physical environment, properties of planes within the spatial mesh, the identities of various physical objects in the augmented reality environment, and / or information about various virtual objects in the augmented reality environment. In addition to the scene understanding dataset, the head-mounted device may maintain historical network intensity information. The historical network intensity information may be considered part of the scene understanding dataset, or it may be considered separate from the scene understanding dataset.

[0053] Historical network strength information can be accumulated over time during the operation of the head-mounted device. Network strength can be characterized in any desired manner. For example, the head-mounted device can characterize the network strength associated with the radio communication using the measured bitrate of the uplink transmission, the measured bitrate of the downlink transmission, and / or the expected bitrate associated with the type of radio communication being used. As an additional example, network strength may be measured by the head-mounted device 10 using milliwatts (mW), decibels per milliwatt (dBm), or the Received Signal Strength Indicator (RSSI). Any subset of the factors described above can be used to characterize the network strength.

[0054] Figure 3 identifies three different locations in a three-dimensional environment, namely Location A, Location B, and Location C. As the head-mounted device 10 spends time in each of these locations, the network strength associated with the location may be stored in the head-mounted device 10. In addition to being associated with a location, the stored network strength information may have associated timestamps (to identify trends in network strength as a function of time), network type (e.g., Wi-Fi vs. cellular), etc.

[0055] In one example, all historical network strength information stored in the electronic device 10 can be retrieved using the head-mounted device 10. In another possible example, at least a portion of the historical network strength information may be retrieved by one or more external electronic devices. Multiple devices may contribute to a shared network strength database accessible by the head-mounted device 10. This may allow the head-mounted device 10 to use network strength information for locations it has not yet moved to.

[0056] Consider a scenario where historical network strength information indicates a first network strength at location A, a second network strength at location B, and a third network strength at location C. The second network strength may be weaker than the first network strength. The third network strength may be weaker than the second network strength. A head-mounted device may detect that the user is moving from location A to location B. In this scenario, the head-mounted device may predict a decrease in network strength (for example, if the user reaches location B with a weaker network strength than at location A). In an alternative scenario, the head-mounted device may detect that the user is moving from location C to location B. In this scenario, the head-mounted device may predict an increase in network strength (for example, if the user reaches location B with a stronger network strength than at location C).

[0057] In an example where all historical network strength information stored in the electronic device 10 is acquired using the head-mounted device 10, the head-mounted device 10 needs to acquire network strength information before it can move to each of locations A, B, and C and make network strength predictions related to those locations. In an example where at least a portion of the historical network strength information can be acquired by one or more external electronic devices, the head-mounted device 10 can use the network strength information acquired using one or more external electronic devices to make network strength predictions related to locations A, B, and C without moving to locations A, B, and C.

[0058] Depending on the predicted changes in the wireless communication network strength, the head-mounted device can make various adjustments to the wireless communication. These adjustments may include changing the forward error correction applied to the wireless communication, changing the bitrate of the wireless communication, changing the number of retries during wireless communication, or other desired adjustments.

[0059] This specification describes an example in which a head-mounted device 10 predicts network strength using historical network strength information (associated with location, time, etc.). This example is for illustrative purposes only. Alternatively, or in addition, network strength prediction may be performed using a scene understanding dataset acquired by the head-mounted device 10.

[0060] The scene understanding dataset may include a spatial mesh of the room currently occupied by the head-mounted device. The spatial mesh can be used to determine the dimensions of the room containing the head-mounted device. The room dimensions can then be used to make predictions about network strength.

[0061] As another example, a scene understanding dataset may contain information about materials in the physical environment. For instance, a scene understanding dataset could identify materials (e.g., metal or concrete) associated with insufficient network strength. Therefore, a head-mounted device could use the materials identified in the scene understanding data to predict changes in network strength.

[0062] As another example, a scene understanding dataset may contain information about physical objects within the physical environment. For instance, a scene understanding dataset may identify physical objects associated with insufficient network strength. Therefore, a head-mounted device could use the physical objects identified in the scene understanding data to predict changes in network strength.

[0063] Consider an example where a user wearing a head-mounted device 10 enters a parking lot. Scene understanding data acquired by the head-mounted device 10 may identify a concrete wall around the head-mounted device. The presence of the concrete wall may be associated with insufficient network strength. Therefore, the head-mounted device 10 can predict a decrease in network strength when the user enters the parking lot.

[0064] As another example, scene understanding data acquired by the head-mounted device 10 may identify that the user is in a parking lot. Being in a parking lot may be associated with insufficient network strength. Therefore, the head-mounted device 10 can predict a decrease in network strength in response to detecting that the user is in a parking lot.

[0065] Figure 4 is a flowchart of exemplary method steps for operating a head-mounted device. As shown, in step 202, the head-mounted device 10 may exchange wireless communication with an external electronic device 30. The external electronic device may be a paired electronic device (e.g., a cellular phone, laptop computer, tablet, watch, etc.), an external server, or other external electronic device. Wireless communication may include cellular communication, Bluetooth communication, Wi-Fi communication, or any other desired type of communication.

[0066] In step 204, the head-mounted device 10 can predict changes in network strength related to wireless communication. The head-mounted device 10 can predict changes in network strength using location data (e.g., from position and motion sensors 22), historical network strength data (acquired using the head-mounted device 10 and / or one or more external electronic devices), and / or scene understanding data.

[0067] For example, the head-mounted device 10 can predict changes in location and forecast changes in network strength based on historical network strength data for the current location and the predicted new location. The head-mounted device 10 may also forecast changes in network strength based on historical network strength data for the head-mounted device's current location.

[0068] In some cases, interpolation can be used to determine the predicted network strength using historical network strength information. For example, historical network strength information may include a first network strength for a first location and a second network strength for a second location. If the head-mounted device 10 is in (or is predicted to be in) a third location between the first and second locations, the third network strength for the third location can be estimated using interpolation between the first and second network strengths.

[0069] The head-mounted device may determine a depth map of the physical environment using at least a depth sensor 24. The head-mounted device may use the depth map to predict changes in network strength. For example, the head-mounted device may use the room layout determined using the depth map to predict changes in network strength. The depth map may be considered part of a scene understanding dataset maintained by the head-mounted device.

[0070] The head-mounted device may acquire scene understanding data using a camera 18, position and motion sensors 22, and / or a depth sensor 24. The head-mounted device can use the scene understanding data to predict changes in network strength. For example, the head-mounted device may use the identity of physical objects included in the scene understanding data to predict changes in network strength. As another example, the head-mounted device may use the material included in the scene understanding data (e.g., metal, concrete, drywall, or another desired material) to predict changes in network strength.

[0071] Next, in step 206, the head-mounted device 10 may change the characteristics of the wireless communication in response to a prediction of changes in network strength. Examples of adjustments to the wireless communication include changing the forward error correction applied to the wireless communication, changing the bitrate of the wireless communication, and changing the number of retries during the wireless communication.

[0072] In step 202, consider an example where radio communication is exchanged without forward error correction. Then, in step 204, a decrease in the network strength associated with the radio communication is predicted. In this case, in step 206, forward error correction may be applied to the radio communication.

[0073] In step 202, consider another example where wireless communication is transmitted using a first bitrate. Then, in step 204, a decrease in the network strength associated with the wireless communication is predicted. In this case, the transmission bitrate may be reduced in step 206.

[0074] As explained in relation to Figure 2, one type of wireless communication performed by the head-mounted device 10 is in a remote rendering configuration with a paired electronic device. In the remote rendering configuration, the head-mounted device 10 receives rendered display data from the paired electronic device and displays the rendered display data on the display 16. The paired electronic device may optionally render display data for multiple viewpoints (e.g., a first frame of image data for an expected primary viewpoint, and one or more auxiliary frames of image data for auxiliary viewpoints different from the primary viewpoint). In step 206, the remote rendering communication may be adjusted in response to expected changes in the network strength of the wireless connection between the head-mounted device 10 and the paired electronic device. For example, the head-mounted device 10 may send predictive information and / or commands regarding expected changes in network strength to the paired electronic device. Accordingly, the paired electronic device may render display data for a different number of viewpoints with different complications (e.g., different number of polygons, 3D voxel resolution, number of objects in the content, etc.), different resolutions, and / or different fields of view. The paired electronic device may render display data for more viewpoints (e.g., three discrete frames of image data for each of three viewpoints) in response to a predicted increase in network strength, or for fewer viewpoints (e.g., one frame of image data for one individual viewpoint) in response to a predicted decrease in network strength. The paired electronic device may render display data with greater complexity in response to a predicted increase in network strength, or with less complexity in response to a predicted decrease in network strength. The paired electronic device may render display data at a higher resolution in response to a predicted increase in network strength, or at a lower resolution in response to a predicted decrease in network strength.The paired electronic devices may render the display data in a larger field of view in response to a predicted increase in network strength, or in a smaller field of view in response to a predicted decrease in network strength. As yet another example, in step 206, the head-mounted device 10 may (in response to a predicted weakening of network strength) discontinue the remote rendering configuration and instead render the display data locally using a local graphics processing unit (GPU) (e.g., the GPU in the control circuit 14).

[0075] Figure 5 is a flowchart of exemplary method steps performed by an electronic device for remotely rendering display data for a paired head-mounted device. In step 302, the electronic device exchanges communications with the head-mounted device. The communications may include receiving head pose information from the head-mounted device and transmitting rendered display data to the head-mounted device.

[0076] In step 304, the electronic device can modify the characteristics of the wireless communication in response to predictions of changes in network strength related to the wireless communication. The electronic device 30 can locally predict changes in network strength using location information and / or historical network strength information. Alternatively, the electronic device 30 may receive prediction information from the head-mounted device 10. As yet another alternative, the electronic device 30 may receive commands from the head-mounted device 10 for adjustments to be made to the wireless communication with the head-mounted device 10 (without explicit predictions included in the commands).

[0077] In step 304, the electronic device may render display data for the head-mounted device 10 at different resolutions and / or different fields of view. For example, the electronic device 30 may render display data at a higher resolution in response to a predicted increase in network strength, or at a lower resolution in response to a predicted decrease in network strength. The electronic device 30 may also render display data in a larger field of view in response to a predicted increase in network strength, or in a smaller field of view in response to a predicted decrease in network strength.

[0078] Alternatively, or in addition, in step 304, the electronic device 30 may change the forward error correction applied to the wireless communication, change the bitrate of the wireless communication, change the number of retries during the wireless communication, and so on.

[0079] Examples of head pose prediction and rendering display data based on head pose prediction are described herein. It should be noted that head pose prediction may depend on a variety of factors. These include the nature of the content being rendered, the user's position relative to the environment, the user's current movement (e.g., walking vs. standing or sitting), whether the user is near or interacting with other users, and accumulated historical data about both that particular user and the user's habits across a broader population. Regarding the nature of the content, characteristics such as the complexity of the content, whether the content is head-fixed, body-fixed, or world-fixed, and the placement of the content in both physical and virtual environments can also play a role in determining how additional views are rendered or how content transmitted via links is adjusted.

[0080] To this end, the head posture information transmitted from the head-mounted device 10 to the electronic device 30 (for example, in step 102 of Figure 2) may include additional information that may influence head posture prediction, in addition to accelerometer data from the position and motion sensor 22. This additional information may include information about the user's position relative to the environment, the user's current movement (e.g., walking versus standing or sitting), and whether the user is near or interacting with other users.

[0081] As described above, one aspect of this technology is the collection and use of information such as sensor information. This disclosure intends that, in some cases, data may be collected that includes personal information data that uniquely identifies a specific person, or personal information data that can be used to contact a specific person or to indicate the location of a specific person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), birth dates, usernames, passwords, biometric information, or any other identifying or personal information.

[0082] This disclosure acknowledges that such use of personal information in the technology may be used for the benefit of the user. For example, personal data may be used to deliver more interesting and targeted content to the user. Thus, such use of personal data will allow the user to control the content delivered to them. Furthermore, this disclosure envisions other uses of personal data that may benefit the user. For example, health and fitness data may be used to provide insights into the user's overall wellness, or as positive feedback to individuals using the technology to pursue wellness goals.

[0083] This disclosure implies that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal data should adhere to a robust privacy policy and / or privacy practices. Specifically, such entities should implement and consistently use a privacy policy and practices that are generally recognized as meeting or exceeding industry or government requirements for the strict confidentiality of personal data. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold for any other purpose. Furthermore, such collection / sharing should be carried out only after informing and obtaining the user's consent. In addition, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others with access to the personal data faithfully adhere to those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), and health data in other countries may be subject to and should be processed accordingly. Therefore, different privacy practices should be maintained in each country with respect to different types of personal data.

[0084] Notwithstanding the foregoing, the Disclosure also envisions embodiments that allow a user to selectively prevent the use of or access to personal data. That is, the Disclosure envisions that hardware and / or software elements may be provided to prevent or prevent access to such personal data. For example, the technology may be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter when registering for the service. In another example, a user may choose not to provide certain types of user data. In yet another example, a user may choose to limit the length of time that user-specific data is retained. In addition to providing “opt-in” and “opt-out” options, the Disclosure envisions providing notices regarding access to or use of personal data. For example, a user may be notified when downloading an application (“App”) that will access their personal data, and then again immediately before the App accesses their personal data.

[0085] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization can be facilitated by removing certain identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means, where necessary.

[0086] Accordingly, while this disclosure extensively involves using information that may include personal data to implement one or more different disclosed embodiments, this disclosure also intends that the different embodiments may also be implemented without requiring access to personal data. In other words, the different embodiments of the technology will not be rendered inoperable by the absence of all or part of such personal data.

[0087] According to one embodiment, a method is provided for operating an electronic device, which includes exchanging wireless communication with an external electronic device, predicting changes in network strength related to wireless communication, and changing the characteristics of wireless communication in response to the predicted changes in network strength.

[0088] According to another embodiment, modifying the characteristics of wireless communication includes modifying the forward error correction applied to wireless communication.

[0089] According to another embodiment, changing the characteristics of wireless communication includes changing the bitrate of wireless communication.

[0090] According to another embodiment, modifying the characteristics of wireless communication includes modifying the number of retries during wireless communication.

[0091] According to another embodiment, predicting changes in network strength includes predicting changes in network strength based on historical network strength information.

[0092] According to another embodiment, the historical network strength information includes historical network strength information at different locations, and predicting changes in network strength includes predicting a change in location from a first location having a first historical network strength to a second location having a second historical network strength different from the first historical network strength.

[0093] According to another embodiment, predicting changes in network strength based on historical network strength information includes interpolating between a first historical network strength at a first location and a second historical network strength at a second location, based on a third location associated with an electronic device.

[0094] According to another embodiment, the historical network strength information includes network strength information acquired by one or more additional electronic devices.

[0095] According to another embodiment, the electronic device includes one or more depth sensors, and the method further includes using one or more depth sensors to obtain a depth map of the physical environment, and predicting changes in network strength includes predicting changes in network strength based on the depth map.

[0096] According to another embodiment, predicting network strength changes based on a depth map includes predicting network strength changes based on a room layout determined using a depth map.

[0097] According to another embodiment, the electronic device includes one or more sensors, the method further includes using one or more sensors to acquire scene understanding data of the physical environment, and predicting changes in network strength includes predicting changes in network strength based on the scene understanding data.

[0098] According to another embodiment, predicting network strength changes based on scene understanding data includes predicting network strength changes based on the identity of physical objects in the physical environment.

[0099] According to another embodiment, predicting changes in network intensity based on scene understanding data includes predicting changes in network intensity based on materials identified in the scene understanding data.

[0100] In another embodiment, the electronic device is a head-mounted device, and the external electronic device is a cellular telephone.

[0101] According to another embodiment, exchanging external electronic devices with wireless communication includes wirelessly transmitting head posture information to a cellular telephone and wirelessly receiving display data from the cellular telephone.

[0102] According to one embodiment, a method is provided for operating an electronic device configured to wirelessly communicate with a head-mounted device, the method comprising rendering display data and wirelessly transmitting it to the head-mounted device; predicting changes in network strength related to a wireless connection between the electronic device and the head-mounted device; and modifying the characteristics of rendering the display data and wirelessly transmitting it to the head-mounted device in response to the predicted changes in network strength.

[0103] According to another embodiment, the characteristics include characteristics selected from the group consisting of forward error correction, bit rate, and number of retries.

[0104] According to another embodiment, modifying the characteristics of rendering and wirelessly transmitting display data to a head-mounted device includes changing the resolution of the display data while rendering the display data.

[0105] According to another embodiment, modifying the characteristics of rendering display data and wirelessly transmitting it to a head-mounted device includes changing the field of view of the display data while rendering the display data.

[0106] According to one embodiment, a method is provided for operating an electronic device, which includes exchanging a head-mounted device with wireless communication, the exchange of wireless communication including wirelessly transmitting rendered display data to the head-mounted device, and modifying the characteristics of the wireless communication in response to predictions about changes in network strength related to the wireless communication.

[0107] The above is merely illustrative, and various modifications may be made to the described embodiments. The aforementioned embodiments may be implemented individually or in any combination.

Claims

1. A method for operating an electronic device including one or more sensors, wherein the method is The exchange of external electronic devices and wireless communication, Using one or more of the aforementioned sensors, a three-dimensional model of the physical environment is obtained, To predict changes in network strength related to wireless communication based at least partially on the three-dimensional model of the physical environment, In accordance with predicting the changes in the network strength, the characteristics of the wireless communication are changed, Methods that include...

2. The method according to claim 1, wherein changing the characteristics of the wireless communication includes changing the forward error correction applied to the wireless communication.

3. The method according to claim 1, wherein changing the characteristics of the wireless communication includes changing the bit rate of the wireless communication.

4. The method according to claim 1, wherein changing the characteristics of the wireless communication includes changing the number of retries during the wireless communication.

5. The method according to claim 1, wherein predicting the change in network strength includes predicting the change in network strength based on historical network strength information.

6. The method according to claim 5, wherein the historical network strength information includes historical network strength information at different locations, and predicting the change in the network strength includes predicting a change in location from a first location having a first historical network strength to a second location having a second historical network strength different from the first historical network strength.

7. The method according to claim 5, wherein predicting the change in the network strength based on the historical network strength information includes interpolating between a first historical network strength at a first location and a second historical network strength at a second location based on a third location associated with the electronic device.

8. The method according to claim 5, wherein the historical network strength information includes network strength information acquired by one or more additional electronic devices.

9. The method according to claim 1, wherein the one or more sensors include one or more depth sensors, the three-dimensional model of the physical environment includes a depth map of the physical environment, and predicting the change in network strength includes predicting the change in network strength based on a room layout determined using the depth map.

10. The method according to claim 1, wherein the three-dimensional model includes a spatial mesh or a polygon model.

11. The aforementioned method, The method according to claim 1, further comprising using one or more sensors to acquire scene understanding data of the physical environment, and predicting the changes in the network strength, comprising predicting the changes in the network strength based on the scene understanding data.

12. The method according to claim 11, wherein predicting the change in network strength based on the scene understanding data includes predicting the change in network strength based on the identity of physical objects in the physical environment.

13. The method according to claim 11, wherein predicting the change in network strength based on the scene understanding data includes predicting the change in network strength based on materials identified in the scene understanding data.

14. The method according to claim 1, wherein the electronic device is a head-mounted device and the external electronic device is a cellular telephone.

15. The method according to claim 14, wherein exchanging the external electronic device with the wireless communication includes wirelessly transmitting head posture information to the cellular telephone and wirelessly receiving display data from the cellular telephone.

16. A method for operating an electronic device configured to communicate wirelessly with a head-mounted device, wherein the method is Rendering the display data and wirelessly transmitting it to the head-mounted device, To predict changes in network strength related to the wireless connection between the electronic device and the head-mounted device, In accordance with predicting the changes in the network strength, the characteristics of rendering the display data and wirelessly transmitting it to the head-mounted device are changed. Methods that include...

17. The method according to claim 16, wherein the characteristics include characteristics selected from the group consisting of forward error correction, bit rate, and number of retries.

18. The method according to claim 16, wherein changing the characteristics of rendering the display data and wirelessly transmitting it to the head-mounted device includes changing the resolution of the display data during the rendering of the display data.

19. The method of claim 16, wherein changing the characteristics of rendering the display data and wirelessly transmitting it to the head-mounted device includes changing the field of view of the display data during the rendering of the display data.

20. A method for operating an electronic device, wherein the method is The exchange of wireless communication between a head-mounted device and the head-mounted device includes wirelessly transmitting rendered display data to the head-mounted device. In accordance with predictions regarding changes in network strength related to the aforementioned wireless communication, the characteristics of the wireless communication are modified. Methods that include...

Images