Prioritizing host workloads in partitioned extended reality (XR) systems

Abstract

A method for scheduling workloads by a host device communicating with a head-mounted display includes prioritizing the processing of a first head-mounted display (HMD) task in response to the host device determining that it is within a target wake time (TWT) window. The method also includes prioritizing the processing of a host task in response to the host device determining that it is outside the TWT window. The method may further include evaluating whether a host task can be completed in time to allow sufficient time for a second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or the processing core clock frequency.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the benefit of Indian Patent Application No. 202341032402, filed on May 8, 2023, entitled "HOST WORKLOAD PRIORITIZATION IN SPLIT EXTENDED REALITY (XR) SYSTEMS", the entire disclosure of which is hereby expressly incorporated by reference herein.

Background Art

[0002] Field of the Invention Aspects of the present disclosure relate to computing devices, and more particularly, to prioritization of host workloads in split extended reality (XR) systems.

[0003] Background Mobile or portable computing devices include mobile phones, laptops, palmtops and tablet computers, portable digital assistants (PDAs), portable game consoles, and other portable electronic devices. Mobile computing devices consist of many electrical components that consume power and generate heat. The components (or computing devices) can include, among other things, system - on - a - chip (SoC) devices, graphics processing unit (GPU) devices, neural processing unit (NPU) devices, digital signal processors (DSPs), and modems.

[0004] Extended reality (XR) includes virtual reality (VR), mixed reality (MR), and augmented reality (AR). Augmented reality is the extension of the real (e.g., physical) world with virtual content. This extension can be achieved using wearable AR devices that can map the physical world, orient themselves within that physical world, and place and render virtual content on a user-visible near-eye display. Many such devices utilize hand and / or fingertip tracking to enable users to control interfaces in augmented reality. It is desirable to improve processing in XR devices. [Overview of the project]

[0005] In aspects of this disclosure, a method for scheduling workloads by a host device communicating with a head-mounted display includes prioritizing the processing of head-mounted display tasks in response to the host device determining that it is within a target wake time (TWT) window.

[0006] Other aspects of this disclosure relate to a device comprising at least one memory and one or more processors coupled to at least one memory. The processor(s) are configured to prioritize processing a head-mounted display task in response to the host device determining that it is within a target wake time (TWT) window.

[0007] Other aspects of this disclosure relate to a device, which includes means for prioritizing the processing of a head-mounted display task in response to determining that a host device is within a target wake time (TWT) window. The device further includes means for prioritizing the processing of a host task in response to determining that a host device is outside the TWT window.

[0008] In other aspects of this disclosure, a non-temporary computer-readable medium recording program code is disclosed. The program code is executed by a processor and includes program code for prioritizing the processing of a head-mounted display task in response to the determination that the host device is within a target wake time (TWT) window.

[0009] The above provides a fairly broad overview of the features and technical advantages of this disclosure in order to better understand the following "Modes for Carrying Out the Invention." Additional features and advantages of this disclosure are described below. Those skilled in the art will understand that this disclosure can be readily used as a basis for modifying or designing other structures to accomplish the same objectives of this disclosure. Those skilled in the art will also recognize that such equivalent configurations do not deviate from the teachings of this disclosure as described in the appended claims. Novel features that are considered unique to this disclosure will be better understood, along with further objectives and advantages, with respect to both their organization and operation, by considering the following descriptions in relation to the accompanying drawings. However, it should be clearly understood that each of the drawings is provided for illustrative and explanatory purposes only and is not intended to define any limitations of this disclosure.

[0010] For a more complete understanding of this disclosure, the following description, in conjunction with the attached drawings, should be referenced. [Brief explanation of the drawing]

[0011] [Figure 1] This block diagram shows exemplary implementations of a host system-on-a-chip (SoC) including a target wake time (TWT) aware scheduler, according to some aspects of the present disclosure. [Figure 2] This is a block diagram showing one embodiment of a segmented extended reality (XR) pipeline according to several aspects of the present disclosure. [Figure 3] This is a timing diagram illustrating one embodiment of a TWT timeline according to several aspects of the present disclosure. [Figure 4] This flowchart illustrates one embodiment of TWT-aware workload scheduling and scaling for a host-based computing engine, according to several aspects of the present disclosure. [Figure 5] This is a flowchart illustrating an exemplary process, for example, executed by a host device, according to various aspects of this disclosure. [Figure 6] This block diagram shows an exemplary wireless communication system that may favorably adopt one configuration of the present disclosure. [Figure 7] This block diagram shows a design workstation used for circuit design, layout design, and logic design of components according to various aspects of this disclosure. [Modes for carrying out the invention]

[0012] The modes for carrying out the invention described below with respect to the attached drawings illustrate various configurations and do not represent only configurations in which the described concepts can be put into practice. The “modes for carrying out the invention” include certain details intended to provide a complete understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be put into practice even without these specific details. In some examples, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0013] Where indicated, the use of the terms “and / or” is intended to represent an “inclusive OR,” and the use of the term “or” is intended to represent an “exclusive OR.” Where indicated, the term “exemplary” as used throughout this description means “serving as an example, instance, or illustration,” and should not necessarily be interpreted as being preferable or advantageous to other exemplary configurations. Where indicated, the term “connected” as used throughout this description means “connected electrically, mechanically, or otherwise, whether direct or indirectly through an intervening connection (e.g., a switch),” and is not necessarily limited to physical connections. Furthermore, such connections may be such that the objects are permanently connected or disconnectably connected. Such connections may be via switches. Where indicated, the term “proximity” as used throughout this description means “adjacent, very close, adjacent, or near.” Where indicated, the term “on” as used throughout this description means “directly on” in some configurations and “indirectly on” in other configurations.

[0014] Extended reality (XR) includes virtual reality (VR), mixed reality (MR), and augmented reality (AR). Augmented reality is the extension of the real (e.g., physical) world with virtual content for processes such as room design, virtual shopping, tabletop AR games, turn-by-turn navigation assistance, food and health monitoring, AR video calls, and virtual meetings. This extension can be achieved using wearable AR devices that can map the physical world, orient themselves within that physical world, and place and render virtual content on a near-eye display visible to the user. Many such devices utilize hand and / or fingertip tracking to allow the user to control the interface in augmented reality. Thermal constraints are difficult due to the small form factor required for XR head-mounted displays (HMDs) (determined by, for example, fashion, comfort, etc.). Since HMD devices are typically worn on a person's head, HMDs also have weight and design constraints.

[0015] One approach to reducing power consumption in XR HMDs is to split the processing between the HMD and a host device such as a pack or mobile phone. A split XR system offloads some HMD computations to the host device. While a split XR system addresses the computational constraints of the HMD, it itself becomes subject to latency and workload considerations.

[0016] In a split design, several implementations perform rendering, which is a high-power workload, on a companion device. However, performing rendering on a companion device requires late reprojection on glasses (e.g., HMD) to reduce motion-to-photon latency and avoid user nausea. Late reprojection reprojects previously rendered frames for the most recent head pose. For example, if the user's head moves since rendering on the companion device, the headset warps the rendering based on the updated head pose. Perceptual workloads (e.g., hand tracking, head tracking, body tracking, three-dimensional (3D) reconstruction, etc.) can be handled on either glasses or the companion device, depending on latency and power requirements.

[0017] In segmented XR systems, it is often desirable to reduce the power consumption of the wireless local area network (WLAN) (e.g., Wi-Fi) on the HMD by utilizing the Target Wake Time (TWT) feature. TWT is a Wi-Fi 6 feature that allows agent traffic to be scheduled within a specific wake window and sleep for the rest of the time. The transmitter and receiver improve the power profile by agreeing on the time to wake up the radio and allowing the radio to sleep at other times.

[0018] If the host or companion device is a mobile phone, it has native workloads, such as home screen processing, running applications, and processing notifications, that must be scheduled and balanced in parallel with the rendering or perception workload of the XR HMD. If this balancing is not performed properly, the host's workload may cause the host to miss the TWT timeline. It would be desirable to improve processing in split XR devices.

[0019] Aspects of the present disclosure introduce TWT awareness workload scheduling and scaling for a computing engine on a host device. According to these aspects, host workloads are deferred until immediately after the TWT window. These host tasks may include processes such as gameplay, home screen processing, etc. on the host device.

[0020] In some aspects, the scheduler may elevate the corner for processing the host task, if necessary, to complete the host task such that the HMD task finishes in time for the next TWT window. If the host workload cannot be completed such that the HMD workload finishes in time for the next TWT window (even using the elevated corner), the host workload may be preempted and deferred until after the next TWT window.

[0021] Elevating the corner includes increasing the voltage of the processor core and / or increasing the clock frequency of the processing core. In these cases, the power penalty associated with the elevated corner may be considered acceptable. In some aspects, the lowest frequency and voltage corner that will enable completion within the time is determined. A margin (e.g., 5%) may be added to the estimate to account for uncertainty while ensuring completion of the host task. Hysteresis may be added to prevent the corner from changing frequently. For example, the time since the last change of the corner parameter may be determined. If the time is shorter than a threshold, the corner may be prevented from changing again.

[0022] Certain aspects of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages. In some examples, techniques described, such as prioritizing the processing of head-mounted display tasks in response to determining that a host device is within a target wake time (TWT) window, improve the power consumption of a split XR system. Additionally, a lower motion-to-photon latency for rendering is achieved.

[0023] FIG. 1 shows an exemplary implementation of a host system on-chip (SoC) 100 that includes a TWT recognition scheduler, according to an aspect of the present disclosure. The host SoC 100 includes processing blocks tailored to specific functions, such as a connectivity block 110. The connectivity block 110 can include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, WI-FI connectivity, universal serial bus (USB) connectivity, Bluetooth® connectivity, Secure Digital (SD) connectivity, and the like.

[0024] In this configuration, the host SoC 100 includes various processing units that support multithreaded operation. In the configuration shown in Figure 1, the host SoC 100 includes a multicore central processing unit (CPU) 102, a graphics processor unit (GPU) 104, a digital signal processor (DSP) 106, and a neural processor unit (NPU) 108. The host SoC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, a navigation module 120 which may include a global positioning system (GPS), and memory 118. The multicore CPU 102, GPU 104, DSP 106, NPU 108, and multimedia engine 112 support various functions such as video, audio, graphics, gaming, and artificial networking. Each processor core of the multicore CPU 102 could be a reduced instruction set computing (RISC) machine, an advanced RISC machine (ARM), a microprocessor, or some other type of processor. The NPU 108 could be based on the ARM instruction set.

[0025] According to aspects of this disclosure, the host device includes a prioritizing means, a delaying means, an evaluation means, and a processing means. In one configuration, the computing means may be a CPU, GPU, DSP, NPU, or ISP, as shown in Figure 1. In other aspects, the aforementioned means may be any structure or any material configured to perform the functions enumerated by the aforementioned means.

[0026] Figure 2 is a block diagram showing one embodiment of a segmented extended reality (XR) pipeline according to several aspects of the present disclosure. As shown in Figure 2, the segmented XR pipeline 200 may include a head-mounted display (HMD) 220 and a host device 240 (e.g., a smartphone) communicating with the HMD 220. The HMD communicates with the host device 240 via a communication network such as a Wi-Fi network. The HMD 220 may comprise a camera / inertia measurement unit (IMU) 222, a perception algorithm module 224, a segmented perception encoder module 226, a display processing unit (DPU) 228, a warp module 230, a decoder module 232, and a display panel 234. The camera / IMU 222 captures and records perception information from the environment and movement of the HMD 220. This perception information may be processed into head pose information by a perception algorithm in the perception algorithm module 224. In this embodiment, the perception algorithm module 224 incorporates a 6-degrees-of-freedom (DOF) algorithm for head pose tracking. The segmented perception encoder module 226 converts head pose information and / or perception information into digital or analog signals. The HMD 220 then transmits these signals to the host device 240 via a communication network such as a Wi-Fi 6 network.

[0027] The host device 240 receives transmissions from the HMD 220. The host device 240 may comprise a segmented perception decoder module 242, a tracking module 244, an encoder module 246, a rendering module 248, a software development kit (SDK) 250, and an AR application 252. The tracking module 244 incorporates hand tracking, plane detection, image tracking, and object tracking. The segmented perception decoder module 242 converts the transmissions into information to be used by the rendering module 248 or another module within the host device 240. The tracking module 244 uses the decoded information to perform hand tracking, plane detection, image tracking, and / or object tracking in relation to the HMD 220. The SDK 250 may comprise one or more software tools, libraries, and documentation for developing, testing, or integrating software applications on the host device 240 or another platform. The AR application 252 may comprise one or more device applications that facilitate augmented reality, extended reality, and / or virtual reality on the host device 240. The rendering module 248 can use a combination of information from the segmented perception decoder module 242 and / or tracking data from the tracking module 244, as well as other information from the host device 240 or HMD 220, to generate a 3D model or a visual representation of a scene for display purposes. The host device 240 can encode the rendering for return to the HMD 220 via the encoder module 246.

[0028] The HMD 220 receives a transmission from the host device 240 and decodes the transmission using the decoder module 232. This decoded transmission then generates rendering information that can be used by the warp module 230 and / or the DPU 228. The DPU 228 is a component for rendering and controlling the visual content displayed on the display panel 234. The warp module 230 is a component for applying geometric transformations to the rendering, such as correcting optical distortion, adjusting perspective, or mapping the rendering onto a non-flat surface. The warp module 230 and the DPU 228 can distort and / or manipulate the rendering information into a finished rendering using information from the perception algorithm module 224, the camera / IMU 222, and / or the host device 240. This finished rendering is displayed as virtual content on the display panel 234, which is a surface on which visual information can be displayed.

[0029] In segmented XR systems, it is often desirable to reduce power consumption for communication on the HMD by utilizing Target Wake Time (TWT). TWT is a Wi-Fi 6 feature that allows agent traffic to be scheduled within a specific wake window so that the device can sleep for the remainder of the time.

[0030] Figure 3 is a timing diagram showing one embodiment of a TWT timeline according to several aspects of the present disclosure. As shown in Figure 3, the TWT timeline 300 shows the actions performed by host devices 310 and HMD 330, which may be the same host devices 240 and HMD 220 in Figure 2. In the embodiment of Figure 3, host device 310 is a telephone companion and HMD 330 is an XR HMD. At the beginning of the period shown by the TWT timeline 300, host device 310 renders eye buffer information in XR eye buffer rendering stage 312 based on head pose information provided by host device 310 and / or HMD 330. This eye buffer information may include data necessary to display virtual content. The eye buffer information is then encoded by host device 310 in XR eye buffer encoding stage 314. During the XR eye buffer rendering stage 312 and the XR eye buffer encoding stage 314, host device 310 and HMD 330 communicate little to no with each other, as shown by sleep stage 322. Sleep stage 322 is sometimes referred to as the "sleep window."

[0031] After the host device 310 completes the XR eye buffer encoding stage 314, the host device 310 transmits information to the HMD 330 over the communication network, as indicated by the XR eye buffer transmission stage 324. The XR eye buffer transmission stage 324 is sometimes referred to as the "wake window". Upon receiving the information transmitted from the host device 310, the HMD 330 decodes the transmission in the XR eye buffer decoding stage 332. The HMD 330 then modifies the decoded information in the late-stage reprojection (LSR) block 334. In the late-stage reprojection block 334, the HMD 330 generates virtual content by reprojecting the 3D scene onto a 2D screen or image plane using one or more techniques. This virtual content is then displayed on the display panel in the display stage 336.

[0032] As shown in Figure 3, the stages in the TWT timeline 300 can repeat in cycles. For example, after the host device 310 transmits information during the XR eye buffer transmission stage 324, the host device 310 can immediately re-enter the XR eye buffer rendering stage 312. Furthermore, some stages in the TWT timeline 300 can overlap with other stages. The host device 310 and the HMD 330 can operate in parallel, with the host device 310 processing at one stage while the HMD 330 processes at another. This parallelism can also be extended to the devices themselves. The HMD 330 can process at one stage and simultaneously process at different stages. For example, the display stage 336 can occur in parallel with the late reprojection block 334. The host device 310 can also perform processing at one stage and simultaneously process at different stages. The HMD 330 and the host device 310 can further process their own stages during either the sleep stage 322 or the XR eye buffer transmission stage 324.

[0033] If the host or companion device is a mobile phone, the mobile phone has native workloads, such as a home screen, applications, and notifications, that must be scheduled in parallel with the rendering or perception workload of the XR HMD. If this balancing is not performed properly, the host workload may cause the host to miss processing according to the TWT timeline. Aspects of this disclosure introduce TWT-aware workload scheduling and scaling techniques for the compute engine on the host device. In some configurations, these techniques have several key aspects. For example, in these configurations, (1) the host workload may be delayed until immediately after a TWT window; (2) the scheduler may promote processing corners on the host workload if necessary to complete the host workload in time for the HMD workload to finish for the TWT window; and (3) if the host workload cannot be completed in time for the HMD workload to finish for the TWT window, the host workload may be preempted and delayed until after the next TWT window.

[0034] Figure 4 is a flowchart illustrating one embodiment of TWT-aware workload scheduling and scaling for a computing engine on a host, according to several embodiments of the present disclosure. As shown in Figure 4, the TWT-aware scheduler 400 performs processes to prioritize workloads. In some embodiments, the TWT-aware scheduler 400 may run on one or more agents on a companion device, including a graphics processing unit (GPU) 402, a neural signal processor (NSP) 404, and / or a computer vision accelerator (CVA) 406. While these components are shown in Figure 4, processing may also be performed on other components, such as those shown in Figure 1, e.g., a GPU 104, a multi-core CPU 102, and an NPU 108.

[0035] In block 410, the agent prioritizes the HMD workload during the TWT wake window. The process then proceeds to block 412. In block 412, the agent determines whether the TWT window has completed. If the TWT window has not completed (block 412: no), the agent continues to prioritize the HMD workload in block 410. If the TWT window has completed (block 412: yes), the process proceeds to block 414. In block 414, the agent prioritizes the host workload. The process then proceeds to block 416. In block 416, the agent determines whether the host workload will complete in time for the HMD workload to finish within the TWT window. If the host workload will complete in time for the HMD workload to finish within the TWT window (block 416: yes), the agent continues to prioritize the host workload in block 414. If, in block 416, it is determined that the host workload will not complete in time for the HMD workload to finish within the TWT window (block 416: No), the process proceeds to block 418.

[0036] In block 418, the agent determines whether promoting one or more corners would allow the agent to complete the host workload while outside the TWT window. If block 418 determines that promoting one or more corners would allow the agent to complete the host workload while outside the TWT window (block 418: yes), the agent may then, in block 420, promote one or more corners to complete the host workload, and then, in block 424, prioritize the HMD workload. For example, the agent may increase the clock frequency of the device processing core to the minimum voltage and frequency required to complete the host workload while outside the TWT window. In another embodiment, the agent may determine the minimum voltage and frequency required to complete the host workload while outside the TWT window, and then, for additional certainty, increase the voltage and / or frequency to an amount higher than the required minimum voltage and frequency.

[0037] In block 418, if it is determined that promoting the corner would prevent the agent from completing the host workload while outside the TWT window, or if promoting the corner is not feasible or otherwise not the ideal solution (block 418: No), the agent may preempt the host workload in block 422 and then prioritize the HMD workload in block 424. One example of a situation where promoting the corner is not the desired solution is if the agent has recently promoted the corner or otherwise modified the corner. The process then proceeds to block 410, where the agent prioritizes the HMD workload while within the TWT window. The agent may then repeat the TWT-aware scheduler process in a cycle.

[0038] Figure 5 is a flowchart illustrating exemplary process 500, performed, for example, by a host device, according to various aspects of the present disclosure. Exemplary process 500 is an example of TWT-aware scheduling. As shown in Figure 5, in some embodiments, process 500 may include prioritizing the processing of a first head-mounted display (HMD) task in response to the host device determining that it is within a target wake time (TWT) window (block 502). Examples of head-mounted display tasks include late reprojection (LSR) and perception. In some embodiments, process 500 may optionally include prioritizing the processing of a host task in response to the host device determining that it is outside the TWT window (block 504). Examples of host tasks include rendering and object tracking.

[0039] In some embodiments, process 500 may perform one or more actions to prioritize the HMD task during a TWT window and the host task when outside of a TWT window. For example, the host device may evaluate whether the host task can be completed in time by increasing the processing core voltage and / or the processing core clock frequency to allow sufficient time for the second HMD task to complete processing before the next TWT window ends. The host device may also process the host task using the increased processing core voltage and / or the increased processing core clock frequency, depending on whether the host task can be completed in time. In some embodiments, the host device may delay processing the host task depending on whether the host task can not be completed in time using the increased processing core voltage and / or the increased processing core clock frequency. In other embodiments, the host device may delay processing the host task until after the next TWT window has ended or until the second HMD task has completed. In a further embodiment, the host device may delay the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second HMD task to complete before the next TWT window ends, until the second HMD task is completed.

[0040] Figure 6 is a block diagram showing an exemplary wireless communication system 600 that may favorably adopt one aspect of the present disclosure. For illustrative purposes, Figure 6 shows three remote units 620, 630, and 650, and two base stations 640. It will be recognized that the wireless communication system may have more remote units and base stations. The remote units 620, 630, and 650 include integrated circuit (IC) devices 625A, 625B, and 625C, which include the disclosed TWT-aware scheduler. It will be recognized that other devices, such as base stations, switching devices, and network equipment, may also include the disclosed TWT-aware scheduler. Figure 6 shows a forward link signal 680 from base station 640 to remote units 620, 630, and 650, and a reverse link signal 690 from remote units 620, 630, and 650 to base station 640.

[0041] In Figure 6, remote unit 620 is shown as a mobile phone, remote unit 630 as a portable computer, and remote unit 650 as a fixed-location remote unit in a wireless local loop system. For example, a remote unit may be a portable data unit such as a mobile phone, a handheld personal communication system (PCS) unit, or a personal information terminal; a fixed data unit such as a GPS-enabled device, a navigation device, a set-top box, a music player, a video player, an entertainment unit, or a meter reading device; or another device that stores or retrieves data or computer instructions, or a combination thereof. While Figure 6 illustrates remote units according to the embodiments of this disclosure, the disclosure is not limited to these exemplary units. Embodiments of this disclosure can be appropriately adopted in many devices, including the disclosed TWT-aware scheduler.

[0042] Figure 7 is a block diagram showing a design workstation 700 used for circuit design, layout design, and logic design of semiconductor components, such as the TWT-aware scheduler disclosed above. The design workstation 700 includes a hard disk 701 containing operating system software, support files, and design software such as Cadence or OrCAD. The design workstation 700 also includes a display 702 to facilitate the design of the circuit 710 or semiconductor component 712, such as the TWT-aware scheduler. A storage medium 704 is provided for tangibly storing the design of the circuit 710 or semiconductor component 712 (e.g., PLD). The design of the circuit 710 or semiconductor component 712 may be stored on the storage medium 704 in a file format such as GDSII or GERBER. The storage medium 704 may be a CD-ROM, DVD, hard disk, flash memory, or other suitable device. Furthermore, the design workstation 700 includes a drive device 703 for receiving input from the storage medium 704 or writing output to the storage medium 704.

[0043] The data recorded on the storage medium 704 may specify logic circuit configurations, pattern data for photolithography masks, or mask pattern data for continuous drawing tools such as electron beam lithography. The data may further include logic verification data, such as timing diagrams or net circuits associated with logic simulations. By providing data on the storage medium 704, the design of the circuit 710 or semiconductor component 712 is facilitated by reducing the number of processes required to design the semiconductor wafer.

[0044] Exemplary aspects Embodiment 1: A method for scheduling workloads by a host device communicating with a head-mounted display, comprising prioritizing the processing of a first head-mounted display (HMD) task in response to the host device determining that it is within a target wake time (TWT) window.

[0045] Embodiment 2: The method according to Embodiment 1, further comprising prioritizing the processing of a host task in response to determining that the host device is outside the TWT window.

[0046] Embodiment 3: The method according to Embodiment 1 or 2, further comprising evaluating whether a host task can be completed in time in order to allow sufficient time for a second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

[0047] Embodiment 4: The method according to any one of Embodiments 1 to 3, further comprising processing a host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed within a time limit.

[0048] Embodiment 5: The method according to any one of Embodiments 1 to 4, further comprising delaying the processing of a host task in response to the fact that the host task cannot be completed in time using an increased processing core voltage and / or an increased processing core clock frequency.

[0049] Embodiment 6: The method according to any one of Embodiments 1 to 5, wherein the delay is until after the next TWT window has finished or until the second HMD task has completed.

[0050] Embodiment 7: The method according to any one of Embodiments 1 to 6, further comprising delaying the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second HMD task is completed.

[0051] Embodiment 8: A device for workload scheduling by a host device communicating with a head-mounted display, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to prioritize processing a first head-mounted display (HMD) task in response to the host device determining that it is within a target wake time (TWT) window.

[0052] Embodiment 9: The apparatus according to Embodiment 8, wherein at least one processor is further configured to prioritize processing a host task in response to determining that the host device is outside the TWT window.

[0053] Embodiment 10: The apparatus according to Embodiment 8 or 9, wherein at least one processor is further configured to evaluate whether a host task can be completed in time in order to allow sufficient time for a second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

[0054] Embodiment 11: The apparatus according to any one of embodiments 8 to 10, wherein at least one processor is further configured to process a host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed in a timely manner.

[0055] Embodiment 12: The apparatus according to any one of embodiments 8 to 11, wherein at least one processor is further configured to delay the processing of a host task in response to the fact that the host task cannot be completed in time using an increased processing core voltage and / or an increased processing core clock frequency.

[0056] Embodiment 13: The apparatus according to any one of embodiments 8 to 12, wherein at least one processor configured to delay processing is further configured to delay until after the next TWT window has finished or until the second HMD task has completed.

[0057] Embodiment 14: The apparatus according to any one of embodiments 8 to 13, wherein at least one processor is further configured to delay the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second HMD task is completed.

[0058] Embodiment 15: A non-temporary computer-readable medium recording program code, wherein the program code includes program code that is executed by a processor and determines that the host device is within a target wake time (TWT) window, and which prioritizes the processing of a first head-mounted display (HMD) task.

[0059] Embodiment 16: The non-temporary computer-readable medium according to Embodiment 15, further comprising program code for prioritizing the processing of a host task in response to the host device determining that it is outside the TWT window.

[0060] Embodiment 17: A non-temporary computer-readable medium according to Embodiment 15 or 16, further comprising program code for evaluating whether a host task can be completed in time in order to allow sufficient time for a second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

[0061] Embodiment 18: A non-temporary computer-readable medium according to any one of embodiments 15 to 17, further comprising program code for processing a host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed in a timely manner.

[0062] Embodiment 19: A non-temporary computer-readable medium according to any one of embodiments 15 to 18, further comprising program code for delaying the processing of a host task in response to the host task being unable to complete in time using an increased processing core voltage and / or an increased processing core clock frequency.

[0063] Embodiment 20: A non-temporary computer-readable medium according to any one of embodiments 15 to 19, wherein the program code for delaying processing includes program code for delaying processing until after the next TWT window has finished or until the second HMD task has completed.

[0064] Embodiment 21: A non-temporary computer-readable medium according to any one of Embodiments 15 to 20, further comprising program code for delaying the processing of a host task in response to the host task determining that it cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second head-mounted display (HMD) task is completed.

[0065] Embodiment 22: An apparatus for workload scheduling by a host device communicating with a head-mounted display, comprising: means for prioritizing the processing of a first head-mounted display (HMD) task in response to the host device determining that it is within a target wake time (TWT) window; and means for prioritizing the processing of a host task in response to the host device determining that it is outside the TWT window.

[0066] Embodiment 23: The apparatus according to Embodiment 22, further comprising means for evaluating whether a host task can be completed in time in order to allow sufficient time for a second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

[0067] Embodiment 24: The apparatus according to Embodiment 22 or 23, further comprising means for processing a host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed within a time frame.

[0068] Embodiment 25: The apparatus according to any one of Embodiments 22 to 24, further comprising means for delaying the processing of a host task in cases where the host task cannot be completed in time using an increased processing core voltage and / or an increased processing core clock frequency.

[0069] Embodiment 26: The apparatus according to any one of Embodiments 22 to 25, wherein the means for delaying processing further comprises means for delaying until after the next TWT window has finished or until the second HMD task has been completed.

[0070] Embodiment 27: The apparatus according to any one of Embodiments 22 to 26, further comprising means for delaying the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second HMD task is completed.

[0071] In the case of firmware and / or software implementations, these methodologies may be implemented using modules (e.g., procedures, functions, etc.) that perform the described functions. When implementing the described methodologies, machine-readable media may be used to tangibly embody the instructions. For example, software code may be stored in memory and executed by a processor unit. Memory may be implemented inside or outside the processor unit. When used, the term “memory” refers to long-term memory, short-term memory, volatile memory, non-volatile memory, or other types of memory, and is not limited to a specific type of memory or number of memories, or the type of medium in which the memory is stored.

[0072] Where implemented in firmware and / or software, these functions may be stored as one or more instructions or codes on a computer-readable medium. Examples include computer-readable media encoded using data structures and computer-readable media encoded using computer programs. Computer-readable media include physical computer storage media. Storage media can be any available medium accessible by a computer. Examples, but not limited to, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disc storage or other magnetic storage devices, or other media that can be used to store desired program code in the form of instructions or data structures and are accessible by a computer. When used, "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray® discs. A disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. Any combination of these should also be included within the scope of computer-readable media.

[0073] In addition to storage on computer-readable media, instructions and / or data may be provided as signals on a transmission medium contained within a communication device. For example, the communication device may include transceivers having signals representing instructions and data. These instructions and data are configured to cause one or more processors to perform the functions outlined in the claims.

[0074] While the Disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the technology of the Disclosure as defined by the attached claims. For example, correlated terms such as “up” and “down” are used with respect to a substrate or electronic device. Naturally, if the substrate or electronic device is inverted, up becomes down, and vice versa. Furthermore, if it is oriented sideways, up and down may refer to the sides of the substrate or electronic device. Moreover, the scope of the Disclosure is not intended to be limited to any specific configuration of the processes, machines, manufactures, compositions, means, methods, and steps described herein. It will be readily apparent to those skilled in the art that existing or subsequently developed processes, machines, manufactures, compositions, means, methods, or steps that perform substantially the same function or achieve substantially the same results as the corresponding configurations described herein can be utilized in accordance with the Disclosure. Accordingly, the attached claims are intended to include such processes, machines, manufactures, compositions, means, methods, or steps within their scope.

[0075] Those skilled in the art will further understand that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with this disclosure can be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and steps have been outlined above in terms of their functions. Whether such functions are implemented as hardware or executed as software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

[0076] The various exemplary logic blocks, modules, and circuits described in connection with this disclosure may be implemented or run using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, individual gate logic or transistor logic, individual hardware components, or any combination thereof, designed to perform the functions described. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0077] Steps of the methods or algorithms described in this disclosure may be embodied directly in hardware, in software modules executed by a processor, or in a combination of the two. The software modules may reside in RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and storage medium may reside in the user terminal as separate components.

[0078] The preceding explanations in this disclosure are provided to enable any person skilled in the art to implement or use the disclosure. Various modifications to this disclosure will be readily apparent to a person skilled in the art, and the general principles defined may be applied to other variations without departing from the spirit or scope of this disclosure. Accordingly, this disclosure should be given the broadest scope that is consistent with the disclosed principles and novel features, and not limited to the examples and designs described.

Claims

1. A method for scheduling workloads by a host device communicating with a head-mounted display, the method comprising prioritizing the processing of a first head-mounted display (HMD) task in response to the host device determining that it is within a target wake time (TWT) window.

2. The method according to claim 1, further comprising prioritizing the processing of a host task in response to determining that the host device is outside the TWT window.

3. The method according to claim 2, further comprising evaluating whether the host task can be completed in time to allow sufficient time for the second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

4. The method according to claim 3, further comprising processing the host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed within a time limit.

5. The method according to claim 3, further comprising delaying the processing of the host task in cases where the host task is unable to be completed in time using the increased processing core voltage and / or increased processing core clock frequency.

6. The method according to claim 5, wherein the delay lasts until after the next TWT window has finished or until the second HMD task has completed.

7. The method according to claim 1, further comprising delaying the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second HMD task is completed.

8. A device for workload scheduling by a host device that communicates with a head-mounted display, At least one memory, At least one processor coupled to the at least one memory, The system comprises, wherein at least one processor is configured to prioritize processing a first head-mounted display (HMD) task in response to determining that the host device is within a target wake time (TWT) window. Device.

9. The apparatus according to claim 8, wherein the at least one processor is further configured to prioritize the processing of a host task in response to determining that the host device is outside the TWT window.

10. The apparatus according to claim 9, wherein the at least one processor is further configured to evaluate whether the host task can be completed in time so as to allow sufficient time for the second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

11. The apparatus according to claim 10, wherein the at least one processor is further configured to process the host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed in a timely manner.

12. The apparatus according to claim 10, wherein the at least one processor is further configured to delay the processing of the host task in cases where the host task cannot be completed in time using the increased processing core voltage and / or increased processing core clock frequency.

13. The apparatus according to claim 12, wherein the at least one processor configured to delay processing is further configured to delay until after the next TWT window has finished or until the second HMD task has completed.

14. The apparatus according to claim 8, wherein the at least one processor is further configured to delay the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second HMD task is completed.

15. A non-temporary computer-readable medium recording program code, wherein the program code includes program code that is executed by a processor and prioritizes the processing of a first head-mounted display (HMD) task in response to the determination that the host device is within a target wake time (TWT) window.

16. The non-temporary computer-readable medium according to claim 15, further comprising program code for prioritizing the processing of a host task in response to the determination that the host device is outside the TWT window.

17. The non-temporary computer-readable medium according to claim 16, further comprising program code for evaluating whether the host task can be completed in time so as to allow sufficient time for the second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

18. The non-temporary computer-readable medium according to claim 17, further comprising program code for processing the host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed within a time frame.

19. The non-temporary computer-readable medium according to claim 17, further comprising program code for delaying the processing of the host task in response to the fact that the host task is unable to complete in time using the increased processing core voltage and / or increased processing core clock frequency.

20. The non-temporary computer-readable medium according to claim 19, wherein the program code for delaying processing includes program code for delaying until after the next TWT window has closed or until the second HMD task has completed.

21. The non-temporary computer-readable medium according to claim 15, further comprising program code for delaying the processing of a host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second head-mounted display (HMD) task is completed.

22. A device for workload scheduling by a host device that communicates with a head-mounted display, In response to determining that the host device is within the target wake time (TWT) window, means for prioritizing the processing of the first head-mounted display (HMD) task, In response to determining that the host device is outside the TWT window, means for prioritizing the processing of the host task, A device equipped with the following features.

23. The apparatus according to claim 22, further comprising means for evaluating whether the host task can be completed in time to allow sufficient time for the second head-mounted display (HMD) task to complete processing before the next TWT window ends, by increasing the processing core voltage and / or increasing the processing core clock frequency.

24. The apparatus according to claim 23, further comprising means for processing the host task using an increased processing core voltage and / or an increased processing core clock frequency, depending on whether the host task can be completed within a time limit.

25. The apparatus according to claim 23, further comprising means for delaying the processing of the host task in cases where the host task cannot be completed in time using the increased processing core voltage and / or increased processing core clock frequency.

26. The apparatus according to claim 25, wherein the means for delaying processing further comprises means for delaying until after the next TWT window has finished or until the second HMD task has been completed.

27. The apparatus according to claim 22, further comprising means for delaying the processing of the host task in response to determining that the host task cannot be completed in time, in order to allow sufficient time for the second head-mounted display (HMD) task to complete before the next TWT window ends, until the second HMD task is completed.

Images