Reducing Latency in Wireless Virtual and Augmented Reality Systems

Abstract

Systems, devices, and methods for reducing latency for wireless virtual reality and augmented reality applications are disclosed. A virtual reality (VR) or augmented reality (AR) system includes a transmitter that renders, encodes, and transmits video frames to a receiver coupled to a head-mounted display (HMD). In one scenario, rather than waiting until the entire frame is encoded before transmitting the frame to the receiver, the transmitter transmits the encoded left-eye portion to the receiver while the right-eye portion is being encoded. In another scenario, a frame is divided into multiple slices, and each slice is encoded and then transmitted to the receiver while the next slice is being encoded. In yet another scenario, each slice is encoded while the next slice is being rendered. In yet another scenario, each slice is prepared for display by the receiver while the next slice is being decoded by the receiver.

Description

[Background technology]

[0001] Description of Related Art To create an immersive environment for the user, virtual reality (VR) and artificial reality (AR) video streaming applications typically require high resolution and high frame rates, which equate to high data rates. Standard video codecs such as H.264 and High Efficiency Video Coding (HEVC) are commonly used to encode the video frames that are rendered as part of a VR or AR application. As the resolution and refresh rate of VR or AR displays increase, the latency required to render, encode, transmit, decode, and prepare frames for display becomes a significant limiting factor.

[0002] The additional elements of wireless VR and AR systems introduce additional latency. This latency consists of the relatively constant latency of the encoder and decoder operations, and the variable latency of wireless transmission. This delay, which is not present in traditional wired VR and AR systems, can have a significant impact, such as rendering frames being displayed to the client with a delay from the time each head pose is acquired. Unless special measures are taken to reduce these delays, the images displayed in the headset will lag behind head movements, breaking the immersion and causing nausea and eye strain.

[0003] The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of one embodiment of a system. [Figure 2] FIG. 1 is a block diagram of one embodiment of a system. [Figure 3] FIG. 1 is a timing diagram of one embodiment of a conventional wireless VR / AR system. [Figure 4]FIG. 1 is a timing diagram of an example of a wireless VR / AR system with reduced latency. [Figure 5] FIG. 10 is a timing diagram of another example of a technique for reducing latency in a wireless VR / AR system. [Figure 6] FIG. 10 is a timing diagram of further latency reduction enhancements for wireless VR / AR. [Figure 7] FIG. 1 is a timing diagram of a further enhancement to the wireless VR / AR system to reduce latency. [Figure 8] FIG. 10 is a timing diagram of another enhancement to a wireless VR / AR system to reduce latency. [Figure 9] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for reducing latency in a wireless VR / AR system. [Figure 10] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for reducing latency in a wireless VR / AR system. [Figure 11] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for signaling the end of a slice between units. [Figure 12] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for adjusting slice granularity by a unit of a wireless VR / AR system. DETAILED DESCRIPTION OF THE INVENTION

[0005] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It should be understood that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements.

[0006] Various systems, devices, methods, and computer-readable media for reducing latency in wireless virtual reality and augmented reality applications are disclosed herein. In one embodiment, a virtual reality (VR) or augmented reality (AR) system includes a transmitter that renders, encodes, and transmits video frames to a receiver coupled to a head-mounted display (HMD). In one scenario, rather than waiting until the entire frame is encoded before transmitting the frame to the receiver, the transmitter transmits the left-eye portion to the receiver while the right-eye portion is being encoded. In another scenario, a frame is divided into multiple slices, and each slice is encoded and then transmitted to the receiver while the next slice is being encoded. For example, the transmitter transmits an encoded version of a first slice to the receiver before receiving the entire encoded version of the second slice from the encoder. In a further scenario, each slice is encoded while the next slice is being rendered. In yet another scenario, each slice is prepared for display by the receiver while the next slice is being decoded by the receiver.

[0007] Referring to FIG. 1 , a block diagram of one embodiment of a system 100 is shown. In one embodiment, the system 100 includes a transmitter 105, a channel 110, a receiver 115, and a head-mounted display (HMD) 120. It should be noted that in other embodiments, the system 100 may include components other than those shown in FIG. 1 . In one embodiment, the channel 110 is a wireless connection between the transmitter 105 and the receiver 115. In another embodiment, the channel 110 represents a network connection between the transmitter 105 and the receiver 115. Any type and number of networks may be employed to provide the connection between the transmitter 105 and the receiver 115, depending on the embodiment. For example, the transmitter 105 is part of a cloud service provider in one particular embodiment.

[0008] In one embodiment, the transmitter 105 receives a video sequence that is encoded and transmitted to the receiver 115. In another embodiment, the transmitter 105 includes a rendering unit that renders the video sequence that is encoded and transmitted to the receiver 115. In one embodiment, the rendering unit generates a rendered image from graphics information (e.g., raw image data). Note that the terms "image," "frame," and "video frame" can be used interchangeably herein.

[0009] To reduce the latency of transmitted data, various techniques for slice-based processing of frames are used in transmitter 105 and / or receiver 115. In one embodiment, as receiver 115 receives each slice of a frame, receiver 115 decodes the slice in parallel and prepares it for display. Once a complete frame is prepared, receiver 115 sends the frame to HMD 120. In one embodiment, for each image displayed on HMD 120, the right-eye portion of the image is sent to the right side 125R of HMD 120, while the left-eye portion of the image is sent to the left side 125L of HMD 120. In one embodiment, receiver 115 is separate from HMD 120, and receiver 115 communicates with HMD 120 using a wired or wireless connection. In another embodiment, receiver 115 is integrated within HMD 120.

[0010] Transmitter 105 and receiver 115 represent any type of communication and / or computing device. For example, in various embodiments, transmitter 105 and / or receiver 115 may be a mobile phone, a tablet, a computer, a server, an HMD, another type of display, a router, or other type of computing or communication device. In one embodiment, system 100 executes a virtual reality (VR) application for wirelessly transmitting frames of a rendered virtual environment from transmitter 105 to receiver 115. In other embodiments, other types of applications (e.g., augmented reality (AR) applications) may be executed by system 100 utilizing the methods and mechanisms described herein.

[0011] In one embodiment, the position of the user's head is determined based on one or more head tracking sensors 140 in the HMD 120. In one embodiment, the head position information is transmitted from the head tracking sensor(s) 140 to the transmitter 105 using a communication interface 145. In one embodiment, the head position information is provided to a rendering source of the VR / AR video in the transmitter 105. In one embodiment, the rendering unit uses the head position information when rendering the next frame to be displayed to the user.

[0012] Referring to FIG. 2, a block diagram of one embodiment of a system 200 is shown. The system 200 includes at least a first communication device (e.g., a transmitter 205) and a second communication device (e.g., a receiver 210) operable to wirelessly communicate with each other. Note that the transmitter 205 and the receiver 210 may be referred to as transceivers. In one embodiment, the transmitter 205 and the receiver 210 communicate wirelessly over the unlicensed 60 gigahertz (GHz) frequency band. For example, in this embodiment, the transmitter 205 and the receiver 210 communicate in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 802.11ad standard (i.e., WiGig). In other embodiments, the transmitter 205 and the receiver 210 communicate wirelessly over other frequency bands and / or by adhering to other wireless communication protocols, whether or not they comply with a standard. For example, other wireless communication protocols that may be used include, but are not limited to, Bluetooth®, protocols used in various wireless local area networks (WLANs), WLANs based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard (i.e., WiFi®), mobile communication standards (e.g., CDMA, LTE, GSM, WiMAX), etc.

[0013] Transmitter 205 and receiver 210 represent any type of communication and / or computing device. For example, in various embodiments, transmitter 205 and / or receiver 210 may be a mobile phone, a tablet, a computer, a server, a head-mounted display (HMD), a television, another type of display, a router, or any other type of computing or communication device. In one embodiment, system 200 executes a virtual reality (VR) application for wirelessly transmitting frames of a rendered virtual environment from transmitter 205 to receiver 210. In other embodiments, other types of applications can be executed by system 200 utilizing the methods and mechanisms described herein.

[0014] In one embodiment, transmitter 205 includes at least a radio frequency (RF) transceiver module 225, a processor 230, a memory 235, and an antenna 240. RF transceiver module 225 transmits and receives RF signals. In one embodiment, RF transceiver module 225 is a millimeter wave transceiver module operable to wirelessly transmit and receive signals over one or more channels in the 60 GHz band. RF transceiver module 225 converts baseband signals to RF signals for wireless transmission, and RF transceiver module 225 converts the RF signals to baseband signals for data extraction by transmitter 205. Note that RF transceiver module 225 is shown as a single unit for illustrative purposes. It should be understood that RF transceiver module 225 may be implemented in any number of different units (e.g., chips) depending on the embodiment. Similarly, processor 230 and memory 235 represent any number and types of processors and memory devices, respectively, implemented as part of transmitter 205. In one embodiment, processor 230 includes a rendering unit 231 that renders frames of the video stream and an encoder 232 that encodes (i.e., compresses) the video stream before transmitting it to receiver 210. In other embodiments, rendering unit 231 and / or encoder 232 are implemented separately from processor 230. In various embodiments, rendering unit 231 and encoder 232 are implemented using any suitable combination of hardware and / or software.

[0015] The transmitter 205 also includes an antenna 240 for transmitting and receiving RF signals. The antenna 240 represents one or more antennas, such as a phased array, a single-element antenna, or a set of switched-beam antennas, that can be configured to change the directionality of the transmission and reception of radio signals. As an example, the antenna 240 includes one or more antenna arrays, where the amplitude or phase for each antenna in the antenna array can be configured independently from other antennas in the array. While the antenna 240 is shown as being external to the transmitter 205, it should be understood that the antenna 240 can be included internally within the transmitter 205 in various embodiments. It should also be understood that the transmitter 205 can include any number of other components not shown to avoid obscuring the illustration. Similar to the transmitter 205, the components embodied within the receiver 210 include at least an RF transceiver module 245, a processor 250, a decoder 252, a memory 255, and an antenna 260, which are similar to the components described above for the transmitter 205. It should also be understood that receiver 210 may include or be coupled to other components (eg, a display).

[0016] Referring to Figure 3, a timing diagram of one embodiment of a conventional wireless VR / AR system is shown. In one embodiment, the wireless VR / AR system includes a computing device 302 coupled to a mobile device 304 having a head-mounted display (HMD) for displaying VR / AR video to a user. In one embodiment, the computing device 302 represents transmitter 105 (of Figure 1), and the mobile device 304 represents receiver 115. In some embodiments, the computing device 302 can be any type of computing device (e.g., a personal computer (PC)).

[0017] Each row in the timing diagram for computing device 302 corresponds to a different component of the computing device, including a CPU, a graphics engine, and an encoder, shown in Figure 3. Similarly, each row in the timing diagram for mobile device 304 corresponds to a different component of mobile device 304. These different components include a CPU, a decoder, and a graphics engine. In other embodiments, computing device 302 and mobile device 304 may include other components that perform the steps shown in Figure 3. In one embodiment, computer 302 renders VR / AR frames, then encodes the frames, and then transmits the encoded frames over a wireless network to be decoded and displayed by mobile device 304.

[0018] In one embodiment, the mobile device 304 transmits the user's head pose 310 to the computing device 302 over a wireless network. In one embodiment, the term "head pose" is defined as both the head position (e.g., X, Y, Z coordinates in three-dimensional space) and the head orientation. The head orientation may be specified as a quaternion, as a set of three angles called Euler angles, or in other ways. It should be understood that the head pose 310 may also include an indication of the user's eye or gaze direction, acceleration, and other parameters in addition to the user's head orientation. The computing device 302 queues the head pose 310 and waits 315 until rendering of the next frame begins. When the graphics engine is ready to begin rendering the next frame, it uses the head pose received from 304 to perform rendering for the game (or another application in another embodiment) and then sends the rendered frame to the compositor. This is represented by the rendering and compositing block 320 in FIG. 3. In one embodiment, after rendering and compositing, a color space conversion (CSC) 325 is performed to convert the rendered frame from the RGB color space to the YUV color space.

[0019] The output of the CSC 325 is then provided to an encoder, which performs encoding 330 of the rendered, color-space-converted frame. The output of the encoding step 330 is a left-eye and right-eye portion of a frame 335, which is transmitted over the network to the mobile device 304. Upon receiving the encoded frame, the mobile device 304 performs a decoding step 340 to decode the encoded frame. The mobile device 304 then enters a wait mode 345 until the frame is ready to be sent to the display. Next, the CSC 350 is performed to convert the frame back to RGB space. A warping step 355 is then performed to time-warp the frame for display based on the user's updated head pose. The frame is then sent to the display at the next vertical synchronization (VSync) signal 360. In one embodiment, the boundaries of each frame period are determined by the VSync signal. In other words, the start of a frame period coincides with a VSync signal, and the end of a frame period coincides with the next VSync signal. As used herein, the term "frame period" is defined as the inverse of the frame rate.

[0020] As can be seen from the multiple steps of the rendering, encoding, and decoding processes, the latency experienced by a typical VR / AR system can be relatively long. This is caused by each step of the process waiting for the previous step to finish for an entire frame before starting. Therefore, techniques for reducing the latency experienced by a typical VR / AR system are desirable.

[0021] Referring to Figure 4, a timing diagram of an example of a reduced latency wireless VR / AR system is shown. In one embodiment, the wireless VR / AR system includes a computing device 402 and a mobile device 404 with an HMD. Note that computing device 402 represents any type of computing device (e.g., laptop, desktop, server). Similar to the timing diagram shown in Figure 3, the first step in the process is to capture the user's current head pose 410 and communicate an indication of the current head pose 410 from mobile device 402 to computing device 404 over a network. Computing device 404 then performs a wait phase 415 while it waits to begin rendering the next frame.

[0022] A rendering phase 420 occurs after a wait phase 415, after which a CSC phase 425 follows the rendering phase 420, converting the rendered frame from RGB color space to YUV color space. Using a technique to reduce latency, the encoder then splits the encoding of the rendered frame into two steps: a left-eye portion encoding step 430 and a right-eye portion encoding step 435. Once the left-eye portion encoding 430 is complete, the encoded left-eye portion is transmitted over the network in step 440 in parallel with the right-eye portion being encoded in step 435. This helps reduce latency compared to the timing diagram shown in FIG. 3, in which the computing device 302 waits until the entire frame is encoded before beginning to transmit the encoded frame over the network to the mobile device 304.

[0023] The right eye portion is encoded in step 435 while the encoded left eye portion is transmitted over the network in step 440. Then, once the encoding of the right eye portion is complete and the encoded left eye portion has been transmitted to the mobile device 404, the encoded right eye portion is transmitted over the network to the mobile device 404 in step 445. While the right eye portion is transmitted over the network in step 445, the decoder decodes the left eye portion in step 450. Note that steps 445 and 450 are performed in parallel to reduce the overall latency of the frame processing steps.

[0024] After the left-eye portion of the frame is decoded in step 450, the right-eye portion of the frame is decoded in step 455. Next, the mobile device 404 performs a wait phase 460, the duration of which depends on the frame refresh rate. After wait phase 460, a CSC step 465 is performed, followed by a warp step 470 that completes in time for VSync 475. The parallel step execution for the left-eye and right-eye portions of the frame shown in the timing diagram of FIG. 4 helps reduce the overall latency of operation for a wireless VR / AR system.

[0025] 5, a timing diagram of another example technique for reducing latency in a wireless VR / AR system is shown. In the first step of the process, a mobile device 504 captures a user's head pose 510 and sends an indication of the head pose 510 to a computing device 502, where it is queued while the computing device 502 waits 515 for the start of the next frame according to the video refresh rate.

[0026] Next, frame rendering and compositing 520 is performed, followed by CSC step 525. In an enhancement designed to reduce system latency, the encoder then divides the frame into multiple slices 530 and encodes each slice separately. As the slice is encoded, the encoder transmits each slice over the network to the mobile device 504. While the first slice is being transmitted over the network, the second slice is encoded in parallel. This process continues, with a given slice being encoded while the previous slice is being transmitted over the network. A frame can be divided into any number of slices, with the number of slices varying depending on the embodiment.

[0027] Once all of the coded slices 535 for the left eye portion have been transmitted over the network to the mobile device 504, the decoder begins decoding the left eye portion at step 545. Then, once all of the coded slices 540 for the right eye portion have been transmitted over the network and received by the mobile device 504, the decoder performs a decoding step 550 for the right eye portion. Once the decoder has finished decoding step 550 for the right eye portion, the mobile device 504 transitions to a wait mode 560 according to the refresh rate of the display (e.g., HMD). A CSC step 565 is then performed, and a warp step 570 is performed to display the frame in time for the next VSync 575. By dividing a frame into multiple slices 530 and encoding and transmitting these slices 530 in parallel, system latency is reduced.

[0028] Referring to Figure 6, a timing diagram of further latency reduction enhancements for wireless VR / AR is shown. At the start of a cycle, a mobile device 604 captures a user's head pose 610 and transfers it over the network to a computing device 602, which queues the head pose 610 and then waits 615. Next, a graphics engine renders and composites the frame in step 620, followed by a CSC step 625. An encoder then encodes individual slices 630 of the frame and transmits the encoded slices over the network in step 635.

[0029] Next, as an enhancement to the previous timing diagram (of FIG. 5), the decoder of the mobile device 604 decodes individual slices 640 as they are received, rather than waiting for the entire eye of the encoded frame to be received. This allows the decoder to begin decoding the first slice as soon as it is received in its entirety, and the decoding of the first slice can occur in parallel with the transmission of the second slice over the network. Then, once the second slice is completely received, the decoding of the second slice occurs in parallel with the transmission of the third slice over the network. This process continues for each slice transmitted over the network. The above process helps reduce the latency of the decoding step. The decoding step is followed by a wait phase 645, a CSC step 650, and a warp step 655. The frame is displayed after the warp step 655 in time for the next VSync 660.

[0030] Referring to Figure 7, a timing diagram of a further enhancement to a wireless VR / AR system for reducing latency is shown. While the initial steps of capturing a head pose 710 and waiting 715 at the computing device 702 after receiving the head pose 710 remain the same, the rendering step 720 is now performed on slices of the frame rather than the entire frame. This allows each slice of the frame to be rendered independently of the other slices, and once the first slice of the frame is rendered, the encoder encodes the first slice while the second slice is being rendered. This process continues for the other slices being rendered while slice 725 is being encoded in parallel.

[0031] Then, once a given slice has been encoded, it is transmitted over the network to the mobile device 704 in step 730. As an enhancement to the prior art, steps 720, 725, and 730 for rendering, encoding, and transmitting over the network are each performed in parallel. Also, as each slice is received by the mobile device 704, it is decoded in step 735. Note that step 735 is performed in parallel with steps 720, 725, and 730. After the decoding step 735, the mobile device 704 waits 740 before performing a CSC step 745 and a warp step 750. Following the warp step 750, the frame is displayed in synchronization with VSync 755.

[0032] Referring to FIG. 8, a timing diagram of another enhancement to a wireless VR / AR system for reducing latency is shown. After a head pose 810 is captured and transmitted to a computing device 802, the head pose 810 is queued by the computing device 802 during a wait phase 815 while it waits for the start of the next frame according to the refresh rate. Next, a rendering step 820 and an encoding step 825 are performed in parallel for each slice of the frame that is transmitted over the network in step 830. The mobile device 804 then decodes the slice in step 835, which is performed in parallel with steps 820, 825, and 830. Also, the graphics engine on the mobile device 804 begins preparing each slice for display in step 840 as it is decoded. This allows the display step 840 to be performed in parallel with steps 820, 825, 830, and 835. In one embodiment, the display step 840 includes performing CSC and time warping on each slice. After the slice is prepared for display in step 840, the mobile device 804 waits 845 until the next VSync 850, after which the frame is displayed to the user.

[0033] Referring to Figure 9, one embodiment of a method 900 for reducing latency in a wireless VR / AR system is shown. For purposes of illustration, the steps of this embodiment and those of Figures 10-12 are shown sequentially. However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or devices described herein may be configured to perform method 900.

[0034] A rendering unit renders a frame and communicates the rendered frame to an encoder (block 905). The encoder encodes a first slice of the frame (block 910). In one embodiment, the first slice of the frame is a left-eye portion of the frame. After encoding the first slice, the encoder transmits an encoded version of the first slice to a receiver (block 915). Concurrently with transmitting the encoded version of the first slice to the receiver, the encoder encodes a second slice of the frame (block 920). In one embodiment, the second slice of the frame is a right-eye portion of the frame. Then, after completing encoding of the second slice, the encoder transmits an encoded version of the second slice of the frame to a receiver (block 925). After block 925, method 900 ends.

[0035] 10, one embodiment of a method 1000 for reducing latency in a wireless VR / AR system is shown. A frame is divided into slices (block 1005). A rendering unit then renders a given slice of the frame (block 1010). In a first iteration of method 1000, the rendering unit renders a first slice of the frame in block 1010. In a second iteration, the rendering unit renders a second slice, in a third iteration, the rendering unit renders a third slice, and so on until all slices of the frame have been rendered.

[0036] After block 1010, two separate steps begin in parallel for two separate iterations of method 1000. For the first step, if there are remaining slices of the frame that still need to be rendered (conditional block 1015: "yes"), the rendering unit selects the next slice (block 1020), after which method 1000 returns to block 1010. For the second step, which runs in parallel with conditional block 1015, the encoder encodes a given slice of the frame (block 1025). Now, if all slices of the frame have been rendered (conditional block 1015: "no"), this iteration of method 1000 ends.

[0037] After the encoding of a given slice is complete, the transmitter transmits the encoded version of the given slice to the receiver (block 1030). After the given slice is received by the receiver, the decoder decodes the given slice (block 1035). After the given slice is decoded, the graphics engine prepares the given slice for display (block 1040). If all slices of a frame are ready for preparation (conditional block 1045: "yes"), the frame is sent to the display at the next VSync (block 1050). After block 1050, method 1000 ends. Otherwise, if there are one or more slices that have not yet been prepared (conditional block 1045: "no"), this iteration of method 1000 ends. Note that at any given time, any number of iterations of method 1000 can traverse steps 1010, 1025, 1030, 1035, and 1040 for different slices of a frame. Note also that method 1000 can be repeated for each frame.

[0038] Referring to FIG. 11 , one embodiment of a method 1100 for signaling the end of a slice between units is shown. The rendering unit, when communicating the rendered slice to the encoder, generates a first type of indicator (indication) to notify the encoder that a particular slice is being transmitted (block 1105). In one embodiment, the first type of indicator is a slice ID that allows the encoder to assemble the slices back into a full frame. The slice ID can be communicated in many different ways and can be located anywhere within the data packet representing the slice. In another embodiment, the first type of indicator is a slice end indicator that is transmitted at the end of a particular slice. In some embodiments, the slice end indicator is a particular coding, bit pattern, signal over a sideband channel interface, or other indicator. In other embodiments, other types of slice indicators can be generated. The encoder then generates a second type of indicator (indication) to notify the decoder that a particular slice is being transmitted (block 1110) when communicating the encoded slice to the decoder. In one embodiment, the second type of indicator is different from the first type of indicator. The decoder then generates a third type of indicator (indication) when the decoded slice is being communicated to the graphics engine to inform the graphics engine that a particular slice is being transmitted (block 1115). In one embodiment, the third type of indicator is different from the second type of indicator. After block 1115, method 1100 ends.

[0039] By using different types of indicators throughout the wireless VR / AR system, different units recognize that they can begin consuming a slice while simultaneously receiving data from a subsequent slice. Each unit in the system may follow a different protocol and may have a different type of processing unit responsible for receiving and processing individual slices. Method 1100 may customize processing for each unit in the system to increase efficiency during processing of frames of a video sequence.

[0040] Referring to FIG. 12, one embodiment of a method 1200 for adjusting slice granularity by a unit of a wireless VR / AR system is shown. A rendering unit renders a first plurality of slices of a frame (block 1205). The rendering unit communicates each slice of the first plurality of slices to an encoder as each slice is rendered (block 1210). The encoder encodes a second plurality of slices of the frame (block 1215). In one embodiment, the number of slices in the second plurality of slices is different from the number of slices in the first plurality of slices. For example, in one scenario, the encoder encodes one slice for every two slices received from the rendering unit. In this scenario, the rendering unit renders twice the number of slices encoded by the encoder per frame. For example, the rendering unit may render 256 slices per frame, and the encoder may encode 128 slices per frame. In another scenario, the encoder encodes two slices for each slice received from the rendering unit. For example, in this scenario, the rendering unit may render 1024 slices per frame and the encoder may encode 2048 slices per frame. In other scenarios, the encoder may combine multiple slices or divide each slice received from the rendering unit in other ratios to generate each encoded slice.

[0041] Next, the decoder receives and decodes a second plurality of slices of the frame (block 1220). Then, the graphics engine receives the second plurality of slices from the decoder (block 1225). Next, the graphics engine processes a third plurality of slices of the frame (block 1230). In one embodiment, the number of slices in the third plurality of slices is different from the number of slices in the second plurality of slices. After block 1230, method 1200 ends. By using method 1200, each unit of the wireless VR / AR system can process and generate slices at a granularity that better suits the unit's capabilities. A rendering unit may prefer slices of a first size, while an encoder may prefer to encode slices of a second size that differs from the first size slices. Other units may also process slices of a different size than the size of the slices processed by the encoder and rendering unit.

[0042] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose or special-purpose processor are contemplated. In various embodiments, such program instructions are expressed in a high-level programming language. In other embodiments, the program instructions are compiled from a high-level programming language into binary, intermediate, or other form. Alternatively, the program instructions are written to describe the operation or design of hardware. Such program instructions are expressed in a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various embodiments, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Typically, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0043] It should be emphasized that the above-described embodiments are merely non-limiting examples of embodiments. Many variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. A rendering unit; an encoder; The rendering unit: Rendering a first slice of a frame, the first slice being a portion of the frame; Rendering a second slice of the frame; and and The encoder comprises: encoding a first slice of a frame to generate an encoded version of the first slice, the encoded version of the first slice being generated in parallel with the rendering unit rendering the second slice; the rendering unit is configured to generate a first indication to notify the encoder that a particular slice has been communicated to the encoder; the encoder is configured to generate a second indicator to inform the decoder that a coded version of a particular slice has been communicated to the decoder. system.

2. the rendering unit is configured to render a third slice of the frame in parallel with the encoder encoding a second slice of the frame to generate an encoded version of the second slice. The system of claim 1.

3. A receiver is provided, the receiver comprises the decoder configured to decode the coded version of the first slice while a transmitter transmits the coded version of the second slice to the receiver. The system of claim 2.

4. the second index is different from the first index; The system of claim 3.

5. the receiver comprises a graphics engine; the decoder is configured to communicate a decoded version of the first slice to the graphics engine before completing decoding of the encoded version of the second slice. The system of claim 4.

6. the graphics engine is configured to prepare a decoded version of the first slice for display on a display before receiving a decoded version of the second slice. The system of claim 5.

7. the graphics engine is configured to prepare a decoded version of the first slice for presentation on the display in parallel with the decoder decoding the second slice. The system of claim 6.

8. a rendering unit for rendering a first slice of a frame, the first slice being a portion of the frame; the rendering unit rendering a second slice of the frame; and an encoder encoding a first slice of a frame to generate an encoded version of the first slice, the encoded version of the first slice being generated in parallel with the rendering unit rendering the second slice; generating a first indication by the rendering unit to notify the encoder that a particular slice has been transmitted to the encoder; and generating a second indicator by the encoder to notify the decoder that a particular slice has been conveyed to the decoder. method.

9. the rendering unit further comprising: rendering a third slice of the frame in parallel with the encoder encoding the second slice of the frame to generate an encoded version of the second slice.

9. The method of claim 8.

10. the decoder decoding the coded version of the first slice while a transmitter is transmitting the coded version of the second slice.

10. The method of claim 9.

11. the second index is different from the first index; The method of claim 10.

12. the decoder further comprising communicating the decoded version of the first slice to a graphics engine before completing decoding of the encoded version of the second slice. The method of claim 11.

13. the graphics engine further preparing the decoded version of the first slice for display on a display before receiving the decoded version of the second slice.

13. The method of claim 12.

14. and wherein the graphics engine prepares a decoded version of the first slice for display on the display in parallel with the decoder decoding the second slice.

14. The method of claim 13.

15. a rendering unit configured to render a first slice of a frame, the first slice being a portion of the frame, and to render a second slice of the frame; an encoder configured to encode a first slice of the frame to generate an encoded version of the first slice and to encode a second slice of the frame to generate an encoded version of the second slice; a transmitter configured to transmit an encoded version of the first slice of the frame to a receiver before receiving an entire encoded version of the second slice of the frame from the encoder; the rendering unit is configured to generate a first indication to notify the encoder that a particular slice has been communicated to the encoder; the encoder is configured to generate a second indicator to notify the decoder that a particular slice has been conveyed to the decoder. Device.

16. the rendering unit is configured to render a third slice of the frame in parallel with the encoder encoding a second slice of the frame to generate an encoded version of the second slice.

16. The apparatus of claim 15.

17. the receiver comprises the decoder configured to decode the coded version of the first slice while the transmitter is transmitting the coded version of the second slice.

17. The apparatus of claim 16.

18. the second index is different from the first index; 18. The apparatus of claim 17.

19. the receiver comprises a graphics engine; the decoder is configured to communicate a decoded version of the first slice to the graphics engine before completing decoding of the encoded version of the second slice.

20. The apparatus of claim 18.

20. the graphics engine is configured to prepare a decoded version of the first slice for display on a display before receiving a decoded version of the second slice.

20. The apparatus of claim 19.

Images