Video Encoding System
The video encoding system for VR/MR uses wavelet transforms, slice-based and tile-based encoding, and pre-filtering to enhance transmission efficiency, addressing latency and frame rate maintenance under varying channel conditions, thereby optimizing image quality and reducing bandwidth.
Patent Information
- Application Number
- JP2024022596
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-06-28
- Filing Date
- 2024-02-19
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2039-06-27
AI Technical Summary
Existing video encoding systems for virtual and mixed reality (VR/MR) face challenges in encoding high-resolution video with low latency over wireless communication links, particularly in maintaining a desired frame rate under varying channel capacity conditions and minimizing latency in rendering, transmitting, and displaying frames.
The system employs wavelet transforms, slice-based and tile-based rendering and encoding, pre-filtering techniques, and lens warping to decompose pixel data into frequency bands, reducing memory and power requirements, and optimizing transmission efficiency.
This approach maintains a target frame rate and minimizes latency while adapting to varying channel conditions, improving image quality and reducing bandwidth usage by applying frequency band prioritization and graceful degradation strategies.
Smart Images

Figure 0007591675000001 
Figure 0007591675000002 
Figure 0007591675000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a video encoding system. [Background technology]
[0002] Virtual reality (VR) allows users to experience and / or interact with immersive artificial environments, thereby making the user feel as if they are physically in the environment. For example, a virtual reality system may display a stereoscopic scene to the user to create the illusion of depth, and a computer can adjust the scene content in real time to provide the illusion that the user is moving within the scene. When a user views an image through a virtual reality system, the user can thus feel as if they are moving within the scene from a first-person perspective. Similarly, mixed reality (MR) combines computer-generated information (called virtual content) with real-world images or a real-world view to augment or add content to the user's view of the world. Alternatively, mixed reality combines virtual representations of real-world objects with a three-dimensional (3D) virtual world view. Thus, a simulated environment of virtual reality and / or a composite environment of mixed reality can be utilized to provide an interactive user experience of multiple applications. Summary of the Invention
[0003] Various embodiments of a video encoding system are described that can encode high-resolution video sources with low latency for transmission over a communication link (e.g., a wireless link) to a device for decoding and display. Embodiments of the video encoding system can also provide graceful degradation of the encoded video transmitted to the device and maintain a desired frame rate under varying conditions such as changes in the channel capacity of the communication link. An exemplary application of the video encoding system is in a virtual or mixed reality system where video frames including virtual content are rendered, encoded, and transmitted by a base station to a device (e.g., a notebook or laptop computer, a pad or tablet device, a smartphone, or a head-mounted display (HMD), such as a headset, helmet, goggles, or glasses that may be worn by a user) for decoding and display.
[0004] Various methods and apparatus may be implemented by a video encoding system to maintain a target frame rate over a wireless link and minimize latency in rendering, transmitting, and displaying frames. In addition, the methods and apparatus may provide graceful degradation of encoded video transmitted to a device to maintain a desired frame rate under varying conditions, such as changes in channel capacity of the communication link.
[0005] In some embodiments, the video coding system may perform a wavelet transform on pixel data before encoding, decomposing the pixel data into frequency bands. The frequency bands are then organized into blocks that are provided to a block-based encoder and encoded / compressed. The encoded frequency data is then sent to an air interface, where the encoded frequency data is packetized and the packets are sent to a receiving device. At the receiving device, the encoded data is packet decoded and passed through a block-based decoder to recover the frequency bands. Wavelet synthesis is then performed on the recovered frequency bands to reconstruct the pixel data for display.
[0006] In some embodiments, the video coding system may perform slice-based rendering, encoding, and transmission. Instead of rendering and encoding an entire frame in the base station and transmitting the rendered frame to the device, the base station may render and encode a portion or portions (called slices) of the frame and transmit the encoded slices to the device as they are ready. Slice-based rendering and encoding may help reduce latency and may also reduce the amount of memory required for buffers, thereby reducing the memory footprint on the chip(s) or processor(s) as well as power requirements.
[0007] In some embodiments, the video coding system can perform tile-based rendering, encoding, and transmission, in which each slice may be divided into multiple tiles (e.g., four tiles), and the base station may render and encode the tiles and transmit the encoded tiles to the device as soon as they are ready.
[0008] In some embodiments, the video coding system may perform tile-based coding using a single encoder to process tiles from each slice. However, in some embodiments, the video coding system may perform tile-based coding using multiple encoders to process tiles from each slice respectively. For example, in some embodiments, each slice may be divided into four tiles and two encoders may operate on two tiles from each slice. Each encoder may multiplex the processing of blocks from different frequency bands between tiles to allow multiple time units between processing of blocks from the same frequency band. By multiplexing the processing of blocks between tiles, dependencies between blocks within a frequency band may be properly addressed.
[0009] In some embodiments, the video encoding system may perform pre-filtering of pixel data within a frame prior to wavelet transform. In some embodiments, the pre-filtering may include performing a lens warp function on the frame on the base station prior to wavelet transform. Lens warping is performed to correct image distortions introduced by the lenses of the device (e.g., HMD) on which the image is viewed, thereby improving image quality. By performing lens warping on the base station at the pre-filter stage, the resolution of the frame may be reduced before performing the wavelet transform and encoding. This may help improve compression and may reduce latency and bandwidth usage over the wireless link. Additionally, by performing lens warping on the base station at the pre-filter stage, rather than at the device after decoding, filtering of the image data only needs to be performed once. This is in contrast to performing filtering at the base station to reduce resolution prior to encoding, and then performing lens warp filtering on the device.
[0010] In some embodiments, the pre-filtering may include filtering with reduced resolution in the peripheral region while maintaining higher resolution within the foveal region. In this method, eye-tracking information obtained from the device may be used to identify the direction in which the user is currently looking. A region of the frame corresponding to the fovea (referred to as the foveal region) may be identified based at least in part on the determined gaze direction. In some embodiments, the peripheral region (i.e., the portion of the frame outside the foveal region) may be pre-filtered to reduce the resolution in the peripheral region. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a block diagram illustrating a video coding system that uses a wavelet transform to decompose pixel data into frequency bands before encoding, according to at least some embodiments. [Diagram 2]1 illustrates a video coding system including multiple encoders processing tiles from a frame in parallel, according to at least some embodiments. [Figure 3A] 1 illustrates an example frame divided into slices and tiles according to at least some embodiments. [Figure 3B] 1 illustrates an example of a tile divided into blocks, according to at least some embodiments. [Figure 3C] 1 illustrates performing a wavelet transform of a block of stored pixel data to generate frequency band data, according to at least some embodiments. [Figure 4] 3 is a high-level flowchart of a method of operation of a VR / MR system that includes a video encoding system such as that shown in FIGS. 1 and 2, according to at least some embodiments. [Diagram 5] 2 is a flowchart of a method of operation of a video encoding system such as that shown in FIG. 1, according to at least some embodiments. [Figure 6] 3 is a flowchart of a method of operation of a video encoding system such as that shown in FIG. 2, according to at least some embodiments. [Figure 7] 1 illustrates an example VR / MR system in which a video encoding system can be implemented according to at least some embodiments. [Figure 8] FIG. 8 is a block diagram illustrating components of a VR / MR system such as that shown in FIG. 7, according to at least some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] This specification includes references to "one embodiment" or "an embodiment." The phrases "in one embodiment" or "in an embodiment" do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with the present disclosure.
[0013] "Comprising." This term is open-ended. When used in the claims, this term does not exclude additional structures or steps. Consider a claim that reads "Apparatus comprising one or more processor units...." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).
[0014] "Configured to". Various units, circuits, or other components may be described or claimed as being "configured to" perform a task or tasks. In this context, "configured" is used to imply structure by indicating that the unit / circuit / component includes structure (e.g., circuitry) that performs those task or tasks during operation. As such, a unit / circuit / component can be said to be configured to perform a task even when the specified unit / circuit / component is not currently operational (e.g., not turned on). A unit / circuit / component used with the term "configured" includes hardware, e.g., circuitry that stores executable program instructions to perform operations, memory, etc. It is expressly intended that describing a unit / circuit / component as being "configured" to perform one or more tasks does not expose the unit / circuit / component to 35 U.S.C. 112, paragraph (f). Additionally, "configured to" can include a general structure (e.g., a general circuit) that can be manipulated by software and / or firmware (e.g., an FPGA or a general-purpose processor running software) to operate in a manner capable of performing a target task(s). "Configured to" can also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate a device (e.g., an integrated circuit) adapted to perform or execute one or more tasks.
[0015] "first," "second," etc. As used herein, these terms are used as markers for the nouns that follow and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations on "first" and "second" values. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.
[0016] "Based on" or "according to." As used herein, such terms are used to describe one or more factors that influence the determination. Such terms do not exclude additional factors that may influence the determination. That is, the determination may be based solely on those factors, or at least in part on those factors. Consider the phrase "determining A based on B." In this case, B is a factor that influences the determination of A, but such a phrase does not exclude that the determination of A is also based on C. In other examples, A may be determined solely based on B.
[0017] "Or." When used in the claims, the term "or" is used as an inclusive or and not an exclusive or. For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, and any combination thereof.
[0018] Various embodiments of a video encoding system are described. The video encoding system embodiments can encode high-resolution video sources with low latency for transmission over a communications link (e.g., a wireless link) to a device for decoding and display. The video encoding system embodiments can also provide graceful degradation of the encoded video transmitted to the device and maintain a desired frame rate under varying conditions, such as changes in the channel capacity of the communications link.
[0019] An exemplary application of the video encoding system is in a virtual or mixed reality system, where video frames including virtual content are rendered, encoded, and transmitted to a device for decoding and display. Embodiments of a virtual or mixed reality system (referred to herein as a VR / MR system) are described in which embodiments of the video encoding system may be implemented. In some embodiments, the VR / MR system may include a device (e.g., a pad or tablet device, a smartphone, or a headset, helmet, goggles, or glasses worn by a user and referred to herein as a head mounted display (HMD)) and a separate computing device, referred to herein as a base station. In some embodiments, the device and the base station may each include wireless communication technology that allows the device and the base station to communicate and exchange data over a wireless connection. In some embodiments, the device may include sensors that collect information about the user's environment (e.g., video, depth information, lighting information, etc.) and information about the user (e.g., the user's facial expressions, eye movements, hand gestures, etc.). Information collected by the sensors may be transmitted to the base station over a wireless connection. The base station may include software and hardware (e.g., a processor (system on a chip, SOC, CPU, image signal processor (ISP), graphics processing unit (GPU), encoder / decoder (codec), etc.), memory, etc.) configured to generate and render frames including virtual content based at least in part on sensor information received from the device over the wireless connection. The base station may also include an embodiment of a video encoding system as described herein that can pre-filter, compress, and transmit the rendered frames over the wireless connection to the device for display.
[0020] In some embodiments, the VR / MR system may implement a proprietary wireless communication technology that provides a highly directional wireless link between the device and the base station. In some embodiments, the directionality and bandwidth of the wireless communication technology may support multiple devices to communicate with the base station simultaneously, thereby enabling multiple users to use the system simultaneously in a co-located environment. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be supported.
[0021] The main constraints considered for the wireless link include bandwidth and latency. The goal of a VR / MR system is to provide a high-resolution, wide field of view (FOV) virtual display at a frame rate that provides a high-quality VR / MR viewing experience to the user. Another goal is to minimize the latency between the time a frame is rendered by the base station and the time the frame is displayed by the device. However, the channel capacity of the wireless link may change over time, and thus the wireless link may only support a certain amount of information being transmitted at any given time.
[0022] Described herein are various methods and apparatus that may be implemented by a video encoding system to maintain a target frame rate over a wireless link and minimize latency in rendering, transmitting, and displaying frames. In addition, the methods and apparatus may provide for graceful degradation of encoded video transmitted to a device to maintain a desired frame rate under varying conditions, such as changes in channel capacity of a communications link.
[0023] In some embodiments, the video encoding system may perform a wavelet transform on the encoded pixel data before encoding to decompose the pixel data into frequency bands. The frequency bands are then organized into blocks that are provided to a block-based encoder for encoding / compression. As an example, a frame may be divided into 128x128 blocks and a two-level wavelet decomposition may be applied to each of the 128x128 blocks to generate 16 blocks of 32x32 frequency data. The 16 32x32 frequency data blocks, representing 7 frequency bands, may then be sent to an encoder (e.g., a High Efficiency Video Coding (HEVC) encoder) and encoded. The encoded frequency data may then be sent to an air interface, where the encoded frequency data is packetized and the packets are sent to a receiving device (e.g., an HMD). At the receiving device, the encoded data is packet decoded and passed through a block-based decoder to recover the frequency bands. Wavelet synthesis may then be performed on the recovered frequency bands to reconstruct pixel data for display.
[0024] Although embodiments are generally described in which the wavelet transform is a two-level wavelet decomposition applied to each pixel block from a video frame, in various embodiments the wavelet decomposition may be any number N of levels (e.g., one level, two levels, three levels, four levels, etc.), and the number of levels N may be adjusted to trade off quality of the encoded image against complexity of the block being encoded.
[0025] In some embodiments, a coring function can be applied to each of the wavelet coefficients in the wavelet decomposition. "Coring" refers to the soft-thresholding of the coefficients, which is useful for removing noise. This can be done adaptively versus regions of interest versus available bandwidth.
[0026] In some embodiments, the video coding system may perform slice-based rendering, encoding, and transmission. Rendering, encoding, and transmitting an entire frame may have latency and memory impacts, since each frame needs to be completed, stored, and then transmitted to the next stage of the VR / MR system. In slice-based rendering, instead of rendering and encoding an entire frame in the base station and transmitting the rendered frame to the device, the base station may render and encode a portion of the frame (called a slice) and transmit the encoded slice to the device as soon as it is ready. A slice may be, for example, a 128x128 block row, or two or more block rows. Slice-based rendering and encoding may help reduce latency, and may also reduce the amount of memory required for buffers, which may reduce the memory footprint on the chip(s) or processor(s) as well as the power required.
[0027] In some embodiments, the video coding system can perform tile-based rendering, encoding, and transmission, in which each slice may be divided into multiple tiles (e.g., four tiles), and the base station may render and encode the tiles and transmit the encoded tiles to the device as soon as they are ready.
[0028] In some embodiments, the video coding system may perform tile-based coding using a single encoder to process tiles from each slice. However, in some embodiments, the video coding system may perform tile-based coding using multiple encoders to process tiles from each slice, respectively. For example, in some embodiments, each slice may be divided into four tiles, each tile including multiple 128x128 blocks, and two encoders (e0 and e1) may operate on two tiles from each slice (e.g., e0 operates at t0 and t1, and e1 operates at t2 and t3). Each encoder may multiplex the processing of blocks from different frequency bands between the two tiles to allow 16 time units between the processing of blocks from the same frequency band. By multiplexing the processing of blocks between the two tiles, dependencies between blocks of frequency bands may be properly addressed. Although an embodiment is described in which each slice is divided into four tiles and two encoders operate on respective tiles from each slice, in some embodiments a slice may be divided into more tiles (e.g., six or eight tiles) and in some embodiments more encoders may be used (e.g., three or four encoders).
[0029] In some embodiments, the video encoding system can perform pre-filtering of pixel data within a frame prior to wavelet transformation. Pre-filtering can, for example, reduce the resolution of a frame rendered by a base station prior to performing the wavelet transform, encoding, and transmission of the frame to a device over a wireless link. This can help improve compression and can reduce latency and bandwidth usage over the wireless link.
[0030] In some embodiments where the device is an HMD, the pre-filtering may include performing lens warp on the frames on the base station prior to the wavelet transform. Lens warp is performed to correct image distortion introduced by the lenses on the HMD through which the images are viewed, thereby improving the quality of the image. In some embodiments, the HMD may store lens warp data for the lenses, generated, for example, by a calibration process, and may provide the lens warp data to the base station via a wireless connection. The base station may then perform lens warp on the frames based on the lens warp data for that HMD. In traditional VR / MR systems, lens warp is performed on the HMD after decoding and before display. By performing lens warp on the base station in the pre-filter stage, the resolution of the frames may be reduced before performing the wavelet transform and encoding. This may help improve compression and reduce latency and bandwidth usage on the wireless link. Additionally, by performing lens warp on the base station in the pre-filter stage, rather than on the HMD after decoding, filtering of the image data only needs to be performed once. This is in contrast to performing filtering at the base station to reduce the resolution before encoding, and then performing lens warp filtering on the HMD.
[0031] In some embodiments, the pre-filtering may include filtering with reduced resolution in the peripheral region while maintaining higher resolution within the foveal region. In this manner, eye tracking information obtained from the device may be used to identify the direction in which the user is currently looking. The human eye may perceive higher resolution in the foveal region than in the peripheral region. Thus, a region of the frame corresponding to the fovea (referred to as the foveal region) may be identified based at least in part on the determined gaze direction. In some embodiments, the peripheral region (i.e., the portion of the frame outside the foveal region) may be pre-filtered to reduce information based on knowledge of the human visual system, for example, by filtering high frequency information and / or increasing color compression. In some embodiments, the amount of filtering applied to the peripheral region may increase extending towards the periphery of the frame. Pre-filtering the peripheral region may help improve compression of the frame.
[0032] 1 is a block diagram illustrating a video encoding system 120 that uses a wavelet transform before encoding to decompose pixel data into frequency bands, according to at least some embodiments. The VR / MR system 10 may include at least one device 150 (e.g., a pad or tablet device, a smartphone, or an HMD, such as a headset, helmet, goggles, or glasses that may be worn by a user) and a computing device 100 (referred to herein as a base station). The base station 100 renders VR or MR frames containing virtual content, encodes the frames, and transmits the encoded frames to the device 150 over a wireless connection 180 to be decoded and displayed by the device 150.
[0033] In some embodiments, HMD 150 may include sensors 160 that collect information about the environment of user 190 (e.g., video, depth information, lighting information, etc.) and about user 190 (e.g., the user's facial expressions, eye movements, gaze direction, hand gestures, etc.). Device 150 may transmit at least a portion of the information collected by sensors 160 to base station 100 over wireless connection 180. Based at least in part on various information obtained from sensors 160, base station 100 can render frames for display by device 150 including virtual content, encode the frames, and transmit the encoded frames to device 150 over wireless connection 180 for decoding and display to the user.
[0034] Base station 100 and device 150 may implement wireless communication technologies that enable base station 100 and device 150 to communicate and exchange data via wireless connection 180. In some embodiments, wireless connection 180 may be implemented according to a proprietary wireless communication technology that provides a highly directional wireless link between device 150 and base station 100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.
[0035] The primary constraints considered for the wireless connection 180 between the device 150 and the base station 100 in the VR / MR system 10 include bandwidth and latency. For example, in some embodiments, a goal is to provide a high-resolution, wide field of view (FOV) virtual display to the user at a frame rate that provides the user with a high-quality VR / MR view. Another goal is to minimize the latency between the time a video frame is captured by the device and the time a VR / MR frame rendered based on the video frame is displayed by the device.
[0036] The base station 100 may include various hardware components, such as various types of processors, integrated circuits (ICs), central processing units (CPUs), graphic processing units (GPUs), image signal processors (ISPs), encoders / decoders (codecs), etc., for rendering, filtering, encoding, and transmitting video and / or images as described herein. The base station 100 may include a video encoding system 120, which may include, but is not limited to, a GPU rendering 110 component, a wireless interface 130 component, and one or more hardware components implementing various methods that may help maintain a target frame rate and minimize latency in frame rendering, encoding, transmission, and display over a wireless connection 180. The video encoding system 120 may include, but is not limited to, a prefilter 122 component (e.g., an N-channel filter bank), a wavelet transform 124 component, and an encoder 126 component.
[0037] GPU rendering 110 may include one or more hardware components capable of rendering frames for display by device 150 including virtual content based at least in part on various information obtained from sensors 160.
[0038] In some embodiments, the video encoding system 120 may include one or more hardware components that pre-filter 122 pixel data in the rendered frames before performing a wavelet transform 124. The pre-filter 122 may, for example, reduce the resolution of the frames rendered on the base station 100 before performing the wavelet transform 124, encoding 126, and transmission of the frames to the device 150 over the wireless connection 180. This may help improve compression and reduce latency and bandwidth usage over the wireless connection 180.
[0039] In some embodiments, the pre-filter 122 may perform lens warping on the frames on the base station 100 before the wavelet transform 124. Lens warping is performed to correct image distortions introduced by the lens on the device on which the image is viewed, thereby improving the quality of the image. In some embodiments, the device 150 can store lens warp data for the lens, generated, for example, by a calibration process, and can provide the lens warp data to the base station 100 over the wireless connection 180. The pre-filter 122 component of the video encoding system 120 can then perform lens warping on the frames based on the lens warp data of the device 150. In traditional VR / MR systems, lens warping is performed on the device 150 after decoding and before display. By performing lens warping on the base station 100 in the pre-filter 122 stage, the resolution of the frames can be reduced before performing the wavelet transform 124 and encoding 126. This can help improve compression and can reduce latency and bandwidth usage on the wireless connection 180. Additionally, by performing lens warping at base station 100 in the prefilter 122 stage, rather than after decoding at device 150, filtering of the image data only needs to be performed once. This is in contrast to performing filtering at base station 100 to reduce resolution prior to encoding 126, and then performing lens warp filtering on device 150.
[0040] In some embodiments, the pre-filter 122 may also apply one or more filters that reduce resolution in the peripheral region while maintaining higher resolution in the foveal region. In this manner, eye tracking information obtained from the device 150 may be used to identify the direction in which the user is currently looking. The human eye can perceive higher resolution in the foveal region than in the peripheral region. Thus, the region of the frame that corresponds to the fovea (referred to as the foveal region) may be identified based at least in part on the determined gaze direction. In some embodiments, the peripheral region (i.e., the portion of the frame outside the foveal region) may be pre-filtered to reduce information based on knowledge of the human visual system, for example, by filtering high frequency information and / or increasing color compression. In some embodiments, the amount of filtering applied to the peripheral region may increase extending towards the periphery of the frame. Pre-filtering the peripheral region may help improve compression of the frame.
[0041] In some embodiments, the wavelet transform 124 component of the video coding system 120 may include one or more hardware components (e.g., an N-channel filter bank) that perform a wavelet transform on pixel data before encoding and decompose the pixel data into frequency bands. The frequency bands are then organized into blocks that are provided to a block-based encoder 126 for encoding / compression. As an example, as shown in FIGS. 3A-3C, a frame may be divided into 128x128 blocks and a two-level wavelet decomposition may be applied to each of the 128x128 blocks to generate sixteen 32x32 blocks of frequency data. The sixteen 32x32 frequency data blocks, representing seven frequency bands, may then be transmitted to the block-based encoder (e.g., a High Efficiency Video Coding (HEVC) encoder) 126 and encoded. The encoded frequency data is then transmitted to a wireless interface 130 that implements one or more hardware components. This causes the data to be packetized and the packets transmitted to the device 150 over a wireless connection 180.
[0042] The device 150 may include various hardware components, such as various types of processors, integrated circuits (ICs), central processing units (CPUs), graphic processing units (GPUs), image signal processors (ISPs), encoders / decoders (codecs), etc., for decoding and displaying video and / or images as described herein. The device 150 may include, but is not limited to, a radio interface 152, a decoder 154 component (e.g., a High Efficiency Video Coding (HEVC) decoder), a wavelet synthesis 156 component, and a display 158 component. On the device 150, the radio interface 152 receives packets transmitted by the base station 100 over a wireless connection 180. The encoded data is packet decoded and passed through a block-based decoder 154 (e.g., a High Efficiency Video Coding (HEVC) decoder) to recover the frequency bands. Wavelet synthesis 156 is then performed on the recovered frequency data to reconstruct pixel data for the display 158.
[0043] In some embodiments, the video coding system 120 can perform slice-based rendering, encoding, and transmission. Rendering, encoding, and transmitting an entire frame can have latency and memory impacts because each frame needs to be completed, stored, and then transmitted to the next stage of the VR / MR system 10. In slice-based rendering, rather than rendering and encoding an entire frame in the base station 100 and transmitting the rendered frame to the device 150, the base station 100 may render and encode a portion of a frame (called a slice) and transmit the encoded slice to the device 150 as soon as it is ready. A slice may be, for example, a 128x128 row of blocks. Slice-based rendering and encoding can help reduce latency and can also reduce the amount of memory required for buffers, thereby reducing the memory footprint on the chip(s) or processor(s) as well as the power required.
[0044] In some embodiments, video coding system 120 may perform tile-based rendering, encoding, and transmission, in which each slice may be divided into multiple tiles (e.g., four tiles), and base station 100 may render and encode the tiles and transmit the encoded tiles to device 150 as soon as they are ready.
[0045] In some embodiments, video coding system 120 may perform tile-based rendering, encoding, and transmission to process tiles from each slice using a single encoder 126. However, in some embodiments, video coding system 120 may perform tile-based encoding to process tiles from each slice using multiple encoders 126. Figure 2 illustrates a video coding system 220 that includes multiple encoders (two encoders 226A and 226B in this example) that process tiles from a rendered frame in parallel, according to at least some embodiments.
[0046] The GPU rendering 210 component of the base station 200 may include one or more GPUs and / or other components that render a frame (or a slice of a frame) for display. A frame may be divided into slices, for example, as shown in FIG. 3A. As shown in FIG. 3A, each slice may be divided into multiple tiles (four in this example), with each tile including multiple blocks. FIG. 3B shows an example tile including four 128x128 blocks. However, in some embodiments, blocks of other sizes (e.g., 64x64, 32x32, etc.) may be used, and a tile may include more or fewer blocks.
[0047] The prefilter 222 and wavelet transform 224 components of the video encoding system 220 can then process each tile prior to encoding 226. In some embodiments, the video encoding system 220 can include separate prefilter 222 and wavelet transform 224 components for processing each tile. In this example, the prefilter 222A and wavelet transform 224A components process tile 0, the prefilter 222B and wavelet transform 224B components process tile 1, the prefilter 222C and wavelet transform 224C components process tile 2, and the prefilter 222D and wavelet transform 224D components process tile 3. The prefilter 222 component performs prefiltering of the tiles as described herein, and the wavelet transform 224 component decomposes the tiles into frequency bands as described herein. However, in some embodiments, the video encoding system 220 can include a single prefilter 222 and a single wavelet transform 224 component that process the tiles. In some embodiments, the video encoding system 220 may include multiple (e.g., two) prefilter 222 components and multiple (e.g., two) wavelet transform 224 components, each processing multiple (e.g., two) tiles.
[0048] The two encoders 226A and 226B may operate on two tiles from each slice (e.g., encoder 226A operates on tiles 0 and 1, and encoder 226B operates on tiles 2 and 3). Each encoder 226 may multiplex the processing of blocks from different frequency bands (i.e., the 16 32x32 blocks illustrated in FIG. 3C) between the two tiles to allow 16 time units between the processing of blocks from the same frequency band. By multiplexing the processing of blocks between the two tiles, dependencies between blocks in the same frequency band can be properly addressed.
[0049] Although an embodiment is described in which each slice is divided into four tiles and two encoders operate on respective tiles from each slice, in some embodiments a slice may be divided into more tiles (e.g., six or eight tiles) and in some embodiments more encoders may be used (e.g., three, four, or more encoders).
[0050] 3C illustrates performing a wavelet transform of a pixel block to store pixel data and generate frequency band data prior to encoding, according to at least some embodiments. In this example, a two-level wavelet decomposition is applied to a 128x128 pixel block 300 by a wavelet transform 324 component to generate sixteen 32x32 blocks 302 of frequency data representing seven frequency bands. The frequency blocks 302 are then provided to an encoder 326 for encoding. For example, the frequency blocks 302 may be written to a buffer by the wavelet transform 324 component and read from the buffer by the encoder 326 component.
[0051] In the labels of the frequency blocks 302, the letter L represents a low pass filter and the letter H represents a high pass filter. The two letter labeled blocks 302 represent a one level (2D) wavelet transform or decomposition. In the two letter labeled blocks 302 (representing three of the seven frequency bands LH, HL, and HH), the first letter represents the vertical filter (either high or low) that is performed first and the second letter represents the horizontal filter (either high or low) that is performed second. The four letter labeled blocks 302 represent a two level wavelet transform or decomposition. In the four letter labeled blocks 302, the first two letters (LL) indicate that the vertical low pass filter is first, followed by the horizontal low pass filter. The second two letters indicate that the resulting LL block has been filtered four ways: LL, LH, HL, and HH (thus generating four of the seven frequency bands: LLLL, LLLH, LLHL, and LLHH).
[0052] Decomposing pixel data into frequency bands as shown in FIG. 3C allows the frequency bands to be buffered and processed as separate streams by the encoder 326. Processing the frequency bands as separate streams allows components of the encoder 326 to multiplex the processing of independent streams. In block-based encoding methods such as HEVC encoding, blocks (called coding tree units (CTUs)) are processed in multiple stages in a block processing pipeline. Two or more blocks may be in different stages of the pipeline in a given clock cycle, and blocks move through the pipeline as cycles of the clock progress. The processing of a given block may have dependencies on one or more neighboring blocks that were previously processed, for example, one or more blocks in the row above the given block and / or blocks to the left of the given block. By multiplexing the processing of streams of frequency band data, the encoder 326 spaces out the processing of blocks in a given stream, thereby providing additional clock cycles for processing neighboring blocks on which the given block has dependencies. For example, when a given block reaches a stage where it depends on a previously processed neighboring block, the block to the left of the given block may be a block that precedes the given block by several stages in the pipeline, allowing the encoder 326 to better handle dependencies on previously processed blocks and reduce or eliminate the need to wait for completion of processing of neighboring blocks in the pipeline before proceeding with processing of the given block in a stage where it depends on the neighboring block.
[0053] In addition, decomposing the pixel data into frequency bands as shown in FIG. 3C allows for frequency band prioritization by the encoder 326 and the wireless interface. Typically, in image and video transmission, lower frequencies are more important while higher frequencies are less important. Higher frequencies usually correspond to image details and therefore may be considered lower priority. Higher frequency bands contain a smaller percentage of the energy in the image. Most of the energy is contained in the lower frequency bands. In this manner, decomposing the pixel data into frequency bands provides prioritization to the data streams that may be exploited by the encoder 326 and the wireless interface when encoding and transmitting the data streams. For example, in some embodiments, different compression techniques may be used for different frequency bands, applying more aggressive compression to lower priority bands and less aggressive compression to higher priority bands. As another example, prioritizing frequency bands may help provide graceful degradation for VR / MR systems. The performance of the wireless connection may be monitored and feedback from the devices may be taken into account to track the performance of the overall system. If system performance degrades for any reason, e.g., if the quality of the wireless connection degrades and the bandwidth capacity of the wireless connection falls below a threshold, the encoder 326 and radio interface may prioritize the encoding and transmission of one or more lower frequency bands and may reduce or decrease the encoding and / or transmission of one or more frequency levels assigned a lower priority level, e.g., one or more higher frequency bands.
[0054] As mentioned above, the wavelet transform decomposes an image into frequency bands. In some embodiments, this may be exploited to send the same signal to displays of different resolutions. As an example, a two-level wavelet decomposition may be applied to decompose the signal into seven frequency bands. If four bands (LLLL, LLLH, LLHL, and LLHH) are sent to a display panel, the bands may be reconstructed to their original intended resolution with lower visual quality. Alternatively, they may be reconstructed at 1 / 4 resolution (1 / 2 in each dimension), which may be suitable for a display panel with a lower display resolution.
[0055] 4 is a high-level flowchart of a method of operation of a VR / MR system including a video encoding system such as that shown in FIG. 1 and FIG. 2, according to at least some embodiments. As shown at 400, the device transmits data to a base station over a wireless connection. As shown at 410, the base station renders frames including virtual content based at least in part on the data received from the device. As shown at 420, the base station compresses the rendered data and transmits the compressed data to the device over the wireless connection. As shown at 430, the device unpacks and displays the virtual content to generate a 3D virtual view for viewing by the user. As shown by the arrow from 430 back to 400, the method continues for as long as the user is using the VR / MR system.
[0056] In some embodiments, rather than rendering and encoding an entire frame in the base station and transmitting the rendered frame to the device, the base station may render and encode a portion of the frame (called a slice) and transmit the encoded slice to the device as soon as it is ready. A slice may be, for example, a 128x128 row of blocks. In some embodiments, the video encoding system may perform tile-based rendering, encoding, and transmission. In tile-based rendering, encoding, and transmission, each slice may be divided into multiple tiles (e.g., four tiles each containing four blocks), each containing one or more blocks, and the base station may render and encode the tiles and transmit the encoded tiles to the device as soon as they are ready.
[0057] Figure 5 is a flow chart of a method of operation of a video coding system such as that shown in Figure 1, according to at least some embodiments. The method of Figure 5 may be performed, for example, at 420 of Figure 4. The method of Figure 5 assumes that slice-based coding and transmission is being performed. However, in some embodiments, tile-based coding and transmission may be performed.
[0058] As shown at 510, the prefilter component applies lens warping and / or a foveal filter to pixel blocks in a slice of a frame. In some embodiments, prefiltering may include performing lens warping on the frame on the base station prior to the wavelet transform. Lens warping is performed to correct for image distortion introduced by a lens on the device on which the image is viewed, thereby improving the quality of the image. In some embodiments, the device may store lens warp data for the lens, generated, for example, by a calibration process, and may provide the lens warp data to the base station via a wireless connection. The base station may then perform lens warping on the frame based on the lens warp data of the device. By performing lens warping on the base station at the prefilter stage, the resolution of the frame may be reduced before performing the wavelet transform and encoding. This may help improve compression and may reduce latency and bandwidth usage on the wireless link. Additionally, by performing lens warping on the base station at the prefilter stage, rather than at the device after decoding, filtering of the image data only needs to be performed once. This is in contrast to performing filtering at the base station to reduce resolution prior to encoding, and then performing lens warp filtering on the device.
[0059] In some embodiments, the pre-filtering at 510 may also include filtering with reduced resolution in the peripheral region while maintaining higher resolution in the foveal region. In some embodiments, eye tracking information obtained from the device may be used to identify the direction in which the user is currently looking. The region of the frame corresponding to the fovea (referred to as the foveal region) may be identified based at least in part on the determined gaze direction. The peripheral region (i.e., the portion of the frame outside the foveal region) may be pre-filtered to reduce information based on knowledge of the human visual system, for example, by filtering high frequency information and / or increasing color compression. Pre-filtering the peripheral region may help improve compression of the frame.
[0060] As shown at 520, a wavelet transform component applies a wavelet transform technique to the pixel blocks to decompose the pixel data into N (e.g., seven) frequency bands. The frequency bands are then organized into blocks that are provided to a block-based encoder for encoding / compression. As an example, a frame may be divided into 128x128 blocks, and a two-level wavelet decomposition may be applied to each 128x128 block to generate sixteen 32x32 blocks of frequency data representing the seven frequency bands, as shown, for example, in FIG. 3C.
[0061] As shown at 530, the encoder applies encoding techniques to frequency bands within the block to compress the data. The encoder may be, for example, a High Efficiency Video Coding (HEVC) encoder. However, in some embodiments, other encoding techniques may be used. Decomposing the pixel data into frequency bands as shown at element 520 allows the frequency bands to be buffered and processed as separate streams by the encoder. Processing the frequency bands as separate streams allows the encoder components to multiplex the processing of the independent streams. In block-based encoding methods such as HEVC encoding, blocks (called coding tree units (CTUs)) are processed in multiple stages in a pipeline. Two or more blocks may be in different stages of the pipeline in a given clock cycle, and blocks move through the pipeline as cycles of the clock progress. The processing of a given block may have dependencies on one or more neighboring blocks that have been previously processed, for example, one or more blocks in the row above the given block and / or blocks to the left of the given block. By multiplexing the processing of streams, the encoder spaces out the processing of blocks in a given stream, thereby providing additional clock cycles for processing neighboring blocks on which a given block has a dependency. For example, when a given block reaches a stage where it depends on a previously processed neighboring block, the block to the left of the given block may be a block that precedes the given block in the pipeline by several stages. This allows the encoder to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for completion of processing of neighboring blocks in the pipeline before processing a given block in a stage where it depends on the neighboring block.
[0062] As indicated at 540, the wireless interface packetizes the compressed data and transmits the packets over a wireless connection to the device.
[0063] The decomposition of pixel data into frequency bands as shown in element 520 allows for frequency band prioritization by the encoder at element 530 and the wireless interface at element 540. Typically, in image and video transmission, lower frequencies are more important while higher frequencies are less important. Higher frequencies usually correspond to image details and therefore may be considered lower priority. Higher frequency bands contain a smaller percentage of the energy in the image. Most of the energy is contained in the lower frequency bands. In this manner, the decomposition of pixel data into frequency bands provides prioritization to the data streams that may be exploited by the encoder and wireless interface when encoding and transmitting the data streams. For example, in some embodiments, different compression techniques may be used for different frequency bands, applying more aggressive compression to lower priority bands and less aggressive compression to higher priority bands. As another example, prioritization of frequency bands may help provide graceful degradation for VR / MR systems. The performance of the wireless connection may be monitored and feedback from the devices may be taken into account to track the performance of the overall system. If system performance degrades for any reason, e.g., if the quality of the wireless connection degrades and the bandwidth capacity of the wireless connection falls below a threshold, the encoder and radio interface may prioritize the encoding and transmission of one or more lower frequency bands and may reduce or slow down the encoding and / or transmission of one or more frequency levels assigned a lower priority level, e.g., one or more higher frequency bands.
[0064] If there are more slices to encode and transmit at 550, the method returns to element 510 to process the next slice. Otherwise, if there are more frames to encode and transmit at 560, the method returns to element 510 to begin processing the next frame.
[0065] Figure 6 is a flowchart of a method of operation of a video coding system such as that shown in Figure 2, according to at least some embodiments. The method of Figure 6 may be performed, for example, at 420 of Figure 4. In the method of Figure 6, the video coding system may perform tile-based coding using multiple encoders to process tiles from each slice respectively.
[0066] As shown at 600, the rendering engine renders a slice that contains multiple tiles (in this example, four tiles), each of which contains multiple pixel blocks (in this example, four 128x128 pixel blocks).
[0067] As shown at 610, the prefilter component applies lens warping and / or a foveal filter to the slices. In some embodiments, prefiltering may include performing lens warping on the frames on the base station prior to the wavelet transform. Lens warping is performed to correct for image distortion introduced by a lens on the device on which the image is viewed, thereby improving the quality of the image. In some embodiments, the device may store lens warp data for the lens, generated, for example, by a calibration process, and may provide the lens warp data to the base station via a wireless connection. The base station may then perform lens warping on the frames based on the lens warp data of the device. By performing lens warping on the base station at the prefilter stage, the resolution of the frames may be reduced before performing the wavelet transform and encoding. This may help improve compression and may reduce latency and bandwidth usage on the wireless link. Additionally, by performing lens warping on the base station at the prefilter stage, rather than at the device after decoding, filtering of the image data only needs to be performed once. This is in contrast to performing filtering at the base station to reduce resolution prior to encoding, and then performing lens warp filtering on the device.
[0068] In some embodiments, the pre-filtering at 610 may also include filtering with reduced resolution in the peripheral region while maintaining higher resolution in the foveal region. In some embodiments, eye tracking information acquired by the device may be used to identify the direction in which the user is currently looking. The region of the frame corresponding to the fovea (referred to as the foveal region) may be identified based at least in part on the determined gaze direction. The peripheral region (i.e., the portion of the frame outside the foveal region) may be pre-filtered to reduce information based on knowledge of the human visual system, for example, by filtering high frequency information and / or increasing color compression. Pre-filtering the peripheral region may help improve compression of the frame.
[0069] In some embodiments, the video encoding system may include a single prefilter component that processes a tile. In some embodiments, the video encoding system may include a separate prefilter component for processing each tile. In some embodiments, the video encoding system may include multiple (e.g., two) prefilter components, each processing multiple (e.g., two) tiles.
[0070] As shown at 620, a wavelet transform component applies wavelet transform techniques to pixel blocks in a slice to decompose the pixel data into N (e.g., seven) frequency bands. The frequency bands are then organized into blocks (e.g., CTUs) that can be provided to a block-based encoder and encoded / compressed. As an example, a frame can be divided into 128x128 blocks and a two-level wavelet decomposition can be applied to each 128x128 block to generate sixteen 32x32 blocks of frequency data representing the seven frequency bands, e.g., as shown in FIG. 3C.
[0071] In some embodiments, the video coding system may include a single wavelet transform component that processes a tile. In some embodiments, the video coding system may include a separate wavelet transform component for processing each tile. In some embodiments, the video coding system may include multiple (e.g., two) wavelet transform components, each processing multiple (e.g., two) tiles.
[0072] In the method of Figure 6, the example video coding system includes two encoders configured to encode blocks of frequency data from different ones of the slices generated in parallel at element 620. For example, a first encoder may be configured to encode blocks from slices 0 and 1, and a second encoder may be configured to encode blocks from slices 2 and 3. As shown at 630A, the first encoder applies encoding techniques to frequency bands within blocks of tiles 0 and 1, and multiplexes the processing of blocks from the different frequency bands between the two different tiles. As shown at 630B, the second encoder applies encoding techniques to frequency bands within blocks of tiles 2 and 3, and multiplexes the processing of blocks from the different frequency bands between the two different tiles.
[0073] The encoder may be, for example, a High Efficiency Video Coding (HEVC) encoder. However, in some embodiments, other encoding techniques may be used. Decomposing pixel data into frequency bands as shown in element 620 allows the frequency bands to be buffered and processed as separate streams by the encoder in elements 630A and 630B. Processing the frequency bands as separate streams allows the encoder to multiplex the processing of independent streams. In block-based encoding methods such as HEVC encoding, blocks (called coding tree units (CTUs)) are processed in multiple stages in a pipeline. Two or more blocks may be in different stages of the pipeline in a given clock cycle, and blocks move through the pipeline as cycles of the clock progress. The processing of a given block may have dependencies on one or more neighboring blocks that have been previously processed, for example, one or more blocks in the row above the given block and / or blocks to the left of the given block. By multiplexing the processing of streams, the encoder spaces out the processing of blocks in a given stream, thereby providing additional clock cycles for processing neighboring blocks on which a given block has a dependency. For example, when a given block reaches a stage where it depends on a previously processed neighboring block, the block to the left of the given block may be a block that precedes the given block in the pipeline by several stages. This allows the encoder to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for completion of processing of neighboring blocks in the pipeline before processing a given block in a stage where it depends on the neighboring block.
[0074] The wireless interface packetizes the compressed data generated by the encoder in element 530 and transmits the packets to the device over the wireless connection, as indicated at 640. This method continues for as long as the user is using the VR / MR system, as indicated by the arrow from 650 back to 600.
[0075] The decomposition of pixel data into frequency bands as shown in element 620 allows for frequency band prioritization by the encoders at elements 630A and 630B and the wireless interface at element 640. Typically, in image and video transmission, lower frequencies are more important while higher frequencies are less important. Higher frequencies usually correspond to image details and therefore may be considered lower priority. Higher frequency bands contain a smaller percentage of the energy in the image. Most of the energy is contained in the lower frequency bands. In this manner, the decomposition of pixel data into frequency bands provides prioritization to the data streams that may be exploited by the encoder and wireless interface when encoding and transmitting the data streams. For example, in some embodiments, different compression techniques may be used for different frequency bands, applying more aggressive compression to lower priority bands and less aggressive compression to higher priority bands. As another example, prioritization of frequency bands may help provide graceful degradation for VR / MR systems. The performance of the wireless connection may be monitored and feedback from the devices may be taken into account to track the performance of the overall system. If system performance degrades for any reason, e.g., if the quality of the wireless connection degrades and the bandwidth capacity of the wireless connection falls below a threshold, the encoder and radio interface may prioritize the encoding and transmission of one or more lower frequency bands and may reduce or slow down the encoding and / or transmission of one or more frequency levels assigned a lower priority level, e.g., one or more higher frequency bands.
[0076] Although an embodiment is described in which each slice is divided into four tiles and two encoders operate on respective tiles from each slice, in some embodiments a slice may be divided into more tiles (e.g., six or eight tiles) and in some embodiments more encoders may be used (e.g., three or four encoders).
[0077] (Example VR / MR system) 7 illustrates an example VR / MR system 2000 in which a video encoding system can be implemented, according to at least some embodiments. The VR / MR system 2000 may include at least one device 2150 (e.g., a notebook or laptop computer, a pad or tablet device, a smartphone, a handheld computing device, or an HMD such as a headset, helmet, goggles, or glasses that may be worn by a user) and a computing device 2100 (referred to herein as a base station). The base station 2100 renders VR or MR frames containing virtual content, encodes the frames, and transmits the encoded frames to the device 2150 over a wireless connection 2180 to be decoded and displayed by the device 2150.
[0078] The base station 2100 and the device 2150 may each include wireless communication technology that enables the base station 2100 and the device 2150 to communicate and exchange data via wireless connection 2180. In some embodiments, the wireless connection 2180 may be implemented according to a proprietary wireless communication technology that provides a highly directional wireless link between the device 2150 and the base station 2100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.
[0079] In some embodiments, the HMD 2150 may include sensors that collect information about the user's environment (e.g., video, depth information, lighting information, etc.) and / or information about the user (e.g., the user's facial expressions, eye movements, gaze direction, hand gestures, etc.). The device 2150 may transmit at least a portion of the information collected by the sensors to the base station 2100 over a wireless connection 2180. The base station 2100 may render frames for display by the device 2150 including virtual content based at least in part on various information obtained from the sensors, encode the frames, and transmit the encoded frames to the device 2150 over the wireless connection 2180 for decoding and display to the user. To encode and transmit the frames, the base station 2100 may implement a video encoding system such as that shown in FIGS. 1-6.
[0080] FIG. 8 is a block diagram illustrating the functional components and processing of an exemplary VR / MR system as shown in FIG. 7, according to some embodiments. The device 2150 may be, but is not limited to, a notebook or laptop computer, a pad or tablet device, a smartphone, a handheld computing device, or an HMD, such as a headset, helmet, goggles, or glasses that may be worn by a user. The device 2150 may include a display 2156 component or subsystem that may implement any of various types of virtual or augmented reality display technologies. For example, the HMD device 2150 may be a near-eye system that displays the left and right images that the user sees on a screen in front of the user's eyes, such as VR systems with DLP (digital light processing), LCD (liquid crystal display), and LCoS (liquid crystal on silicon) technologies. As another example, the HMD device 2150 may be a direct retinal projection system that scans the left and right images pixel by pixel into the user's eyes. To scan the image, left and right projectors generate beams that are directed to left and right reflective elements (e.g., ellipsoidal mirrors) located in front of the user's eyes, which reflect the beams to the user's eyes. To create a three-dimensional (3D) effect, virtual content at different depths or distances in the 3D virtual view is shifted left or right in the two images, such that closer objects are shifted more than more distant objects, depending on the triangulation distance.
[0081] The device 2150 may also include a controller 2154 configured to implement the device-side functions of the VR / MR system 2000 as described herein. In some embodiments, the HMD 2150 may also include a memory 2170 configured to store software (code 2172) of the device components of the VR / MR system 2000 executable by the controller 2154, as well as data 2174 that may be used by the software when executing on the controller 2154. In various embodiments, the controller 2154 may be a uniprocessor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or other suitable number). The controller 2154 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. For example, in various embodiments, the controller 2154 may include a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. In a multiprocessor system, each of the processors may, but need not, commonly implement the same ISA. The controller 2154 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out-of-order, in-order, speculative, non-speculative, etc., or combinations thereof. The controller 2154 may include circuitry for implementing microcoding techniques. The controller 2154 may include one or more processing cores, each configured to execute instructions. The controller 2154 may include one or more levels of caches, which may be of any size and configuration (set associative, direct-mapped, etc.).In some embodiments, the controller 2154 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry. Generally, a GPU may be configured to render objects (e.g., including pixel data for an entire frame) to be displayed in a frame buffer. A GPU may include one or more graphics processors capable of executing graphics software that performs some or all of the graphics operations, or hardware acceleration of certain graphics operations. In some embodiments, the controller 2154 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder (codec), etc. In some embodiments, the controller 2154 may include at least one system on chip (SOC).
[0082] The memory 2170 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, or low power versions of SDRAM, such as LPDDR2), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single in-line memory module (SIMM), dual in-line memory module (DIMM), etc. Alternatively, the device may include integrated circuits that implement the system in a chip-on-chip, package-on-package, or multi-chip module configuration.
[0083] In some embodiments, the sensor 2160 may include, but is not limited to, one or more eye tracking sensors (e.g., an infrared camera with an infrared illumination source) that may be used to track the position and movement of the user's eyes. In some embodiments, there may be two eye tracking sensors, with each eye tracking sensor tracking a respective eye. In some embodiments, information collected by the eye tracking sensors may be used to adjust the rendering of images by the base station 2100 and / or to adjust the projection of images by the projection system of the device 2150 based on the direction and angle at which the user's eyes are looking. For example, in some embodiments, the content of the image in the area around where the user's eyes are currently looking may be rendered with more detail and higher resolution than the content in the area where the user is not looking. This allows available processing time for image data to be spent on the content seen by the foveal area of the eye rather than the content seen by the peripheral area of the eye. Similarly, the content of the image in the area where the user is not looking may be compressed more than the content of the area around where the user is currently looking. In some embodiments, there may be two eye tracking sensors located on the inner surface of the device 2150 in a position such that the sensor has a view of each of the user's eyes. However, in various embodiments, more or fewer eye tracking sensors may be used, and the eye tracking sensors may be located in other locations. In an exemplary non-limiting embodiment, each eye tracking sensor may include an infrared light source and an infrared camera, for example, a 400x400 pixel camera with a frame rate of 120 FPS or more, a horizontal field of view (HFOV) of 70 degrees, and a working distance of 10 to 80 millimeters (mm).
[0084] In some embodiments, the device 2150 may include at least one inertial-measurement unit (IMU) 2162 configured to detect a position, orientation, and / or movement of the device 2150 and provide the detected position, orientation, and / or movement data to the controller 2154 of the HMD 2150 and / or the base station 2100.
[0085] The device 2150 may also include a wireless interface 2152 configured to communicate with an external base station 2100 via a wireless connection 2180 to transmit sensor inputs to the base station 2100 and to receive rendered and compressed frames, slices, or tiles from the base station 2100. In some embodiments, the wireless interface 2152 may implement a proprietary wireless communication technology that provides a highly directional wireless link between the device 2150 and the base station 2100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.
[0086] The base station 2100 may be an external device (e.g., a computing system, a game console, etc.) communicatively coupled to the device 2150 via the wireless interface 2180. The base station 2100 may include one or more of various types of processors (e.g., SOC, CPU, ISP, GPU, codec, and / or other components) for rendering, filtering, encoding, and transmitting video and / or images. The base station 2100 may render frames (each frame including left and right images) including virtual content based at least in part on various inputs obtained from the sensors 2160 via the wireless connection 2180, filter and compress the rendered frames (or slices of frames) using a video encoding system as described herein, and transmit the compressed frames or slices to the device 2150 for display.
[0087] The base station 2100 may be or include any type of computing system or device, such as a desktop computer, a notebook or laptop computer, a pad or tablet device, a smartphone, a handheld computing device, a game controller, a gaming system, etc. The base station 2100 may include a controller 2110 with one or more processors that implement the base station side functions of the VR / MR system 2000, including the video encoding system as described herein. The base station 2100 may also include a memory 2120 configured to store software (code 2122) for the base station components of the VR / MR system 2000 executable by the base station controller 2110, as well as data 2124 that may be used by the software when executing on the controller 2110.
[0088] In various embodiments, the base station controller 2110 may be a uniprocessor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or other suitable number). The controller 2110 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. For example, in various embodiments, the controller 2110 may include a general purpose or embedded processor implementing any of a variety of ISAs, such as the x86, PowerPC, SPARC, RISC, or MIPS instruction set architectures (ISAs), or any other suitable ISA. In a multiprocessor system, each of the processors may, but need not, commonly implement the same ISA. The controller 2110 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out-of-order, in-order, speculative, non-speculative, etc., or combinations thereof. The controller 2110 may include circuitry for implementing microcoding techniques. The controller 2110 may include one or more processing cores, each configured to execute instructions. The controller 2110 may include one or more levels of caches, which may be of any size and configuration (set associative, direct-mapped, etc.). In some embodiments, the controller 2110 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry. Generally, a GPU may be configured to render objects (e.g., including pixel data for an entire frame) to be displayed in a frame buffer. A GPU may include one or more graphics processors, which may execute graphics software that performs some or all of the graphics operations, or hardware acceleration of certain graphics operations.In some embodiments, the controller 2110 may include one or more other components for processing, rendering, filtering, and encoding video and / or images as described herein, such as one or more of various types of integrated circuits (ICs), image signal processors (ISPs), encoders / decoders (CODECs), etc. In some embodiments, the controller 2110 may include at least one system on a chip (SOC).
[0089] The base station memory 2120 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, or low power versions of SDRAM, such as LPDDR2), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single in-line memory module (SIMM), dual in-line memory module (DIMM), etc. Alternatively, the device may include integrated circuits that implement the system in a chip-on-chip, package-on-package, or multi-chip module configuration.
[0090] The base station 2100 may also include one or more wireless technology interfaces 2130 configured to communicate with the devices 2150 via wireless connections 2180 to receive sensor inputs from the devices 2150 and transmit compressed frames, slices, or tiles from the base station 2100 to the devices 2150. In some embodiments, the wireless technology interfaces 2130 may implement a proprietary wireless communication technology that provides a highly directional wireless link between the devices 2150 and the base station 2100. In some embodiments, the directionality and bandwidth of the wireless communication technology may support multiple devices 2150 to communicate with the base station 2100 simultaneously, thereby enabling multiple users to use the system 2000 simultaneously in a co-located environment. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.
[0091] In some embodiments, the base station 2100 may be configured to render and transmit frames to the device 2150 to provide a 3D virtual view for the user based at least in part on sensor 2160 input received from the device 2150. In some embodiments, the virtual view may include a rendering of the user's environment, including renderings of real objects in the user's environment, based on video captured by one or more scene cameras (e.g., RGB (visible light) video cameras) that capture and display high quality, high resolution video of the user's environment in real time. In some embodiments, the virtual view may also include virtual content (e.g., virtual objects, virtual tags for real objects, the user's avatar, etc.) rendered by the base station 2100 and composited with the projected 3D view of the user's real environment.
[0092] Although not shown in Figures 7 and 8, in some embodiments, the VR / MR system 2000 may include one or more other components. For example, the system may include a cursor control device (e.g., a mouse) for moving a virtual cursor in the 3D virtual view to interact with the virtual content. Although Figures 7 and 8 show a single device 2150, in some embodiments, the VR / MR system 2000 may support multiple devices 2150 communicating with the base station 2100 simultaneously, thereby enabling multiple users to use the system simultaneously in a co-located environment.
[0093] The methods described herein may be implemented in different embodiments in the form of software, hardware, or a combination thereof. In addition, the order of blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and alterations may be made as would be apparent to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Thus, multiple instances may be provided for components described herein as a single instance. Boundaries between various components, operations, and data stores are somewhat arbitrary, and certain operations are illustrated in the context of specific example configurations. Other allocations of functionality are contemplated and may be included within the scope of the following claims. Finally, structures and functionality presented as separate components in the example configurations may be implemented as combined structures or components. These and other variations, modifications, additions, and improvements may be included within the scope of the embodiments, as defined in the following claims.
Claims
1. 1. A video encoding system, comprising: one or more processors; a memory, the memory storing program instructions that, when executed by the one or more processors, cause the one or more processors to: A wavelet transform component, applying a wavelet transform to each of a plurality of pixel blocks to decompose each of the plurality of pixel blocks into a plurality of frequency bands; Organizing the plurality of frequency bands into a plurality of frequency blocks, each frequency block including frequency data corresponding to a respective one of the plurality of frequency bands; a wavelet transform component configured to provide each of the frequency blocks as a separate stream, each corresponding to a respective one of the frequency bands, to two or more encoders; and two or more encoders, each of the two or more encoders comprising: configured to process the separate streams of frequency blocks; and wherein, to process the frequency blocks in a given stream, the respective one of the two or more encoders is further configured to apply an encoding technique to the frequency blocks to compress the frequency data in the frequency blocks; the processing of the frequency blocks in the given stream is multiplexed; A video encoding system configured to provide the compressed frequency data from the separate stream to a wireless interface for transmission to a receiving device over a wireless connection.
2. 2. The video encoding system of claim 1, wherein the multiplexed processing encodes multiple time units in each frequency block processing corresponding to a given frequency band.
3. 2. The video coding system of claim 1, wherein the multiplexed processing of the frequency blocks is based at least in part on a dependency of a second frequency block on a previous processing of a first frequency block.
4. the program instructions further cause the one or more processors to implement a pre-filter component; the prefilter component is configured to prefilter the pixel block, the pixel block being from a rendered video frame; and to pre-filter the pixel blocks, the pre-filter component is further configured to apply a filter to the rendered video frames to compensate for distortions that would be introduced at the receiving device. The video encoding system according to claim 1 .
5. the one or more processors: and further configured to receive, via the wireless connection, lens warp data for the lens of the receiving device, which is used by the pre-filter component to correct for the distortion to be introduced by the lens of the receiving device. The video encoding system according to claim 4.
6. To prefilter the pixel block, the prefilter component and further configured to apply a filter to peripheral regions of the rendered video frames to reduce resolution of the peripheral regions while maintaining higher resolution within a foveal region of the rendered video frames. The video encoding system according to claim 4.
7. 2. The video coding system of claim 1, wherein the applied wavelet transform is a two-level wavelet decomposition applied to each pixel block from a rendered video frame.
8. 2. The video coding system of claim 1, wherein the applied wavelet transform is an N-level wavelet decomposition applied to each pixel block from a rendered video frame, where N is adjustable.
9. To apply the wavelet transform, the wavelet transform component The video encoding system of claim 1 , further configured to apply a coring function to one or more wavelet coefficients of the wavelet decomposition.
10. 1. A method comprising: applying a wavelet transform to each of a plurality of pixel blocks to decompose each of the plurality of pixel blocks into a plurality of frequency bands; Organizing the plurality of frequency bands into a plurality of frequency blocks, each frequency block including frequency data corresponding to a respective one of the plurality of frequency bands; providing each of the frequency blocks to two or more encoders as separate streams, each corresponding to a respective one of the frequency bands; Processing the separate streams of frequency blocks by multiplexing, processing the frequency blocks in a given stream includes applying an encoding technique to the frequency blocks in a respective one of the two or more encoders to compress the frequency data in the frequency blocks; processing, wherein the processing of the frequency blocks in the given stream is multiplexed; and transmitting the compressed frequency data from the separate stream to a receiving device over a wireless connection; A method comprising:
11. The method of claim 10 , wherein the processing by multiplexing encodes multiple time units in the processing of each frequency block corresponding to a given frequency band.
12. The method of claim 10 , wherein the processing by the multiplexing is based at least in part on a dependency of a second frequency block relative to a first frequency block.
13. prefiltering the pixel blocks, the pixel blocks being from a rendered video frame; and pre-filtering the video frames to compensate for distortions that would be introduced at the receiving device. The method of claim 10.
14. receiving lens warp data for a lens of the receiving device; the applying of the filter to the rendered video frames to correct for the distortion introduced at the receiving device is based at least in part on the received lens warp data. The method of claim 13.
15. 1. A system comprising: a device including one or more processors and a display subsystem; 1. A transmitting device, comprising: One or more processors, applying a wavelet transform to each of a plurality of pixel blocks containing pixel data to decompose the pixel data into a plurality of frequency bands; Organizing the plurality of frequency bands into a plurality of frequency blocks, each frequency block including frequency data corresponding to a respective one of the plurality of frequency bands; one or more processors configured to provide each of the frequency blocks as a separate stream, each stream corresponding to a respective one of the frequency bands, to two or more encoders; each of the two or more encoders configured to process the separate stream of frequency blocks; to process the frequency blocks in a given stream, the respective one of the two or more encoders is further configured to apply an encoding technique to the frequency blocks in the given stream to compress the frequency band; the two or more encoders, wherein the processing of the frequency blocks in the given stream is multiplexed; a wireless interface configured to transmit the compressed frequency band to the device over a wireless connection; a transmitting device, the one or more processors of the device; Decompressing the compressed frequency band received from the transmitting device; performing wavelet synthesis on the restored frequency bands to reconstruct the pixel data for a video frame; The system is configured to provide the reconstructed pixel data to the display subsystem for display.
16. 16. The system of claim 15, wherein the multiplexed processing encodes multiple time units in the processing for each frequency block corresponding to a given frequency band.
17. 16. The system of claim 15, wherein the multiplexed processing of the frequency blocks is based at least in part on a dependency of a second frequency block on the processing of a first frequency block.
18. the one or more processors: and further configured to pre-filter the pixel blocks, the pre-filtering including applying a filter to correct for distortions that would be introduced in the display subsystem. The system of claim 15.
19. the one or more processors: and further configured to receive, via the wireless connection, lens warp data for the lens of the device, for use in the pre-filtering of the pixel blocks to correct for the distortion to be introduced by a lens of the device.
20. The system of claim 18.
20. The system of claim 15 , wherein the device is a head mounted display (HMD) device.
Citation Information
Patent Citations
Image processing method
JP1997212623A
Method and system for displaying image
JP2003202850A
Code conversion apparatus, code conversion method, program, and recording medium
JP2007318711A
Selective peripheral vision filtering in a foveated rendering system
US20170287112A1
Systems and methods for combined pipeline processing of panoramic images
US20170301065A1
