Rate control for low-delay video coding and transmission

Abstract

The present disclosure relates to rate control for low-delay video encoding and transmission. A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands of a frame's slices can be buffered so that complexity statistics values on the frequency bands can be computed prior to encoding. The statistics values can then be used by a rate control component to determine quantization parameters for the frequency bands for use in modulating the rate in the encoder for a current slice. The quantization parameters for the frequency bands can be jointly computed to optimize the quality of the displayed frame after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames can be combined with the statistics values for a current slice to determine the quantization parameters for the current slice.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980040176.0, application date June 27, 2019, entitled "Rate Control for Low-Latency Video Coding and Transmission". Background Technology

[0002] Virtual reality (VR) allows users to experience and / or interact with immersive artificial environments, making them feel as if they are actually in those environments. For example, a VR system can display stereoscopic scenes to create a depth illusion, and the computer can adjust the scene content in real time to provide the illusion that the user is moving within the scene. When a user views images through a VR system, they can thus feel as if they are moving within the scene from a first-person perspective. Similarly, mixed reality (MR) combines computer-generated information (called virtual content) with real-world images or views to enhance or add content to a user's world view, or alternatively, combine virtual representations of real-world objects with views of a three-dimensional (3D) virtual world. Therefore, the simulated environments of virtual reality and / or the hybrid environments of mixed reality can be used to provide interactive user experiences for a variety of applications. Summary of the Invention

[0003] Various implementations of a video encoding system are described, which can encode high-resolution video sources with low latency for transmission over a communication link (e.g., a wireless link) to a device for decoding and display. Implementations of this video encoding system can also provide moderate degradation of the encoded video transmitted to the device to maintain the desired frame rate under changing conditions, such as variations in the channel capacity of the communication link. An exemplary application of this video encoding system is in virtual or mixed reality systems, where video frames containing virtual content are rendered, encoded, and transmitted by a base station to a device (e.g., a laptop or laptop computer, tablet or tablet device, smartphone, or head-mounted display (HMD), such as a headset, helmet, goggles, or glasses that can be worn by a user) for decoding and display.

[0004] In some implementations, the video encoding system may perform a wavelet transform on the pixel data before encoding to decompose the pixel data into frequency bands. The frequency bands are then organized into blocks, which are provided to a block-based encoder for encoding / compression. The encoded frequency data is then sent to a wireless interface that groups the encoded frequency data and transmits the groups to a receiving device. At the receiving device, the encoded data is degrouped and passed through a block-based decoder to recover the frequency bands. Wavelet synthesis is then performed on the recovered frequency bands to reconstruct the pixel data for display.

[0005] This video encoding system enables various methods and apparatuses to maintain a target frame rate over a wireless link, minimize latency in frame rendering, transmission, and display, and provide appropriate degradation of the encoded video transmitted to the device to maintain the desired frame rate under changing conditions, such as variations in the channel capacity of the wireless connection.

[0006] The goal of an implementation of a video coding system can be to provide very low latency (e.g., 4-5 milliseconds) between the time virtual content is rendered at the base station and the time it is displayed on the device, while minimizing errors (e.g., dropped packets) and maximizing bandwidth utilization over the wireless connection. This implementation of the video coding system may implement rate control methods to help achieve the system's goal of low latency when encoding data to be transmitted to the device over the wireless connection.

[0007] In some implementations, the video coding system includes: a wavelet transform unit that performs a two-level wavelet decomposition, which decomposes pixel blocks from a strip of frames into multiple distinct frequency bands; and a block-based encoder, such as HEVC encoding the frequency band. In some implementations, to perform rate control under the stringent latency requirements of VR / MR systems, the rate control unit may adjust encoder parameters (e.g., quantization parameters of the frequency band) for each strip, rather than working on the time constant of one or two frames to adjust the encoder parameters. In some implementations, the frequency band of the strip can be buffered in local memory after the wavelet transform, allowing complexity statistics on that frequency band to be computed before encoding, for example, by a complexity analysis unit of the video coding system. In some implementations, the complexity statistics may include, but are not limited to, one or more of the pixel gradient and size variance of the CTU block. This complexity statistics can then be used by the rate control unit to determine the quantization parameters of the frequency band for modulating the rate in the encoder for the current strip. Computing complexity statistics on the buffered frequency band allows the video coding system to estimate the degree of complexity in the strip before encoding it.

[0008] In some implementations, the rate control unit may examine statistics from one or more previously processed frames to determine the actual number of bits used to encode the co-sited block or stripe of that frame. This information may then be combined with complexity statistics of the current stripe to determine the quantization parameters for the current stripe.

[0009] In some implementations, the quantization parameters of the frequency bands can be jointly computed to optimize the overall display quality of the frame after decoder reconstruction and wavelet synthesis are performed on-device. A method for rate allocation between frequency bands is to find operating points in each band that result in equal slopes on the operating rate distortion curve. In some implementations, a combination of statistics from the entropy coding stage of the encoder pipeline can be used to estimate the operating rate distortion curve, where the rate of the coefficients can be very close to the quantization parameter step size.

[0010] In some implementations, a region of interest (ROI) may exist within an image (e.g., a concave region) that should have a higher resolution than surrounding regions (e.g., peripheral regions) in the same image. In some implementations, complexity statistics from previous frames, along with concave regions from previous frames, may be stored and used to calculate quantization parameters for bands in the current frame's stripes. For example, for each block to be encoded, the concave region of a previous frame can be used to modulate the complexity statistics calculated for the previous frame. The resulting complexity statistics can then be used, in addition to those calculated for the current frame's stripes, to determine the quantization parameters.

[0011] In some implementations, the rate control unit can adjust encoder parameters (e.g., quantization parameters for bands or priorities) at the block level rather than the strip level. In some implementations, complexity statistics for each block of the strip can be calculated, and the quantization parameters for that block can be determined. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating a video coding system that uses wavelet transform to decompose pixel data into frequency bands prior to encoding, according to at least some embodiments.

[0013] Figure 2 A video coding system comprising multiple encoders that process tiles from a frame in parallel, according to at least some embodiments, is shown.

[0014] Figure 3A Exemplary frames divided into slices and tiles according to at least some embodiments are shown.

[0015] Figure 3B An exemplary block diagram divided into blocks according to at least some embodiments is shown.

[0016] Figure 3C Wavelet transform is performed on pixel blocks storing pixel data according to at least some embodiments to generate frequency band data before encoding.

[0017] Figure 4 It is based on at least some implementation schemes for such Figure 1and Figure 2 The diagram shown is a high-level flowchart illustrating the operation of a VR / MR system, including a video encoding system.

[0018] Figure 5 It is based on at least some implementation schemes for such Figure 1 The flowchart shows the operation method of the video encoding system.

[0019] Figure 6 It is based on at least some implementation schemes for such Figure 2 The flowchart shows the operation method of the video encoding system.

[0020] Figure 7 It is based on at least some of the embodiments shown, such as Figure 1 or Figure 2 The block diagram of the video coding system shown in the figure is used to calculate the complexity statistics of the frequency band and generate quantization parameters for coding.

[0021] Figure 8 It is based on at least some implementation schemes for such Figure 7 The flowchart shows the operation method of the video encoding system.

[0022] Figure 9 An exemplary VR / MR system is shown, which implements a video encoding system according to at least some of the embodiments.

[0023] Figure 10 It is based on at least some of the embodiments shown, such as Figure 9 The diagram shows the components of the VR / MR system.

[0024] This specification includes references to "one embodiment" or "implementation". The use of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. Specific features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0025] The term "comprising" is open-ended. As used in the claims, it does not exclude additional structures or steps. Consider the following cited claim: "An apparatus comprising one or more processor units..." Such claims do not exclude the inclusion of additional components (e.g., network interface units, graphics circuitry, etc.).

[0026] "Configured as" refers to various units, circuits, or other components that can be described or stated as being "configured as" to perform one or more tasks. In such a context, "configured as" is used to imply a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs this one or more tasks during operation. Thus, a unit / circuit / component is allegedly configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not switched on). Units / circuits / components used with the language "configured as" include hardware—e.g., circuits, memory storing program instructions that can be executed to perform the operation, etc. The reference to a unit / circuit / component being "configured as" to perform one or more tasks is explicitly intended to exclude paragraph 6 of 35 U.S.SC §112 for that unit / circuit / component. Furthermore, "configured as" can include general structures (e.g., general circuits) manipulated by software or firmware (e.g., an FPGA or a general-purpose processor executing software) in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adjusting the manufacturing process (e.g., a semiconductor manufacturing facility) to manufacture equipment (e.g., an integrated circuit) suitable for performing one or more tasks.

[0027] "First," "second," etc. As used herein, these terms serve as labels for the nouns preceding them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations on a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0028] The terms "based on" or "depending on," as used herein, are used to describe one or more factors that influence a determination. These terms do not exclude additional factors that may influence the determination. That is, a determination may be based solely on these factors or at least in part on them. Consider the phrase "A is determined based on B." In this case, B is a factor influencing the determination of A, and such phrases do not exclude the possibility that the determination of A may also be based on C. In other instances, A may be determined solely on B.

[0029] When used in the claims, the term "or" is interpreted as an inclusive "or," not an exclusive "or." For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, and any combination thereof. Detailed Implementation

[0030] Various implementations of a video coding system are described. These implementations can encode high-resolution video sources with low latency for transmission to a device via a communication link (e.g., a wireless link) for decoding and display. Furthermore, these implementations can provide moderate degradation of the encoded video transmitted to the device to maintain the desired frame rate under varying conditions, such as changes in the channel capacity of the communication link.

[0031] An exemplary application of this video encoding system is in virtual or mixed reality systems, where video frames containing virtual content are rendered, encoded, and transmitted to a device for decoding and display. Implementations of virtual or mixed reality systems (referred to herein as VR / MR systems) are described, in which implementations of the video encoding system can be implemented. In some implementations, the VR / MR system may include a device (e.g., a tablet or tablet device, a smartphone, or a headset, helmet, goggles, or glasses worn by a user, referred to herein as a head-mounted display (HMD)) and a separate computing device, referred to herein as a base station. In some implementations, the device and the base station may each include wireless communication technology that allows the device and the base station to communicate and exchange data via a wireless connection. In some implementations, the device may include sensors that collect information about the user's environment (e.g., video, depth information, lighting information, etc.) and information about the user (e.g., the user's facial expressions, eye movements, gestures, etc.). The information collected by the sensors may be transmitted to the base station via a wireless connection. The base station may include software and hardware (e.g., processors (system-on-chip (SOC), CPU, image signal processor (ISP), graphics processing unit (GPU), encoder / decoder (codec)), memory, etc.) configured to generate and render frames including virtual content based at least in part on sensor information received from the device via a wireless connection. The base station may also include embodiments of a video encoding system as described herein, which pre-filters, compresses, and transmits the rendered frames to the device for display via a wireless connection.

[0032] In some implementations, VR / MR systems may utilize proprietary wireless communication technologies that provide a highly directional wireless link between devices and base stations. In some implementations, the directionality and bandwidth of the wireless communication technology can support multiple devices communicating simultaneously with the base station, enabling multiple users to utilize the system simultaneously in a cooperative positioning environment. However, in some implementations, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be supported.

[0033] The main constraints to consider on wireless links include bandwidth and latency. The goal of VR / MR systems is to provide high-resolution, wide-field-of-view (FOV) virtual displays at frame rates to offer users a high-quality VR / MR view. Another objective is to minimize the latency between the time the base station renders a frame and the time the device displays it. However, the channel capacity of a wireless link can vary over time, and therefore the wireless link may only support a specific amount of information transmitted at any given time.

[0034] This paper describes various methods and apparatuses that can be implemented by video coding systems to maintain a target frame rate over a wireless link and minimize latency in frame rendering, transmission, and display. Furthermore, these methods and apparatuses can provide moderate degradation of the encoded video transmitted to the device to maintain the desired frame rate under changing conditions, such as variations in the channel capacity of the communication link.

[0035] In some implementations, the video coding system may perform wavelet transform on the pixel data before encoding to decompose the pixel data into frequency bands. The frequency bands are then organized into blocks, which are provided to a block-based encoder for encoding / compression. As an example, a frame can be divided into 128×128 blocks, and a two-level wavelet decomposition can be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing seven frequency bands. These frequency data blocks can then be sent to an encoder (e.g., a High Efficiency Video Coding (HEVC) encoder) for encoding. The encoded frequency data is then sent to a wireless interface that groups the encoded frequency data and transmits the groups to a receiving device (e.g., an HMD, tablet or tablet device, smartphone, etc.). At the receiving device, the encoded data is degrouped and passed through a block-based decoder to recover the frequency bands. Wavelet synthesis is then performed on the recovered frequency bands to reconstruct the pixel data for display.

[0036] While the general description describes an implementation in which the wavelet transform is applied to a two-level wavelet decomposition of each pixel block from a video frame, in various implementations the wavelet decomposition can be any number of levels (e.g., one level, two levels, three levels, four levels, etc.) and can be adjusted to balance the quality of the encoded image with the complexity of the blocks to be encoded.

[0037] In some implementations, video coding systems can perform stripe-based rendering, encoding, and transmission. Rendering, encoding, and transmitting an entire frame can have latency and memory impacts because each frame needs to be completed, stored, and then transmitted to the next stage of the VR / MR system. In stripe-based rendering, instead of rendering and encoding the entire frame at the base station and transmitting the rendered frame to the device, a stripe can be rendered and encoded as a portion of the frame (called a stripe) and transmitted to the device when the encoded stripe is ready. A stripe can be, for example, a single 128×128 block, or two or more line blocks. Stripe-based rendering and encoding can help reduce latency and also reduce the amount of memory required for buffering, which reduces the memory footprint and power requirements on the chip or processor.

[0038] In some implementations, the video encoding system can perform tile-based rendering, encoding, and transmission. In tile-based rendering, encoding, and transmission, each stripe can be divided into multiple tiles (e.g., four tiles), and the base station can render and encode the tiles and transmit them to the device when the encoded tiles are ready.

[0039] In some implementations, a video coding system may use a single encoder to perform tile-based coding to process tiles from each stripe. However, in other implementations, a video coding system may use multiple encoders to perform tile-based coding to process corresponding tiles from each stripe. For example, in some implementations, each stripe may be divided into four tiles, each tile comprising multiple 128×128 blocks, and two encoders (e0 and e1) may operate on two tiles from each stripe (e.g., e0 operates on t0 and t1; e1 operates on t2 and t3). Each encoder may multiplex the processing of blocks from different frequency bands between its two tiles to allow for 16 time units between the processing of blocks from the same frequency band. By multiplexing the processing of blocks between two tiles, dependencies between blocks in a frequency band can be appropriately handled. Although an embodiment in which each strip is divided into four blocks and two encoders operate on the corresponding blocks from each strip is described, in some embodiments the strip may be divided into more blocks (e.g., six or eight blocks), and in some embodiments more encoders (e.g., three or four encoders) may be used.

[0040] In some implementations, the video coding system may perform pre-filtering on the pixel data in the frame before wavelet transform. For example, pre-filtering can reduce the resolution of the frame rendered by the base station before performing wavelet transform, encoding, and frame transmission over the wireless link to the device. This can help improve compression and reduce latency and bandwidth usage on the wireless link.

[0041] In some implementations where the device is an HMD, pre-filtering may include performing lens warping on the frame at the base station before wavelet transform. This lens warping corrects image distortion introduced by the lens on the HMD (through which the image is viewed), thereby improving image quality. In some implementations, the HMD may store lens warping data, for example, generated through a calibration process, and may provide this data to the base station via a wireless connection. The base station can then perform lens warping on the frame based on this HMD lens warping data. In conventional VR / MR systems, lens warping is performed on the HMD after decoding and before display. Performing lens warping on the base station during the pre-filtering stage reduces the frame resolution before wavelet transform and encoding, which can help improve compression and reduce latency and bandwidth usage on the wireless link. Furthermore, by performing lens warping on the base station during the pre-filtering stage instead of on the HMD after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on the base station before encoding to reduce resolution and then performing lens warping filtering on the HMD.

[0042] In some implementations, pre-filtering may include filtering to reduce resolution in the peripheral region while maintaining higher resolution in the foveal region. In this method, gaze tracking information obtained from the device can be used to identify the direction the user is currently looking. The human eye perceives higher resolution in the foveal region than in the peripheral region. Therefore, the region of the frame corresponding to the fovea (referred to as the foveal region) can be identified at least in part based on the determined gaze direction. In some implementations, the peripheral region (i.e., the portion of the frame outside the foveal region) may be pre-filtered based on knowledge of the human visual system to reduce information, for example, by filtering high-frequency information and / or increasing color compression. In some implementations, the amount of filtering applied to the peripheral region may extend towards the periphery of the frame. Pre-filtering of the peripheral region can contribute to providing improved frame compression.

[0043] Figure 1 This is a block diagram illustrating a video coding system 120 that decomposes pixel data into frequency bands using wavelet transform prior to encoding, according to at least some embodiments. The VR / MR system 10 may include at least one device 150 (e.g., a tablet or tablet device, smartphone, or HMD, such as a headset, helmet, goggles, or glasses that can be worn by a user) and a computing device 100 (referred to herein as a base station). The base station 100 renders VR or MR frames including virtual content, encodes the frames, and transmits the encoded frames to the device 150 via a wireless connection 180 for decoding and display.

[0044] In some implementations, device 150 may include sensors 160 that collect information about the environment of user 190 (e.g., video, depth information, lighting information, etc.) and information about user 190 (e.g., user facial expressions, eye movements, gaze direction, gestures, etc.). Device 150 may transmit at least some of the information collected by sensors 160 to base station 100 via wireless connection 180. Base station 100 may render frames, including virtual content, for display by device 150 based at least in part on the various information obtained from sensors 160, encode the frames, and transmit the encoded frames to device 150 via wireless connection 180 for decoding and display to user.

[0045] Base station 100 and device 150 can implement wireless communication technologies that allow base station 100 and device 150 to communicate and exchange data via wireless connection 180. In some embodiments, wireless connection 180 may be implemented based on proprietary wireless communication technologies that provide a highly directional wireless link between device 150 and base station 100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0046] Key constraints to consider on the wireless connection 180 between device 150 and base station 100 in VR / MR system 10 include bandwidth and latency. For example, in some implementations, the goal is to provide a high-resolution, wide-field-of-view (FOV) virtual display to the user at a frame rate that delivers a high-quality VR / MR view. Another goal is to minimize the latency between the time it takes for the device to capture video frames and the time it takes for the device to display rendered VR / MR frames based on those video frames.

[0047] Base station 100 may include various hardware components as described herein for rendering, filtering, encoding, and transmitting video and / or images, such as various types of processors, integrated circuits (ICs), central processing units (CPUs), graphics processing units (GPUs), image signal processors (ISPs), encoders / decoders, etc. Base station 100 may include, but is not limited to, a GPU rendering component 110, a wireless interface component 130, and a video encoding system 120, which may include one or more hardware components implementing various methods that can help maintain a target frame rate over wireless connection 180 and minimize latency in frame rendering, encoding, transmission, and display. Video encoding system 120 may include, but is not limited to, a pre-filter component 122 (e.g., an N-channel filter bank), a wavelet transform component 124, and an encoder component 126.

[0048] GPU rendering 110 may include one or more hardware components capable of rendering frames for display by device 150, the hardware components including virtual content based at least in part on various information obtained from sensor 160.

[0049] In some implementations, the video encoding system 120 may include one or more hardware components that pre-filter 122 the pixel data in the rendered frame before performing wavelet transform 124. Pre-filter 122 may, for example, reduce the resolution of the frame rendered on base station 100 before performing wavelet transform 124, encoding 126, and transmission to device 150 via wireless connection 180. This can help improve compression and reduce latency and bandwidth usage on wireless connection 180.

[0050] In some implementations, pre-filter 122 may perform lens warping on frames at base station 100 before wavelet transform 124. Lens warping corrects image distortion introduced by lenses on the device (through which the image is viewed), thereby improving image quality. In some implementations, device 150 may store lens warping data, for example, generated through a calibration process, and may provide the lens warping data to base station 100 via wireless connection 180. The pre-filter 122 component of video encoding system 120 may then perform lens warping on frames based on the lens warping data from device 150. In conventional VR / MR systems, lens warping is performed on device 150 after decoding and before display. Performing lens warping on base station 100 during pre-filter 122 stage can reduce frame resolution before performing wavelet transform 124 and encoding 126, which can help improve compression and reduce latency and bandwidth usage on wireless connection 180. Furthermore, by performing lens warping on base station 100 in the pre-filter 122 stage instead of on device 150 after decoding, filtering of image data may only need to be performed once, which is the opposite of performing filtering on base station 100 before encoding 126 to reduce resolution and then performing lens warping filtering on device 150.

[0051] In some embodiments, pre-filter 122 may also apply one or more filters to reduce resolution in the peripheral region while maintaining higher resolution in the foveal region. In this method, gaze tracking information obtained from device 150 can be used to identify the direction the user is currently looking. The human eye perceives higher resolution in the foveal region than in the peripheral region. Therefore, the region of the frame corresponding to the fovea (referred to as the foveal region) can be identified at least in part based on the determined gaze direction. In some embodiments, information in the peripheral region (i.e., the portion of the frame outside the foveal region) may be reduced based on knowledge of the human visual system, for example, by filtering high-frequency information and / or increasing color compression. In some embodiments, the amount of filtering applied to the peripheral region may extend towards the periphery of the frame. Pre-filtering of the peripheral region may contribute to improved frame compression.

[0052] In some implementations, the wavelet transform 124 component of the video coding system 120 may include one or more hardware components (e.g., an N-channel filter bank) that perform a wavelet transform on the pixel data prior to encoding to decompose the pixel data into frequency bands. The frequency bands are then organized into blocks, which are provided to a block-based encoder 126 for encoding / compression. As an example, such as Figures 3A to 3C As shown, a frame can be divided into 128×128 blocks, and a two-level wavelet decomposition can be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing seven frequency bands. These frequency data blocks can then be sent to a block-based encoder (e.g., a High Efficiency Video Coding (HEVC) encoder) 126 for encoding. The encoded frequency data is then sent to a wireless interface 130 implemented by one or more hardware components, which packets the data and transmits the packets to device 150 via a wireless connection 180.

[0053] Device 150 may include various hardware components as described herein for decoding and displaying video and / or images, such as various types of processors, integrated circuits (ICs), central processing units (CPUs), graphics processing units (GPUs), image signal processors (ISPs), encoders / decoders, etc. Device 150 may include, but is not limited to, a wireless interface 152, a decoder 154 component (e.g., a High Efficiency Video Coding (HEVC) decoder), a wavelet synthesis 156 component, and a display 158 component. On device 150, wireless interface 152 receives packets transmitted by base station 100 via wireless connection 180. Encoded data is depacketized and passed through block-based decoder 154 (e.g., a High Efficiency Video Coding (HEVC) decoder) to recover frequency bands. Wavelet synthesis 156 is then performed on the recovered frequency data to reconstruct pixel data for display 158.

[0054] In some implementations, the video encoding system 120 can perform stripe-based rendering, encoding, and transmission. Rendering, encoding, and transmitting an entire frame can have latency and memory impacts because each frame needs to be completed, stored, and then transmitted to the next stage of the VR / MR system 10. In stripe-based rendering, instead of rendering and encoding the entire frame in base station 100 and transmitting the rendered frame to device 150, base station 100 can render and encode portions of a frame (called stripes) and transmit the encoded stripe to device 150 when it is ready. A stripe can be, for example, a 128×128 block. Stripe-based rendering and encoding can help reduce latency and also reduce the amount of memory required for buffering, which reduces memory footprint and power requirements on the chip or processor.

[0055] In some implementations, the video encoding system 120 can perform tile-based rendering, encoding, and transmission. In tile-based rendering, encoding, and transmission, each stripe can be divided into multiple tiles (e.g., four tiles), and the base station 100 can render and encode the tiles and transmit them to the device 150 when the encoded tiles are ready.

[0056] In some implementations, the video coding system 120 may use a single encoder 126 to perform tile-based rendering, encoding, and transmission to process tiles from each stripe. However, in some implementations, the video coding system 120 may use multiple encoders 126 to perform tile-based encoding to process the corresponding tiles from each stripe. Figure 2 A video coding system 220 is shown according to at least some embodiments, which includes multiple encoders (two encoders 226A and 226B in this example) that process tiles from a rendered frame in parallel.

[0057] The GPU rendering component 210 of base station 200 may include one or more GPUs and / or rendering frames (or frame stripes) for use in displaying other components. Frames may be divided into stripes, for example, Figure 3A As shown. Figure 3A As shown, each strip can be divided into multiple tiles (four in this example), and each tile includes multiple blocks. Figure 3B An exemplary tile is shown, comprising four 128×128 blocks. However, in some embodiments, other block sizes (e.g., 64×64, 32×32, etc.) may be used, and the tile may include more or fewer blocks.

[0058] Then, the pre-filter 222 and wavelet transform 224 components of the video coding system 220 can process each tile before encoding 226. In some embodiments, the video coding system 220 may include separate pre-filter 222 and wavelet transform 224 components for processing each tile. In this example, pre-filter 222A and wavelet transform 224A components process tile 0, pre-filter 222B and wavelet transform 224B components process tile 1, pre-filter 222C and wavelet transform 224C components process tile 2, and pre-filter 222D and wavelet transform 224D components process tile 3. The pre-filter 222 component performs pre-filtering of the tile as described herein, and the wavelet transform 224 component decomposes the tile into frequency bands as described herein. However, in some embodiments, the video coding system 220 may include a single pre-filter 222 component and a single wavelet transform 224 component for processing the tile. In some implementations, the video coding system 220 may include multiple (e.g., 2) pre-filter 222 components and multiple (e.g., 2) wavelet transform 224 components, each wavelet transform component processing multiple (e.g., 2) tiles.

[0059] Two encoders, 226A and 226B, can operate on two blocks from each strip (e.g., encoder 226A operates on blocks 0 and 1; encoder 226B operates on blocks 2 and 3). Each encoder 226 can multiplex blocks from different frequency bands between its two blocks (i.e., in...). Figure 3C The processing of 16 32×32 blocks (shown in the diagram) allows for 16 time units between processing blocks from the same frequency band. By multiplexing the processing of blocks between two blocks, dependencies between blocks in the same frequency band can be appropriately handled.

[0060] Although an embodiment in which each strip is divided into four blocks and two encoders operate on the corresponding blocks from each strip is described, in some embodiments the strip may be divided into more blocks (e.g., six or eight blocks), and in some embodiments more encoders may be used (e.g., three, four or more encoders).

[0061] Figure 3C The illustration shows the performance of wavelet transform on a pixel block storing pixel data to generate frequency band data prior to encoding, according to at least some embodiments. In this example, a two-level wavelet decomposition is applied to a 128×128 pixel block 300 via wavelet transform 324 to generate sixteen 32×32 frequency data blocks 302 representing seven frequency bands. The frequency blocks 302 are then provided to encoder 326 for encoding. For example, the frequency blocks 302 can be written to a buffer by wavelet transform 324 and read from the buffer by encoder 326.

[0062] In the label of frequency block 302, the letter L indicates a low-pass filter, and the letter H indicates a high-pass filter. Block 302 labeled with two letters indicates a first-order (2D) wavelet transform or decomposition. In block 302 labeled with two letters (representing three of the seven frequency bands LH, HL, and HH), the first letter indicates the vertical filter (high or low) performed first, and the second letter indicates the horizontal filter (high or low) performed second. Block 302 labeled with four letters indicates a second-order wavelet transform or decomposition. In block 302 labeled with four letters, the first two letters (LL) indicate that a vertical low-pass filter is present first, followed by a horizontal low-pass filter; the last two letters indicate that the resulting LL block is then subjected to four-way filtering, LL, LH, HL, and HH (thus representing four of the seven frequency bands (LLLL, LLLH, LLHL, and LLHH)).

[0063] Decompose pixel data into such Figure 3CThe frequency band shown allows encoder 326 to buffer and process the frequency band as a separate stream. Processing the frequency band as a separate stream allows encoder 326 components to multiplex the processing of independent streams. In block-based coding methods such as HEVC coding, blocks (called coding tree units (CTUs)) are processed in a block processing pipeline at multiple stages; two or more blocks may be at different stages of the pipeline at a given clock cycle, and the block moves through the pipeline with that clock cycle. The processing of a given block may depend on one or more previously processed adjacent blocks, such as one or more blocks in the row above the given block and / or blocks to the left of the given block. By multiplexing the processing of the frequency band data stream, encoder 326 spaces the processing of blocks in a given stream, thereby providing additional clock cycles to process the adjacent blocks that the given block depends on. For example, when a given block reaches a stage that depends on a previously processed adjacent block, a block to the left of the given block may be several stages ahead of the given block in the pipeline. This allows encoder 326 to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for the processing of adjacent blocks in the pipeline to complete before processing a given block at a stage that depends on adjacent blocks.

[0064] In addition, such as Figure 3C As shown, decomposing pixel data into frequency bands allows for band prioritization by the encoder 326 and the wireless interface. Typically, in image and video transmission, lower frequencies are more important, while higher frequencies are less important. Higher frequencies often correspond to details in the image and are therefore considered lower priority. Higher frequency bands contain a smaller percentage of the energy in the image. Most of the energy is contained in lower frequency bands. Therefore, decomposing pixel data into frequency bands provides a prioritization of the data stream, which the encoder 326 and the wireless interface can utilize when encoding and transmitting the data stream. For example, in some implementations, different compression techniques can be used on different frequency bands, with more aggressive compression applied to lower priority bands and more conservative compression applied to higher priority bands. As another example, frequency band prioritization can help provide a degree of degradation for VR / MR systems. The performance of the wireless connection can be monitored, and feedback from the device can be taken into account to track the performance of the entire system. If the system falls behind for some reason, such as if the wireless connection is degraded and the bandwidth capacity of the wireless connection drops below a threshold, the encoder 326 and the wireless interface may prioritize the encoding and transmission of one or more lower frequency bands in the lower frequency bands, and may reduce or decrease the encoding and / or transmission of one or more frequency levels in the frequency levels that have been assigned a lower priority level (e.g., one or more higher frequency bands in the higher frequency bands).

[0065] As described above, wavelet transform decomposes an image into frequency bands. In some implementations, this can be used to send the same signal to displays with different resolutions. As an example, suppose a second-order wavelet decomposition is applied to decompose the signal into seven frequency bands. If four of the frequency bands (LLLL, LLLH, LLHL, and LLHH) are sent, the bands can be reconstructed to the initial desired resolution with less visual quality. Alternatively, the frequency bands can also be reconstructed at 1 / 4 resolution (1 / 2 of each dimension), which is suitable for display panels with smaller display resolutions.

[0066] Figure 4 It is based on at least some implementation schemes for such Figure 1 and Figure 2 The diagram shows a high-level flowchart of the operation of a VR / MR system, including a video encoding system. As indicated at 400, the device transmits data to a base station via a wireless connection. As indicated at 410, the base station renders frames including virtual content based at least in part on the device data. As indicated at 420, the base station compresses the rendered data and transmits the compressed data to the device via a wireless connection. As indicated at 430, the device decompresses and displays the virtual content to generate a 3D virtual view for the user to view. As indicated by the arrow returning from 430 to 400, the method continues as long as the user is using the VR / MR system.

[0067] In some implementations, instead of rendering and encoding the entire frame at the base station and transmitting the rendered frame to the device as the encoded stripe is ready, a stripe can be rendered and encoded in the base station. A stripe can be, for example, a line of 128×128 blocks. In some implementations, the video coding system can perform tile-based rendering, encoding, and transmission. In tile-based rendering, encoding, and transmission, each stripe can be divided into multiple tiles, each tile comprising one or more blocks (e.g., four tiles, each tile comprising four blocks), and the base station can render and encode the tiles and transmit them to the device as the encoded tiles are ready.

[0068] Figure 5 It is based on at least some implementation schemes for such Figure 1 The flowchart shows the operation method of the video encoding system. Figure 5 The method can be, for example, in Figure 4 420 executions. Figure 5 The method assumes that stripe-based encoding and transmission are being performed. However, in some implementations, tile-based encoding and transmission can be performed.

[0069] As indicated at 510, the pre-filtering component applies lens warping and / or concave filtering to pixel blocks in the stripe of a frame. In some embodiments, pre-filtering may include performing lens warping on the frame at the base station before wavelet transform. Performing lens warping corrects image distortion introduced by the lens on the device (through which the image is viewed), thereby improving image quality. In some embodiments, the device may store lens warping data, for example, generated by a calibration process, and may provide the lens warping data to the base station via a wireless connection. The base station may then perform lens warping on the frame based on the lens warping data from the device. Performing lens warping on the base station during the pre-filtering stage can reduce the resolution of the frame before performing wavelet transform and encoding, which can help improve compression and reduce latency and bandwidth usage on the wireless link. Furthermore, by performing lens warping on the base station during the pre-filtering stage rather than on the device after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on the base station before encoding to reduce resolution and then performing lens warping filtering on the device.

[0070] In some implementations, the pre-filtering at 510 may further include filtering to reduce resolution in the peripheral region while maintaining higher resolution in the foveal region. In some implementations, gaze tracking information obtained from the device can be used to identify the direction the user is currently looking. The region of the frame corresponding to the fovea (referred to as the foveal region) can be identified at least in part based on the determined gaze direction. Pre-filtering of the peripheral region (i.e., the portion of the frame outside the foveal region) can reduce information based on knowledge of the human visual system, for example, by filtering high-frequency information and / or increasing color compression. Pre-filtering of the peripheral region can help provide improved frame compression.

[0071] As indicated at 520, the wavelet transform component applies wavelet transform techniques to pixel blocks to decompose the pixel data into N (e.g., 7) frequency bands. These frequency bands are then organized into blocks, which are provided to a block-based encoder for encoding / compression. As an example, a frame can be divided into 128×128 blocks, and a two-level wavelet decomposition can be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing the seven frequency bands, for example... Figure 3C As shown.

[0072] As indicated at 530, the encoder applies encoding techniques to the frequency bands within a block to compress the data. The encoder may be, for example, a High-Efficiency Video Coding (HEVC) encoder. However, in some implementations, other encoding techniques may be used. As indicated at element 520, breaking down pixel data into frequency bands allows the encoder to buffer and process the frequency bands as separate streams. Processing frequency bands as separate streams allows encoder components to multiplex the processing of independent streams. In block-based encoding methods such as HEVC encoding, blocks (called coding tree units (CTUs)) are processed in a pipeline at multiple stages; two or more blocks may be at different stages of the pipeline at a given clock cycle, and the block moves through the pipeline with that clock cycle. The processing of a given block may depend on one or more previously processed adjacent blocks, such as one or more blocks in the row above the given block and / or the block to the left of the given block. By multiplexing the processing of the streams, the encoder spaces the processing of blocks in a given stream, thereby providing additional clock cycles to process the adjacent blocks that the given block depends on. For example, when a given block reaches a stage that depends on a previously processed adjacent block, the block to the left of the given block may be several stages ahead of the given block in the pipeline. This allows the encoder to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for the processing of adjacent blocks in the pipeline to complete before processing the given block at a stage that depends on adjacent blocks.

[0073] As indicated at 540, the wireless interface packets the compressed data and transmits the packets to the device via a wireless connection.

[0074] As indicated at element 520, decomposing pixel data into frequency bands allows for band prioritization by the encoder at element 530 and by the wireless interface at element 540. Generally, in image and video transmission, lower frequencies are more important than higher frequencies. Higher frequencies typically correspond to details in the image and are therefore considered lower priority. Higher frequency bands contain a smaller percentage of the energy in the image. Most of the energy is contained in lower frequency bands. Therefore, decomposing pixel data into frequency bands provides a prioritization of the data stream, which the encoder and wireless interface can utilize when encoding and transmitting the data stream. For example, in some implementations, different compression techniques can be used on different frequency bands, with more aggressive compression applied to lower priority bands and more conservative compression applied to higher priority bands. As another example, frequency band prioritization can help provide a degree of degradation for VR / MR systems. The performance of the wireless connection can be monitored, and feedback from the device can be taken into account to track the performance of the entire system. If the system falls behind for some reason, such as if the wireless connection is degraded and the bandwidth capacity of the wireless connection drops below a threshold, the encoder and wireless interface may prioritize the encoding and transmission of one or more lower frequency bands in the lower frequency bands, and may reduce or decrease the encoding and / or transmission of one or more frequency levels in the frequency levels that have been assigned a lower priority level (e.g., one or more higher frequency bands in the higher frequency bands).

[0075] At 550, if there are more slices to be encoded and transmitted, the method returns to element 510 to process the next slice. Otherwise, at 560, if there are more frames to be encoded and transmitted, the method returns to element 510 to begin processing the next frame.

[0076] Figure 6 It is based on at least some implementation schemes for such Figure 2 The flowchart shows the operation method of the video encoding system. Figure 6 The method can be, for example, in Figure 4 Executed at 420 locations. Figure 6 In this method, the video coding system can use multiple encoders to perform tile-based coding to process the corresponding tiles from each strip.

[0077] As indicated at 600, the rendering engine renders stripes consisting of multiple tiles (four tiles in this example), each tile comprising multiple pixel blocks (four 128×128 pixel blocks in this example).

[0078] As indicated at 610, the pre-filtering component applies lens warping and / or concave filtering to the stripe. In some embodiments, pre-filtering may include performing lens warping on the frame at the base station before wavelet transform. Performing lens warping corrects image distortion introduced by the lens on the device (through which the image is viewed), thereby improving image quality. In some embodiments, the device may store lens warping data, for example, generated by a calibration process, and may provide the lens warping data to the base station via a wireless connection. The base station may then perform lens warping on the frame based on the lens warping data from the device. Performing lens warping on the base station during the pre-filtering stage can reduce the resolution of the frame before performing wavelet transform and encoding, which can help improve compression and reduce latency and bandwidth usage on the wireless link. Furthermore, by performing lens warping on the base station during the pre-filtering stage rather than on the device after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on the base station before encoding to reduce resolution and then performing lens warping filtering on the device.

[0079] In some embodiments, pre-filtering at 610 may further include filtering to reduce resolution in the peripheral region while maintaining higher resolution in the foveal region. In some embodiments, gaze tracking information obtained by the device can be used to identify the direction the user is currently looking. The region of the frame corresponding to the fovea (referred to as the foveal region) can be identified at least in part based on the determined gaze direction. Pre-filtering of the peripheral region (i.e., the portion of the frame outside the foveal region) can reduce information based on knowledge of the human visual system, for example, by filtering high-frequency information and / or increasing color compression. Pre-filtering of the peripheral region can help provide improved frame compression.

[0080] In some embodiments, the video coding system may include a single pre-filtering unit for processing tiles. In some embodiments, the video coding system may include a separate pre-filtering unit for processing each tile. In some embodiments, the video coding system may include multiple pre-filtering units, each processing multiple (e.g., two) tiles.

[0081] As indicated at 620, the wavelet transform component applies wavelet transform techniques to pixel blocks within a strip to decompose the pixel data into N (e.g., 7) frequency bands. These frequency bands are then organized into blocks (e.g., CTUs), which can be provided to a block-based encoder for encoding / compression. As an example, a frame can be divided into 128×128 blocks, and a two-level wavelet decomposition can be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing the seven frequency bands, for example... Figure 3C As shown.

[0082] In some embodiments, the video coding system may include a single wavelet transform component for processing tiles. In some embodiments, the video coding system may include a separate wavelet transform component for processing each tile. In some embodiments, the video coding system may include multiple wavelet transform components, each processing multiple (e.g., two) tiles.

[0083] exist Figure 6 In the method, an exemplary video coding system includes two encoders configured to encode frequency data blocks from different stripes in a stripe generated in parallel at element 620. For example, a first encoder may be configured to encode blocks from stripes 0 and 1, and a second encoder may be configured to encode blocks from stripes 2 and 3. As indicated at 630A, the first encoder applies encoding techniques to the frequency bands in blocks of tiles 0 and 1, thereby multiplexing the processing of blocks from different frequency bands between the two different tiles. As indicated at 630B, the second encoder applies encoding techniques to the frequency bands in blocks of tiles 2 and 3, thereby multiplexing the processing of blocks from different frequency bands between the two different tiles.

[0084] The encoder may be, for example, a High-Efficiency Video Coding (HEVC) encoder. However, in some implementations, other coding techniques may be used. As indicated at element 620, decomposing pixel data into frequency bands allows the encoder to buffer and process the frequency bands as separate streams at elements 630A and 630B. Processing frequency bands as separate streams allows the encoder to multiplex the processing of independent streams. In block-based coding methods such as HEVC coding, blocks (called coding tree units (CTUs)) are processed in a pipeline at multiple stages; two or more blocks may be at different stages of the pipeline at a given clock cycle, and the block moves through the pipeline with that clock cycle. The processing of a given block may depend on one or more previously processed adjacent blocks, such as one or more blocks in the row above the given block and / or blocks to the left of the given block. By multiplexing the processing of the streams, the encoder spaces the processing of blocks in the given stream, thereby providing additional clock cycles to process the adjacent blocks that the given block depends on. For example, when a given block reaches a stage that depends on a previously processed adjacent block, a block to the left of the given block may be several stages ahead of the given block in the pipeline. This allows the encoder to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for the processing of adjacent blocks in the pipeline to complete before processing a given block at a stage that depends on adjacent blocks.

[0085] As indicated at 640, the wireless interface packets the compressed data generated by the encoder at element 530 and transmits the packets to the device via a wireless connection. As indicated by the arrow returning from 650 to 600, this method continues as long as the user is using the VR / MR system.

[0086] As indicated at element 620, decomposing pixel data into frequency bands allows for frequency band prioritization by the encoder at elements 630A and 630B, and by the wireless interface at element 640. Generally, in image and video transmission, lower frequencies are more important than higher frequencies. Higher frequencies typically correspond to details in the image and are therefore considered lower priority. Higher frequency bands contain a smaller percentage of the energy in the image. Most of the energy is contained in lower frequency bands. Therefore, decomposing pixel data into frequency bands provides a prioritization of the data stream, which the encoder and wireless interface can utilize when encoding and transmitting the data stream. For example, in some implementations, different compression techniques can be used on different frequency bands, with more aggressive compression applied to lower priority bands and more conservative compression applied to higher priority bands. As another example, frequency band prioritization can help provide a degree of degradation for VR / MR systems. The performance of the wireless connection can be monitored, and feedback from the device can be taken into account to track the performance of the entire system. If the system falls behind for some reason, such as if the wireless connection is degraded and the bandwidth capacity of the wireless connection drops below a threshold, the encoder and wireless interface may prioritize the encoding and transmission of one or more lower frequency bands in the lower frequency bands, and may reduce or decrease the encoding and / or transmission of one or more frequency levels in the frequency levels that have been assigned a lower priority level (e.g., one or more higher frequency bands in the higher frequency bands).

[0087] Although an embodiment in which each strip is divided into four blocks and two encoders operate on the corresponding blocks from each strip is described, in some embodiments the strip may be divided into more blocks (e.g., six or eight blocks), and in some embodiments more encoders (e.g., three or four encoders) may be used.

[0088] Rate control

[0089] Encoders can employ rate control methods or algorithms to dynamically adjust encoder parameters to achieve a target bitrate. Rate control methods estimate the video bitrate based on the available bandwidth of the wireless connection. The estimated bitrate is used to adjust encoder parameters to ensure successful transmission of the encoded bitstream while optimizing the use of available bandwidth. Typically, rate control methods for video streaming systems operate on the time constant of one or two frames to estimate the video bitrate and adjust encoder parameters accordingly.

[0090] The goal of a VR / MR system implementing the video encoding system described herein can be to provide very low latency (e.g., 4-5 milliseconds) between the time virtual content is rendered at the base station and the time it is displayed on the device, while minimizing errors (e.g., dropped packets) and maximizing bandwidth utilization over the wireless connection. This low latency objective presents a challenge for rate control in the encoder when encoding data to be transmitted to the device wirelessly.

[0091] In some embodiments, the video coding system includes: a wavelet transform unit that performs a two-level wavelet decomposition, which decomposes pixel blocks (e.g., 64x64 or 128x128 blocks) from a strip of frames into multiple (e.g., seven) distinct frequency bands; and a block-based encoder, such as HEVC encoding the frequency band. In some embodiments, to perform rate control under the stringent latency requirements of VR / MR systems, the rate control unit may adjust encoder parameters (e.g., quantization parameters of the frequency band) for each strip, rather than working on the time constant of one or two frames to adjust the encoder parameters. In some embodiments, the frequency band of the strip can be buffered in local memory after the wavelet transform, allowing complexity statistics on that frequency band to be computed before encoding, for example, by a complexity analysis unit of the video coding system. In some embodiments, the complexity statistics may include, but are not limited to, one or more of the pixel gradient and size variance of the CTU block. This complexity statistics can then be used by the rate control unit to determine the quantization parameters of the frequency band for modulating the rate in the encoder for the current strip. Calculating the complexity statistics of the buffered bands allows the video coding system to estimate the complexity of a strip before encoding it.

[0092] In some implementations, the rate control unit may examine statistics from one or more previously processed frames to determine the actual number of bits used to encode the co-sited block or stripe of that frame. This information may then be combined with complexity statistics of the current stripe to determine the quantization parameters for the current stripe.

[0093] In some implementations, the quantization parameters of the frequency bands can be jointly computed to optimize the overall display quality of the frame after decoder reconstruction and wavelet synthesis are performed on-device. A method for rate allocation between frequency bands is to find operating points in each band that result in equal slopes on the operating rate distortion curve. In some implementations, a combination of statistics from the entropy coding stage of the encoder pipeline can be used to estimate the operating rate distortion curve, where the rate of the coefficients can be very close to the quantization parameter step size.

[0094] In some implementations, a region of interest (ROI) may exist within an image (e.g., a concave region) that should have a higher resolution than surrounding regions (e.g., peripheral regions) in the same image. In some implementations, complexity statistics from previous frames, along with concave regions from previous frames, may be stored and used to calculate quantization parameters for bands in the current frame's stripes. For example, for each block to be encoded, the concave region of a previous frame can be used to modulate the complexity statistics calculated for the previous frame. The resulting complexity statistics can then be used, in addition to those calculated for the current frame's stripes, to determine the quantization parameters.

[0095] In some implementations, the rate control unit can adjust encoder parameters (e.g., quantization parameters for bands or priorities) at the block level rather than the strip level. In some implementations, complexity statistics for each block of the strip can be calculated, and the quantization parameters for that block can be determined.

[0096] Figure 7 It is based on at least some of the embodiments shown, such as Figure 1 or Figure 2 The diagram shows a block diagram of a video coding system, in which the complexity statistics of the frequency band are calculated before encoding and used to generate quantization parameters for encoding.

[0097] In some implementations, the rendering strip of a frame, including pixel data, is first processed by a pre-filter 1004 component (e.g., an N-channel filter bank). In some implementations, gaze tracking information obtained from the device can be used to identify the direction the user is currently looking. The region of the frame corresponding to the fovea (referred to as the foveal region) can be identified at least in part based on the determined gaze direction. Information can be reduced by pre-filtering peripheral regions (i.e., the portion of the frame outside the foveal region) based on knowledge of the human visual system, for example, by filtering high-frequency information and / or increasing color compression.

[0098] The output of pre-filter 1004 is processed by wavelet transform 1006. In some embodiments, wavelet transform 1006 performs a two-level wavelet decomposition that decomposes the pixel blocks (e.g., 64x64 or 128x128 blocks) of a strip from a frame into multiple (e.g., seven) distinct frequency bands. The frequency bands of the strip can be buffered after wavelet transform 1005, allowing complexity statistics on the frequency bands of the strip to be computed by complexity analysis 1050 of the video coding system before encoding. In some embodiments, the complexity statistics may include, but are not limited to, one or more of pixel gradients and size variances of CTU blocks. Complexity analysis 1050 provides the complexity statistics of the frequency bands in the strip to rate control 1018 of encoder 1008. Rate control 1018 then computes quantization parameters for each frequency band in the strip, at least in part, based on the complexity statistics.

[0099] In some implementations, the quantization parameters of the frequency bands can be jointly calculated by the rate control 1018 components to allocate rates between frequency bands after decoder reconstruction and wavelet synthesis are performed on the device, while simultaneously optimizing the overall display quality of the frame. The method for rate allocation between frequency bands is to find the operating points in each frequency band that result in equal slopes on the operating rate distortion curve. In some implementations, a combination of statistics from the entropy coding stage of the encoder pipeline can be used to estimate the operating rate distortion curve, where the rate of the coefficients can be very close to the quantization parameter step size 6.

[0100] In some implementations, the rate control unit may examine statistics from one or more previously processed frames to determine the actual number of bits used to encode the co-sited block or stripe of that frame. This information may then be combined with complexity statistics of the current stripe to determine the quantization parameters for the current stripe.

[0101] In some implementations, a region of interest (ROI) may exist within an image (e.g., a concave region) that should have a higher resolution than surrounding regions (e.g., peripheral regions) in the same image. In some implementations, complexity statistics from previous frames, along with concave regions from previous frames, may be stored by the rate control 1018 component and used to calculate quantization parameters for bands in the current frame's stripes. For example, for each block to be encoded, the concave regions of previous frames can be used to modulate complexity statistics calculated for the previous frames. The resulting complexity statistics can then be used, in addition to those calculated for the current frame's stripes, to determine the quantization parameters.

[0102] The rate control unit 1018 provides quantization parameters of the frequency bands in the strip to the block processing pipeline 1020, which uses these quantization parameters to encode the frequency bands of the current strip. The encoded frequency bands are then provided to the wireless interface 1010, which groups and transmits the encoded frequency bands to the device via the wireless connection 1080.

[0103] The goal of this video encoding system is to optimize the quality of the video stream to the device while maximizing bandwidth utilization and minimizing latency on the wireless connection 1080 to the device. In some implementations, to help achieve this goal, the wireless interface 1010 can provide current wireless connection information as feedback to the rate control 1018 component of the encoder 1008. The rate control component 1018 can use this information to determine the quantization parameters of the frequency bands. In some implementations, the wireless interface 1010 can track the instantaneous total bit rate of the wireless connection 1080 and can also track the packet drop rate on the connection 1080. In some implementations, the wireless interface 1010 feeds the current bit rate of the wireless connection 1080 back to the rate control 1018 component, which can use this information to allocate available bit rates across different frequency bands. For example, if the bit rate of the wireless connection 1080 is currently low, the rate control 1018 component can adjust the quantization parameters of one or more frequency bands in the higher frequency bands, such that fewer bits are allocated to those bands. In some implementations, the wireless interface 1010 will also be used to feed the current packet drop rate of the wireless connection 1080 back to the rate control 1018 component; the rate control 1018 component can use the current packet drop rate and the current bit rate to balance the adjustment of the quantization parameters of the frequency band to reduce or minimize the packet drop rate, while maximizing the bandwidth utilization at the current bit rate in order to optimize the quality of the video stream.

[0104] In some implementations, the rate control 1018 component can adjust the quantization parameters of the frequency band at the block level rather than at the strip level.

[0105] Figure 8 It is based on at least some implementation schemes for such Figure 7The flowchart illustrates a method for operating a video encoding system. As indicated at 1100, a pre-filtering unit determines the fovea region of a frame and pre-filters the pixel data in the strip. In some embodiments, the rendering strip of a frame, including pixel data, is first processed by the pre-filtering unit (e.g., an N-channel filter bank). In some embodiments, gaze tracking information obtained from the device can be used to identify the direction the user is currently looking. The region of the frame corresponding to the fovea (referred to as the fovea region) can be identified at least in part based on the determined gaze direction. Information can be reduced by pre-filtering peripheral regions (i.e., the portion of the frame outside the fovea region) based on knowledge of the human visual system, for example, by filtering high-frequency information and / or increasing color compression.

[0106] As indicated at 1110, the wavelet transform unit decomposes the pre-filtered pixel data into N (e.g., 7) frequency bands. In some embodiments, the wavelet transform 1006 unit performs a two-level wavelet decomposition, which decomposes the pixel blocks (e.g., 64x64 or 128x128 blocks) from the strip of the frame into multiple (e.g., seven) distinct frequency bands. The frequency bands of the strip can be buffered after the wavelet transform, so that the complexity statistics on the frequency band of the strip can be calculated by the complexity analysis unit of the video coding system before encoding.

[0107] As indicated at 1120, the complexity analysis unit calculates the complexity statistics for that frequency band. In some embodiments, the complexity statistics may include, but are not limited to, one or more of the pixel gradient and size variance of the CTU block. The complexity analysis unit provides the complexity statistics of the frequency band in the strip to the rate control unit of the encoder.

[0108] As indicated at 1130, the encoder rate control unit determines the quantization parameters for each frequency band in the strip based at least in part on this complexity statistic. In some embodiments, the quantization parameters of the frequency bands can be jointly calculated by the rate control 1018 unit to allocate rates between frequency bands after decoder reconstruction and wavelet synthesis are performed on the device, while simultaneously optimizing the overall display quality of the frame. The method for rate allocation between frequency bands is to find the operating points in each frequency band that result in equal slopes on the operating rate distortion curve. In some embodiments, a combination of statistics from the entropy coding stage of the encoder pipeline can be used to estimate the operating rate distortion curve, where it is known that the rate of the coefficients can very closely approximate the quantization parameter step size 6.

[0109] In some implementations, the rate control unit may examine statistics from one or more previously processed frames to determine the actual number of bits used to encode the co-sited block or stripe of that frame. This information may then be combined with complexity statistics of the current stripe to determine the quantization parameters for the current stripe.

[0110] In some implementations, a region of interest (ROI) may exist within an image (e.g., a concave region) that should have a higher resolution than surrounding regions (e.g., peripheral regions) in the same image. In some implementations, complexity statistics from previous frames, along with concave regions from previous frames, may be stored by the rate control 1018 component and used to calculate quantization parameters for bands in the current frame's stripes. For example, for each block to be encoded, the concave regions of previous frames can be used to modulate complexity statistics calculated for the previous frames. The resulting complexity statistics can then be used, in addition to those calculated for the current frame's stripes, to determine the quantization parameters.

[0111] As indicated at 1140, the encoder (e.g., an HEVC encoder) uses the quantization parameters to apply encoding techniques to the frequency band to compress the data. The rate control unit provides the quantization parameters of the frequency bands in the strip to the encoder's block processing pipeline, which uses these quantization parameters to encode the frequency bands of the current strip. The encoded frequency bands are then provided to the base station's radio interface.

[0112] As indicated at 1150, the wireless interface packets compressed data and transmits the compressed data to the device via a wireless connection. In some embodiments, the wireless interface may provide current wireless connection information as feedback to the rate control unit of the encoder. The rate control unit may use this information to determine the quantization parameters of the frequency band. In some embodiments, the wireless interface may track the instantaneous total bit rate of the wireless connection and may also track the packet drop rate on the connection. In some embodiments, the rate control unit may use the current bit rate of the wireless connection to allocate available bit rate across different frequency bands. In some embodiments, the rate control unit may use the current packet drop rate and the current bit rate to balance adjustments to the quantization parameters of the frequency band to reduce or minimize the packet drop rate in order to optimize the quality of the video stream at the current bit rate.

[0113] As indicated by the arrow returning from element 1160 to element 1100, the method can continue as long as there is data to be transmitted to the device.

[0114] Exemplary VR / MR System

[0115] Figure 9An exemplary VR / MR system 2000, which implements a video encoding system according to at least some embodiments, is illustrated. The VR / MR system 2000 may include at least one device 2150 (e.g., a laptop or laptop computer, tablet or tablet device, smartphone, handheld computing device, or HMD, such as a headset, helmet, goggles, or glasses that can be worn by a user) and a computing device 2100 (referred to herein as a base station). The base station 2100 renders VR or MR frames including virtual content, encodes the frames, and transmits the encoded frames to the device 2150 via a wireless connection 2180 for decoding and display by the device 2150.

[0116] Base station 2100 and device 2150 may each include wireless communication technologies that allow base station 2100 and device 2150 to communicate and exchange data via wireless connection 2180. In some embodiments, wireless connection 2180 may be implemented based on proprietary wireless communication technologies that provide a highly directional wireless link between device 2150 and base station 2100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0117] In some implementations, device 2150 may include sensors that collect information about the user's environment (e.g., video, depth information, lighting information, etc.) and / or information about the user (e.g., the user's facial expressions, eye movements, gaze direction, gestures, etc.). Device 2150 may transmit at least some of the information collected by the sensors to base station 2100 via wireless connection 2180. Base station 2100 may render frames, including virtual content, for display by device 2150 based at least in part on the various information obtained from the sensors, encode the frames, and transmit the encoded frames to device 2150 via wireless connection 2180 for decoding and display to the user. To encode and transmit the frames, base station 2100 may implement, as follows: Figures 1 to 8 The video encoding system shown.

[0118] Figure 10 This illustrates some implementation schemes, such as Figure 9The diagram illustrates the functional components and processing in an exemplary VR / MR system. Device 2150 may be, but is not limited to, a laptop or laptop computer, tablet or tablet device, smartphone, handheld computing device, or HMD, such as a headset, helmet, goggles, or glasses that can be worn by a user. Device 2150 may include a display 2156 component or subsystem capable of implementing any of various types of virtual or augmented reality display technologies. For example, HMD device 2150 may be a near-eye system that displays left and right images (viewed by the subject) on a screen in front of the user's eyes, such as DLP (Digital Light Processing), LCD (Liquid Crystal Display), and LCoS (Liquid Crystal on Silicon) technology VR systems. As another example, HMD device 2150 may be a direct retinal projector system that scans the left and right images pixel-by-pixel to the subject's eyes. To scan the images, the left and right projectors generate beams of light that are directed to left and right reflective components (e.g., ellipsoidal mirrors) located in front of the user's eyes; the reflective components reflect the beams of light to the user's eyes. To create a three-dimensional (3D) effect, virtual content at different depths or distances in a 3D virtual view is shifted left or right in two images as a function of distance triangulation, with closer objects shifting more than farther objects.

[0119] Device 2150 may also include controller 2154 configured to implement device-side functions of the VR / MR system 2000 as described herein. In some embodiments, device 2150 may also include memory 2170 configured to store software (code 2172) of the device components of the VR / MR system 2000 executable by controller 2154, and data 2174 that can be used by the software when executed on controller 2154. In various embodiments, controller 2154 may be a single-processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number). Controller 2154 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. For example, in various embodiments, controller 2154 may include a general-purpose processor or embedded processor implementing any of a variety of instruction set architectures (ISAs) (such as x86, PowerPC, SPARC, RISC, or MIPS ISA, or any other suitable ISA). In a multiprocessor system, each processor may collectively implement the same ISA, but this is not required. Controller 2154 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, unordered, ordered, speculative, non-speculative, etc., or combinations thereof. Controller 2154 may include circuitry implementing microcode technology. Controller 2154 may include one or more processing cores, each configured to execute instructions. Controller 2154 may include one or more levels of caches, which may be of any size and configuration (set-associative, direct mapping, etc.). In some embodiments, controller 2154 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry. Typically, the GPU may be configured to render objects to be displayed into a frame buffer (e.g., a frame buffer containing pixel data for the entire frame). The GPU may include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of some graphics operations. In some embodiments, controller 2154 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder (codec), etc. In some embodiments, controller 2154 may include at least one system-on-a-chip (SOC).

[0120] Memory 2170 may include any type of memory, such as Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Dual Data Rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, or low-power versions of SDRAM, such as LPDDR2), RAMBUSDRAM (RDRAM), Static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled onto a circuit board to form a memory module, such as a Single In-line Memory Module (SIMM), a Dual In-line Memory Module (DIMM), etc. Alternatively, the device may be mounted with the integrated circuits implementing the system in a chip stack configuration, a package stack configuration, or a multi-chip module configuration.

[0121] In some embodiments, sensor 2160 may include, but is not limited to, one or more gaze-tracking sensors (e.g., an IR camera with an IR illumination source) that can be used to track the position and movement of a user's eyes. In some embodiments, two gaze-tracking sensors may be present, each tracking a corresponding eye. In some embodiments, information collected by the gaze-tracking sensors may be used by base station 2100 to adjust the rendering of images and / or to adjust the projection of images through the projection system of device 2150 based on the direction and angle of the user's eye view. For example, in some embodiments, the content of the image in the area surrounding the user's currently viewed position may be rendered with more detail and may be at a higher resolution than the content in areas not viewed by the user, allowing available image data processing time to be spent on the content viewed by the foveal region of the eye rather than on the content viewed by the peripheral region of the eye. Similarly, the content of the image in the area not viewed by the user may be compressed more than the content in the area surrounding the point currently viewed by the user. In some embodiments, two gaze-tracking sensors may be present on the inner surface of device 2150, at positions that allow the sensors to have a view of the corresponding eye in the user's eye. However, in various embodiments, more or fewer gaze tracking sensors may be used, and the gaze tracking sensors may be positioned in other locations. In an exemplary non-limiting embodiment, each gaze tracking sensor may include an IR light source and an IR camera, such as a 400×400 pixel camera with a frame rate of 120 FPS or greater, an HFOV of 70 degrees, and a working distance of 10 mm to 80 mm.

[0122] In some embodiments, device 2150 may include at least one inertial measurement unit (IMU) 2162 configured to detect the position, orientation and / or motion of device 2150 and provide the detected position, orientation and / or motion data to controller 2154 of device 2150 and / or to base station 2100.

[0123] Device 2150 may also include a wireless interface 2152 configured to communicate with an external base station 2100 via a wireless connection 2180 to transmit sensor input to the base station 2100 and receive compressed rendering frames, stripes, or tiles from the base station 2100. In some embodiments, wireless interface 2152 may implement proprietary wireless communication technology that provides a highly directional wireless link between device 2150 and base station 2100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0124] Base station 2100 may be an external device (e.g., a computing system, game console, etc.) communicatively coupled to device 2150 via wireless interface 2180. Base station 2100 may include one or more processors of various types (e.g., SOC, CPU, ISP, GPU, codec, and / or other components) for rendering, filtering, encoding, and transmitting video and / or images. Base station 2100 may render frames including virtual content (each frame including a left image and a right image) based at least in part on various inputs obtained from sensor 2160 via wireless connection 2180, filter and compress the rendered frames (or stripes of frames) using a video encoding system as described herein, and transmit the compressed frames or stripes to device 2150 for display.

[0125] Base station 2100 may be or may include any type of computing system or computing device, such as a desktop computer, laptop or notebook computer, tablet or tablet device, smartphone, handheld computing device, game controller, gaming system, etc. Base station 2100 may include controller 2110, which includes one or more processors implementing base-side functions of VR / MR system 2000 (including video encoding systems as described herein). Base station 2100 may also include memory 2120 configured to store software (code 2122) of the base station components of VR / MR system 2000 executable by base station controller 2110, and data 2124 that can be used by the software when executed on controller 2110.

[0126] In various embodiments, the base station controller 2110 may be a single-processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number). The controller 2110 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture and may be configured to execute instructions defined in that instruction set architecture. For example, in various embodiments, the controller 2110 may include a general-purpose processor or embedded processor implementing any of a variety of instruction set architectures (ISAs) (such as x86, PowerPC, SPARC, RISC, or MIPS ISA, or any other suitable ISA). In a multiprocessor system, each processor may collectively implement the same ISA, but this is not required. The controller 2110 may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out-of-order, ordered, speculative, non-speculative, etc., or combinations thereof. The controller 2110 may include circuitry implementing microcode technology. The controller 2110 may include one or more processing cores, each configured to execute instructions. Controller 2110 may include one or more levels of cache, which may be of any size and configuration (set-associative, direct mapping, etc.). In some embodiments, controller 2110 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuitry. Typically, the GPU may be configured to render objects to be displayed into a frame buffer (e.g., a frame buffer that includes pixel data for the entire frame). The GPU may include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of some graphics operations. In some embodiments, controller 2110 may include one or more other components as described herein for processing, rendering, filtering, and encoding video and / or images, such as one or more of various types of integrated circuits (ICs), image signal processors (ISPs), encoders / decoders (codecs), etc. In some embodiments, controller 2110 may include at least one system-on-a-chip (SOC).

[0127] The base station memory 2120 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), dual data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, or low-power versions of SDRAM, such as LPDDR2), RAMBUSDRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled onto a circuit board to form a memory module, such as a single in-line memory module (SIMM), a dual in-line memory module (DIMM), etc. Alternatively, the devices may be mounted with the integrated circuits implementing the system in a chip stack configuration, a package stack configuration, or a multi-chip module configuration.

[0128] Base station 2100 may also include one or more wireless technology interfaces 2130 configured to communicate with device 2150 via wireless connection 2180 to receive sensor input from device 2150 and transmit compressed frames, stripes, or tiles from base station 2100 to device 2150. In some embodiments, wireless technology interface 2130 may implement proprietary wireless communication technology that provides a highly directional wireless link between device 2150 and base station 2100. In some embodiments, the directionality and bandwidth of the wireless communication technology can support multiple devices 2150 communicating simultaneously with base station 2100, thereby enabling multiple users to use system 2000 simultaneously in a cooperative positioning environment. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0129] In some implementations, base station 2100 may be configured to render frames and transmit the frames to device 2150 at least in part based on sensor 2160 input received from device 2150 to provide a 3D virtual view to a user. In some implementations, the virtual view may include rendering the user's environment, including rendering real-world objects in the user's environment based on video captured by one or more scene cameras (e.g., RGB (visible light) cameras) that capture high-quality, high-resolution video of the user's environment in real time for display. In some implementations, the virtual view may also include virtual content (e.g., virtual objects, virtual labels for real-world objects, user avatars, etc.) rendered by base station 2100 and composited with a projected 3D view of the user's real-world environment.

[0130] Although not in Figure 9 and Figure 10As shown, however, in some embodiments, the VR / MR system 2000 may include one or more other components. For example, the system may include a cursor control device (e.g., a mouse) for moving a virtual cursor within a 3D virtual view to interact with virtual content. Although Figure 9 and Figure 10 A single device 2150 is shown, but in some implementations, the VR / MR system 2000 may support multiple devices 2150 communicating with the base station 2100 simultaneously, thereby enabling multiple users to use the system simultaneously in a cooperative positioning environment.

[0131] In various implementations, the methods described herein can be implemented in software, hardware, or a combination thereof. Furthermore, the order of the blocks of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. Various modifications and changes will be apparent to those skilled in the art who benefit from this disclosure. The various implementations described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions, and improvements are possible. Thus, multiple examples may be provided for a component described herein as a single example. The boundaries between various components, operations, and data repositories are somewhat arbitrary, and specific operations are shown in the context of a particular exemplary configuration. Other assignments of functionality are contemplated, which may fall within the scope of the appended claims. Finally, the structures and functions of discrete components presented in exemplary configurations can be implemented as combined structures or components. These and other variations, modifications, additions, and improvements may fall within the scope of the implementations as defined in the following claims.

Claims

1. A video encoding system, the video encoding system comprising: Wavelet transform component, the wavelet transform component including one or more processors, the one or more processors being configured to: Wavelet transform is applied to pixel blocks from the current video frame to decompose the pixel blocks into multiple frequency bands; and The output includes frequency blocks for frequency data of the multiple frequency bands; Complexity analysis component, the complexity analysis component including one or more processors, the one or more processors being configured to: The frequency data in the frequency block is analyzed to calculate complexity statistics across the multiple frequency bands of the current video frame; as well as Output the complexity statistics for the multiple frequency bands of the current video frame; A block-based encoder, wherein the block-based encoder is configured to: Obtain the central concave region from the previous video frame; Determine the number of bits that have been used to encode the co-address block from the previous video frame; The quantization parameters of each of the multiple frequency bands of the current video frame are determined at least in part based on the complexity statistics of the multiple frequency bands for the current video frame, other complexity statistics from the previous video frame, the central concave region of the previous video frame, and the number of bits determined to have been used to encode the co-address block from the previous video frame. as well as The coding technique is applied to the frequency block of the current video frame, wherein the coding technique uses determined quantization parameters of the plurality of frequency bands of the current video frame to compress the frequency data in the frequency block; as well as A wireless interface configured to transmit compressed frequency data to a receiving device via a wireless connection.

2. The video coding system of claim 1, wherein, To determine the quantization parameters for each of the plurality of frequency bands in the current video frame, the block-based encoder is further configured to: The central concave region of the previous video frame is determined based on the user's gaze direction, wherein the central concave region within the image has a higher resolution than the surrounding region.

3. The video encoding system according to claim 1, wherein, To determine the quantization parameters for each of the plurality of frequency bands in the current video frame, the block-based encoder is further configured to: The other complexity statistics from the previous video frame are modulated based on the concave region of the previous video frame.

4. The video encoding system according to claim 1, wherein, To determine the quantization parameters for each of the plurality of frequency bands in the current video frame, the block-based encoder is further configured to: Examine the other complexity statistics of the previous video frame to determine the number of bits that have been used to encode the co-located frequency block of the previous video frame.

5. The video coding system of claim 1, wherein the complexity statistics include one or more of pixel gradient information and size variance of at least one frequency block in the frequency block relative to another frequency block in the frequency block.

6. The video encoding system according to claim 1, The wireless interface is also configured to provide the currently available bit rate of the wireless connection as feedback to the block-based encoder; and in, When determining the quantization parameters for each of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to allocate the currently available bit rate within the corresponding frequency band of the plurality of frequency bands of the current video frame.

7. The video encoding system according to claim 1, The wireless interface is also configured to provide the current available bit rate of the wireless connection and the current packet drop rate on the wireless connection as feedback to the block-based encoder; in, When determining the quantization parameters for each of the plurality of frequency bands of the current video frame, the block-based encoder is further configured to adjust the quantization parameters of the corresponding frequency bands among the plurality of frequency bands to minimize the packet drop rate while maximizing the available bandwidth utilization at the current available bit rate.

8. The video coding system of claim 1, wherein the wavelet transform is a two-level wavelet decomposition applied to each pixel block from the current video frame.

9. The video encoding system of claim 1, wherein the block-based encoder comprises a block processing pipeline and a rate control unit, wherein: The rate control unit is configured to perform the determination of the quantization parameters for each frequency band in the frequency band and is further configured to provide the quantization parameters to the block processing pipeline; and The block processing pipeline is configured to apply the encoding technique to the frequency block of the current video frame.

10. A method for video encoding, the method comprising: One or more processors, wherein the one or more processors are configured to implement: Wavelet transform is applied to pixel blocks from the current video frame to decompose the pixel blocks into multiple frequency bands; The frequency data of the multiple frequency bands are stored in frequency blocks; The frequency data in the frequency block is analyzed to calculate complexity statistics across the multiple frequency bands of the current video frame; Obtain the central concave region from the previous video frame; Determine the number of bits that have been used to encode the co-address block from the previous video frame; The quantization parameters of each of the multiple frequency bands of the current video frame are determined at least in part based on the complexity statistics of the multiple frequency bands for the current video frame, other complexity statistics from the previous video frame, the central concave region of the previous video frame, and the number of bits determined to have been used to encode the co-address block from the previous video frame. The coding technique is applied to the frequency block, wherein the coding technique uses determined quantization parameters of the plurality of frequency bands of the current video frame to compress the frequency data in the frequency block; as well as The compressed frequency data is transmitted to the receiving device via a wireless connection.

11. The method of claim 10, wherein determining the quantization parameters of each of the plurality of frequency bands of the current video frame comprises: The central concave region of the previous video frame is determined based on the user's gaze direction, wherein the central concave region within the image has a higher resolution than the surrounding region.

12. The method of claim 10, wherein determining the quantization parameters of each of the plurality of frequency bands of the current video frame comprises: The other complexity statistics from the previous video frame are modulated based on the concave region of the previous video frame.

13. The method of claim 10, wherein determining the quantization parameters of each of the plurality of frequency bands of the current video frame comprises: Examine the other complexity statistics of the previous video frame to determine the number of bits that have been used to encode the co-address block of the previous video frame.

14. The method of claim 10, wherein the complexity statistics include one or more of pixel gradient information and size variance of at least one frequency block in the frequency blocks relative to another frequency block in the frequency blocks.

15. The method of claim 10, further comprising tracking the current available bit rate of the wireless connection, wherein the quantization parameter for determining the frequency band includes allocating the current available bit rate among corresponding frequency bands of the plurality of frequency bands.

16. The method of claim 10, further comprising tracking the current available bit rate of the wireless connection and the current packet drop rate on the wireless connection, wherein determining the quantization parameters of the plurality of frequency bands further comprises adjusting the quantization parameters of the respective frequency bands of the plurality of frequency bands to minimize the packet drop rate while maximizing the available bandwidth utilization at the current available bit rate.

17. A video encoding system, the video encoding system comprising: The device includes one or more processors and a display subsystem; The base station includes one or more processors, which are configured to: Wavelet transform is applied to pixel blocks from the current video frame to decompose the pixel blocks into multiple frequency bands; The frequency data of the multiple frequency bands are stored in frequency blocks; The frequency data in the frequency block is analyzed to calculate complexity statistics across the multiple frequency bands of the current video frame; Obtain the central concave region from the previous video frame; Determine the number of bits that have been used to encode the co-address block from the previous video frame; The quantization parameters of each of the multiple frequency bands of the current video frame are determined at least in part based on the complexity statistics of the multiple frequency bands for the current video frame, other complexity statistics from the previous video frame, the central concave region of the previous video frame, and the number of bits determined to have been used to encode the co-address block from the previous video frame. An encoding technique is applied to the frequency block, wherein the encoding technique uses determined quantization parameters of the plurality of frequency bands of the current video frame to compress the frequency data in the frequency block; and The compressed frequency data is transmitted to the device via a wireless connection; The one or more processors of the device are configured to: Decompress the compressed frequency data received from the base station; Wavelet synthesis is performed on the decompressed frequency data to reconstruct the pixel data of the video frame; as well as The reconstructed pixel data is provided to the display subsystem for display.

18. The system according to claim 17, wherein, In order to determine the quantization parameters of each of the plurality of frequency bands of the current video frame, the one or more processors of the base station are further configured to: The other complexity statistics from the previous video frame are modulated based on the central concave region of the previous video frame; as well as The quantization parameters of each of the plurality of frequency bands of the current video frame are determined at least in part based on other modulated complexity statistics from the previous video frame, combined with the complexity statistics for the current video frame.

19. The system according to claim 17, wherein, In order to determine the quantization parameters of each of the plurality of frequency bands of the current video frame, the one or more processors of the base station are further configured to: Examine the other complexity statistics of the previous video frame to determine the number of bits that have been used to encode the co-address block of the previous video frame.

20. The system of claim 17, wherein the device is a head-mounted display (HMD) device.

Images