Rate control for low latency video coding and transmission

Abstract

The invention relates to rate control for low latency video coding and transmission. A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. This frequency band of a slice of a frame may be buffered such that complexity statistics on the frequency band may be calculated prior to encoding. The statistics may then be used by a rate control component to determine a quantization parameter for the frequency band for use in modulating a rate for the current slice in the encoder. Quantization parameters for the frequency band may be calculated jointly to optimize the quality of the displayed frame after decoder reconstruction and wavelet synthesis on the receiving device. Information about one or more previously processed frames may be combined with statistics of a current slice for determining the quantization parameter for the current slice.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980040176.0, application date June 27, 2019, and name “Rate Control for Low-Delay Video Coding and Transmission”. Background Art

[0002] Virtual reality (VR) allows users to experience and / or interact with an immersive artificial environment so that the user feels as if they are in the environment. For example, a virtual reality system may display a stereoscopic scene to a user to create an illusion of depth, and a computer may adjust the scene content in real time to provide the illusion that the user is moving within the scene. When a user views an image through a virtual reality system, the user may therefore feel as if they are moving within the scene from a first-person perspective. Similarly, mixed reality (MR) combines computer-generated information (referred to as virtual content) with real-world images or real-world views to enhance or add content to the user's view of the world, or alternatively combines virtual representations of real-world objects with views of a three-dimensional (3D) virtual world. Therefore, the simulated environment of virtual reality and / or the mixed environment of mixed reality can be used to provide an interactive user experience for a variety of applications. Summary of the invention

[0003] Various embodiments of a video encoding system are described that can encode a high-resolution video source with low latency for transmission to a device over a communication link (e.g., a wireless link) for decoding and display. Embodiments of the video encoding system can also provide for graceful degradation of the encoded video transmitted to the device to maintain a desired frame rate under varying conditions (such as changes in the channel capacity of the communication link). An exemplary application of the video encoding system is in a virtual or mixed reality system, where video frames containing virtual content are rendered, encoded, and transmitted by a base station to a device (e.g., a laptop or notebook computer, a tablet or tablet device, a smart phone, or a head-mounted display (HMD), such as a headset, helmet, goggles, or glasses that can be worn by a user) for decoding and display.

[0004] In some embodiments, the video coding system can perform wavelet transform on the pixel data before encoding to decompose the pixel data into frequency bands. The frequency bands are then organized into blocks, which are provided to a block-based encoder for encoding / compression. The encoded frequency data are then sent to a wireless interface, which groups the encoded frequency data and transmits the packets to a receiving device. On the receiving device, the encoded data is depacketized and passed through a block-based decoder to recover the frequency bands. Wavelet synthesis is then performed to the recovered frequency bands to reconstruct the pixel data for display.

[0005] The video encoding system may implement various methods and apparatus to maintain a target frame rate over a wireless link and minimize delays in frame rendering, transmission, and display, and to provide graceful degradation of the encoded video transmitted to the device to maintain the desired frame rate under changing conditions (such as changes in the channel capacity of the wireless connection).

[0006] A goal of a system implementing an embodiment of a video encoding system may be to provide very low latency (e.g., 4-5 milliseconds) between the time virtual content is rendered on a base station and the time the virtual content is displayed on a device, while minimizing errors (e.g., dropped packets) and maximizing bandwidth usage over a wireless connection. Embodiments of the video encoding system may implement a rate control method to help achieve the system's target low latency when encoding data to be transmitted to a device over a wireless connection.

[0007] In some embodiments, the video encoding system includes: a wavelet transform component that performs a two-layer wavelet decomposition that decomposes a pixel block from a strip of a frame into a plurality of different frequency bands; and a block-based encoder, such as HEVC that encodes the frequency band. In some embodiments, in order to perform rate control under the strict delay requirements of the VR / MR system, the rate control component may adjust the encoder parameters (e.g., the quantization parameter of the frequency band) for each strip, rather than working on a time constant of one or two frames to adjust the encoder parameters. In some embodiments, the frequency band of the strip can be buffered in a local memory after the wavelet transform, so that the complexity statistics on the frequency band can be calculated before encoding, for example, by a complexity analysis component of the video encoding system. In some embodiments, the complexity statistics may include, but are not limited to, one or more of the pixel gradient and size variance of the CTU block. The complexity statistics can then be used by the rate control component to determine the quantization parameter of the frequency band for the rate for the current strip in the modulation encoder. Calculating the complexity statistics for the buffered frequency band allows the video encoding system to estimate the complexity level in the strip before encoding the strip.

[0008] In some embodiments, the rate control component may examine the statistics of one or more previously processed frames to determine the actual number of bits used to encode the co-sited blocks or slices of the frame. This information may then be combined with the complexity statistics of the current slice to determine the quantization parameter for the current slice.

[0009] In some embodiments, the quantization parameters of the bands may be calculated jointly to optimize the overall display quality of the frame after decoder reconstruction and wavelet synthesis are performed on the device. The method for rate allocation between the bands is to find operating points in each of the bands that result in equal slopes on the operating rate distortion curve. In some embodiments, the operating rate distortion curve may be estimated using a combination of statistics from the entropy encoding stage of the encoder pipeline, where the fact is that the rate of the coefficients can be closely approximated by the quantization parameter step size 6.

[0010] In some embodiments, there may be a region of interest within an image (e.g., a foveal region) that should have a higher resolution than surrounding areas in the same image (e.g., a peripheral region). In some embodiments, the complexity statistics from a previous frame along with the foveal region from the previous frame may be stored and used to calculate quantization parameters for bands in a strip of the current frame. For example, for each block to be encoded, the foveal region of the previous frame may be used to modulate the complexity statistics calculated for the previous frame. The resulting complexity statistics may then be used in addition to the complexity statistics calculated for the strip of the current frame to determine the quantization parameter.

[0011] In some embodiments, the rate control component may adjust encoder parameters (e.g., quantization parameters for bands or priorities) at the block level rather than at the slice level. In some embodiments, complexity statistics may be calculated for each block of a slice, and the quantization parameter for the block may be determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating a video encoding system that uses a wavelet transform to decompose pixel data into frequency bands prior to encoding, according to at least some embodiments.

[0013] Figure 2 A video encoding system including multiple encoders that process tiles from a frame in parallel is shown according to at least some embodiments.

[0014] Figure 3A An exemplary frame divided into slices and tiles is shown according to at least some embodiments.

[0015] Figure 3B Example tiles divided into blocks are shown according to at least some embodiments.

[0016] Figure 3C Performing a wavelet transform on a pixel block storing pixel data to generate frequency band data prior to encoding is shown according to at least some embodiments.

[0017] Figure 4 According to at least some embodiments, Figure 1and Figure 2 A high level flow chart of a method of operation of a VR / MR system including a video encoding system is shown.

[0018] Figure 5 According to at least some embodiments, Figure 1 A flow chart of a method of operation of a video encoding system is shown.

[0019] Figure 6 According to at least some embodiments, Figure 2 A flow chart of a method of operation of a video encoding system is shown.

[0020] Figure 7 According to at least some embodiments, Figure 1 or Figure 2 Block diagram of a video coding system shown in , where complexity statistics for frequency bands are calculated and used to generate quantization parameters for encoding.

[0021] Figure 8 According to at least some embodiments, Figure 7 A flow chart of a method of operation of a video encoding system is shown.

[0022] Fig. 9 An exemplary VR / MR system is shown that may implement a video encoding system according to at least some embodiments.

[0023] Fig.10 According to at least some embodiments, Fig. 9 A block diagram of the components of a VR / MR system is shown.

[0024] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0025] The term "comprising" is open ended. As used in the claims, the term does not exclude additional structures or steps. Consider the following cited claim: "an apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, a graphics circuit, etc.).

[0026] "Configured to", various units, circuits or other components may be described or described as "configured to" perform one or more tasks. In such contexts, "configured to" is used to imply a structure (e.g., a circuit) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. In this way, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware-such as circuits, memories storing executable program instructions to implement operations, etc. The reference to a unit / circuit / component "configured to" perform one or more tasks is explicitly intended not to invoke the sixth paragraph of 35 U.S.C. §112 for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit) manipulated by software or firmware (e.g., an FPGA or a general processor executing software) to operate in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0027] "First," "second," etc. As used herein, these terms act as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation of a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0028] "Based on" or "depending on", as used herein, these terms are used to describe one or more factors that influence a determination. These terms do not exclude additional factors that may influence the determination. That is, the determination may be based solely on these factors or at least in part on these factors. Consider the phrase "A is determined based on B." In this case, B is a factor that influences the determination of A, and such phrases do not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B.

[0029] "Or", when used in the claims, the term "or" is used as an inclusive or rather than an exclusive or. For example, the phrase "at least one of x, y or z" means any one of x, y and z and any combination thereof. DETAILED DESCRIPTION

[0030] Various embodiments of video encoding systems are described. Embodiments of the video encoding systems can encode a high-resolution video source with low latency for transmission to a device over a communication link (e.g., a wireless link) for decoding and display. Embodiments of the video encoding systems can also provide graceful degradation of the encoded video transmitted to the device to maintain a desired frame rate under changing conditions (such as changes in the channel capacity of the communication link).

[0031] An exemplary application of the video coding system is in a virtual or mixed reality system, in which a video frame containing virtual content is rendered, encoded and transmitted to a device for decoding and display. An embodiment of a virtual or mixed reality system (referred to herein as a VR / MR system) is described, in which an embodiment of a video coding system can be implemented. In some embodiments, the VR / MR system may include a device (e.g., a tablet or tablet device, a smart phone, or a headset, a helmet, goggles or glasses worn by a user, referred to herein as a head-mounted display (HMD)), and a separate computing device, referred to herein as a base station. In some embodiments, the device and the base station may each include wireless communication technology that allows the device and the base station to communicate and exchange data via a wireless connection. In some embodiments, the device may include a sensor that collects information about the user's environment (e.g., video, depth information, lighting information, etc.) and information about the user (e.g., the user's expression, eye movement, gesture, etc.). The information collected by the sensor can be transmitted to the base station via a wireless connection. The base station may include software and hardware (e.g., processor (system on chip (SOC), CPU, image signal processor (ISP), graphics processing unit (GPU), encoder / decoder (codec), etc.), memory, etc.) configured to generate and render frames including virtual content based at least in part on sensor information received from the device via a wireless connection. The base station may also include an implementation of a video encoding system as described herein that may pre-filter, compress, and transmit rendered frames to the device for display via a wireless connection.

[0032] In some embodiments, the VR / MR system may implement a proprietary wireless communication technology that provides a highly directional wireless link between the device and the base station. In some embodiments, the directionality and bandwidth of the wireless communication technology may support multiple devices communicating with the base station at the same time, thereby enabling multiple users to use the system in a co-located environment at the same time. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be supported.

[0033] The main constraints to consider on a wireless link include bandwidth and latency. A goal of a VR / MR system is to provide a high-resolution, wide field of view (FOV) virtual display at a frame rate to provide a high-quality VR / MR view to the user. Another goal is to minimize the delay between the time a base station renders a frame and the time a device displays the frame. However, the channel capacity of a wireless link may vary over time, and thus a wireless link may only support a certain amount of information being transmitted at any given time.

[0034] Various methods and apparatus are described herein that can be implemented by a video encoding system to maintain a target frame rate over a wireless link and minimize delays in frame rendering, transmission, and display. In addition, the methods and apparatus can provide graceful degradation of the encoded video transmitted to the device to maintain the desired frame rate under changing conditions (such as changes in the channel capacity of the communication link).

[0035] In some embodiments, the video encoding system may perform a wavelet transform on the pixel data before encoding to decompose the pixel data into frequency bands. The frequency bands are then organized into blocks, which are provided to a block-based encoder for encoding / compression. As an example, a frame may be divided into 128×128 blocks, and a two-level wavelet decomposition may be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing seven frequency bands, which may then be sent to an encoder (e.g., a high efficiency video coding (HEVC) encoder) for encoding. The encoded frequency data is then sent to a wireless interface, which groups the encoded frequency data and transmits the grouping to a receiving device (e.g., an HMD, a tablet or tablet device, a smart phone, etc.). On the receiving device, the encoded data is depacketized and passed through a block-based decoder to recover the frequency bands. Wavelet synthesis is then performed on the recovered frequency bands to reconstruct the pixel data for display.

[0036] Although generally described are embodiments in which a wavelet transform is applied to a two-level wavelet decomposition of each pixel block from a video frame, in various embodiments, the wavelet decomposition can be any number of levels (e.g., one level, two levels, three levels, four levels, etc.) and can be adjusted to trade off the quality of the encoded image against the complexity of the block to be encoded.

[0037] In some embodiments, the video encoding system may perform strip-based rendering, encoding, and transmission. Rendering, encoding, and transmitting entire frames may have latency and memory impacts because each frame needs to be completed, stored, and then transmitted to the next stage of the VR / MR system. In strip-based rendering, instead of rendering and encoding the entire frame in the base station and transmitting the rendered frame to the device, the base station may render and encode portions of the frame (called strips) and transmit the encoded strips to the device when the encoded strips are ready. A strip can be, for example, a row of 128×128 blocks, or two or more rows of blocks. Strip-based rendering and encoding can help reduce latency and may also reduce the amount of memory required for buffering, which can reduce memory usage and power requirements on the chip or processor.

[0038] In some embodiments, the video encoding system may perform tile-based rendering, encoding, and transmission, in which each slice may be divided into multiple tiles (e.g., four tiles), and the base station may render and encode the tiles and transmit the encoded tiles to the device when they are ready.

[0039] In some embodiments, the video encoding system may perform tile-based encoding using a single encoder to process tiles from each strip. However, in some embodiments, the video encoding system may perform tile-based encoding using multiple encoders to process corresponding tiles from each strip. For example, in some embodiments, each strip may be divided into four tiles, each tile including multiple 128×128 blocks, and two encoders (e0 and e1) may operate on two tiles from each strip (e.g., e0 operates on t0 and t1; e1 operates on t2 and t3). Each encoder may multiplex the processing of blocks from different frequency bands between its two tiles to allow 16 time units between the processing of blocks from the same frequency band. By multiplexing the processing of blocks between two tiles, dependencies between blocks in the frequency bands can be properly handled. Although embodiments are described in which each slice is divided into four tiles and two encoders operate on corresponding tiles from each slice, in some embodiments, the slices may be divided into more tiles (e.g., six or eight tiles) and in some embodiments more encoders may be used (e.g., three or four encoders).

[0040] In some embodiments, the video encoding system can perform pre-filtering on the pixel data in the frame before wavelet transforming. For example, pre-filtering can reduce the resolution of the frame rendered by the base station before performing wavelet transforming, encoding, and transmitting the frame to the device over the wireless link, which can help improve compression and reduce latency and bandwidth usage on the wireless link.

[0041] In some embodiments where the device is an HMD, pre-filtering may include performing lens warping on the frame on the base station before the wavelet transform. The lens warping is performed to correct the distortion of the image introduced by the lens on the HMD (the image is viewed through the HMD), thereby improving the quality of the image. In some embodiments, the HMD may store lens warping data of the lens, for example, generated by a calibration process, and may provide the lens warping data to the base station via a wireless connection. The base station may then perform lens warping on the frame based on the lens warping data of the HMD. In conventional VR / MR systems, lens warping is performed on the HMD after decoding and before display. Performing lens warping on the base station in the pre-filter stage may reduce the resolution of the frame before performing the wavelet transform and encoding, which may help improve compression and may reduce latency and bandwidth usage on the wireless link. In addition, by performing lens warping on the base station in the pre-filter stage rather than performing lens warping on the HMD after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on the base station to reduce the resolution before encoding and then performing lens warping filtering on the HMD.

[0042] In some embodiments, pre-filtering may include filtering to reduce the resolution in the peripheral area while maintaining a higher resolution in the foveal area. In this method, the gaze tracking information obtained from the device can be used to identify the direction the user is currently looking at. The human eye can perceive a higher resolution in the foveal area than in the peripheral area. Therefore, the area of ​​the frame corresponding to the fovea (referred to as the foveal area) can be identified based at least in part on the determined gaze direction. In some embodiments, the peripheral area (i.e., the portion of the frame outside the foveal area) can be pre-filtered based on the understanding of the human visual system to reduce information, such as by filtering high-frequency information and / or increasing color compression. In some embodiments, the amount of filtering applied to the peripheral area can be increased to extend toward the periphery of the frame. Pre-filtering of the peripheral area can help provide improved frame compression.

[0043] Figure 1 1 is a block diagram illustrating a video encoding system 120 that uses a wavelet transform to decompose pixel data into frequency bands prior to encoding according to at least some embodiments. The VR / MR system 10 may include at least one device 150 (e.g., a tablet or tablet device, a smart phone, or an HMD such as a headset, helmet, goggles, or glasses that can be worn by a user) and a computing device 100 (referred to herein as a base station). The base station 100 renders a VR frame or an MR frame including virtual content, encodes the frame, and transmits the encoded frame to the device 150 via a wireless connection 180 for decoding and display by the device 150.

[0044] In some embodiments, the device 150 may include a sensor 160 that collects information about the environment of the user 190 (e.g., video, depth information, lighting information, etc.) and information about the user 190 (e.g., the user's expression, eye movement, gaze direction, gestures, etc.). The device 150 may transmit at least some of the information collected by the sensor 160 to the base station 100 via the wireless connection 180. The base station 100 may render a frame including virtual content for display by the device 150 based at least in part on the various information obtained from the sensor 160, encode the frame, and transmit the encoded frame to the device 150 via the wireless connection 180 for decoding and display to the user.

[0045] Base station 100 and device 150 may implement wireless communication technology that allows base station 100 and device 150 to communicate and exchange data via wireless connection 180. In some embodiments, wireless connection 180 may be implemented according to a proprietary wireless communication technology that provides a highly directional wireless link between device 150 and base station 100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0046] The main constraints to consider on the wireless connection 180 between the device 150 and the base station 100 in the VR / MR system 10 include bandwidth and latency. For example, in some embodiments, a goal is to provide a high-resolution, wide field of view (FOV) virtual display to the user at a frame rate that provides a high-quality VR / MR view to the user. Another goal is to minimize the delay between the time the device captures a video frame and the time the device displays a rendered VR / MR frame based on the video frame.

[0047] The base station 100 may include various hardware components for rendering, filtering, encoding, and transmitting video and / or images as described herein, such as various types of processors, integrated circuits (ICs), central processing units (CPUs), graphics processing units (GPUs), image signal processors (ISPs), encoders / decoders (codecs), etc. The base station 100 may include, but is not limited to, a GPU rendering 110 component, a wireless interface 130 component, and a video encoding system 120, which may include one or more hardware components that implement various methods that may help maintain a target frame rate over a wireless connection 180 and minimize delays in frame rendering, encoding, transmission, and display. The video encoding system 120 may include, but is not limited to, a pre-filter 122 component (e.g., an N-channel filter bank), a wavelet transform 124 component, and an encoder 126 component.

[0048] GPU rendering 110 may include one or more hardware components that may render frames for display by device 150 , including virtual content based at least in part on various information obtained from sensors 160 .

[0049] In some embodiments, the video encoding system 120 may include one or more hardware components that pre-filter 122 pixel data in a rendered frame prior to performing a wavelet transform 124. The pre-filter 122 may, for example, reduce the resolution of the frame rendered at the base station 100 prior to performing the wavelet transform 124, encoding 126, and transmission to the device 150 over the wireless connection 180, which may help improve compression and may reduce latency and bandwidth usage over the wireless connection 180.

[0050] In some embodiments, the pre-filter 122 may perform lens warping on the frame on the base station 100 before the wavelet transform 124. Lens warping is performed to correct the distortion of the image introduced by the lens on the device through which the image is viewed, thereby improving the quality of the image. In some embodiments, the device 150 may store lens warping data of the lens, such as generated by a calibration process, and the lens warping data may be provided to the base station 100 via the wireless connection 180. The pre-filter 122 component of the video encoding system 120 may then perform lens warping on the frame based on the lens warping data of the device 150. In conventional VR / MR systems, lens warping is performed on the device 150 after decoding and before display. Performing lens warping on the base station 100 in the pre-filter 122 stage may reduce the resolution of the frame before performing the wavelet transform 124 and encoding 126, which may help improve compression and may reduce latency and bandwidth usage on the wireless connection 180. Furthermore, by performing lens warping on base station 100 in the pre-filter 122 stage rather than performing lens warping on device 150 after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on base station 100 to reduce resolution prior to encoding 126 and then performing lens warping filtering on device 150.

[0051] In some embodiments, the pre-filter 122 may also apply one or more filters to reduce the resolution in the peripheral area while maintaining a higher resolution in the foveal area. In this method, the gaze tracking information obtained from the device 150 may be used to identify the direction the user is currently viewing. The human eye may perceive a higher resolution in the foveal area than in the peripheral area. Therefore, the region of the frame corresponding to the fovea (referred to as the foveal area) may be identified based at least in part on the determined gaze direction. In some embodiments, the peripheral area (i.e., the portion of the frame outside the foveal area) may be pre-filtered based on an understanding of the human visual system to reduce information, such as by filtering high-frequency information and / or increasing color compression. In some embodiments, the amount of filtering applied to the peripheral area may increase toward the periphery of the frame. Pre-filtering of the peripheral area may help provide improved frame compression.

[0052] In some embodiments, the wavelet transform 124 component of the video encoding system 120 may include one or more hardware components (e.g., an N-channel filter bank) that performs a wavelet transform on the pixel data to decompose the pixel data into frequency bands before encoding. The frequency bands are then organized into blocks, which are provided to the block-based encoder 126 for encoding / compression. As an example, FIG. 3A to FIG. 3C As shown, the frame can be divided into 128×128 blocks, and a two-level wavelet decomposition can be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing seven frequency bands, which can then be sent to a block-based encoder (e.g., a high efficiency video coding (HEVC) encoder) 126 for encoding. The encoded frequency data is then sent to a wireless interface 130 implemented by one or more hardware components, which packets the data and transmits the packets to the device 150 via a wireless connection 180.

[0053] The device 150 may include various hardware components for decoding and displaying video and / or images as described herein, such as various types of processors, integrated circuits (ICs), central processing units (CPUs), graphics processing units (GPUs), image signal processors (ISPs), encoders / decoders (codecs), etc. The device 150 may include, but is not limited to, a wireless interface 152, a decoder 154 component (e.g., a high efficiency video coding (HEVC) decoder), a wavelet synthesis 156 component, and a display 158 component. At the device 150, the wireless interface 152 receives packets transmitted by the base station 100 over the wireless connection 180. The encoded data is depacketized and passed through a block-based decoder 154 (e.g., a high efficiency video coding (HEVC) decoder) to recover the frequency bands. Wavelet synthesis 156 is then performed on the recovered frequency data to reconstruct pixel data for display 158.

[0054] In some embodiments, the video encoding system 120 may perform strip-based rendering, encoding, and transmission. Rendering, encoding, and transmitting an entire frame may have latency and memory impacts because each frame needs to be completed, stored, and then transmitted to the next stage of the VR / MR system 10. In strip-based rendering, the base station 100 may render and encode portions of a frame (called strips) and transmit the encoded strips to the device 150 when they are ready, rather than rendering and encoding an entire frame in the base station 100 and transmitting the rendered frame to the device 150. A strip may be, for example, a row of 128×128 blocks. Strip-based rendering and encoding may help reduce latency and may also reduce the amount of memory required for buffering, which reduces memory footprint and power requirements on a chip or processor.

[0055] In some embodiments, the video encoding system 120 may perform tile-based rendering, encoding, and transmission. In tile-based rendering, encoding, and transmission, each slice may be divided into multiple tiles (e.g., four tiles), and the base station 100 may render and encode the tiles and transmit the encoded tiles to the device 150 when they are ready.

[0056] In some embodiments, the video encoding system 120 may perform tile-based rendering, encoding, and transmission using a single encoder 126 to process tiles from each slice. However, in some embodiments, the video encoding system 120 may use multiple encoders 126 to perform tile-based encoding to process corresponding tiles from each slice. Figure 2 A video encoding system 220 is shown that includes multiple encoders (in this example, two encoders 226A and 226B) that process tiles from a rendered frame in parallel, according to at least some embodiments.

[0057] The GPU rendering 210 component of the base station 200 may include one or more GPUs and / or other components that render frames (or frame strips) for display. The frames may be divided into strips, such as Figure 3A As shown. Figure 3A As shown, each stripe may be divided into a plurality of tiles (four in this example), each tile including a plurality of blocks. Figure 3B An exemplary tile is shown that includes four 128x128 blocks. However, in some embodiments, blocks of other sizes (e.g., 64x64, 32x32, etc.) may be used, and a tile may include more or fewer blocks.

[0058] The prefilter 222 and wavelet transform 224 components of the video encoding system 220 can then process each tile prior to encoding 226. In some embodiments, the video encoding system 220 may include a separate prefilter 222 component and wavelet transform 224 component for processing each tile. In this example, the prefilter 222A component and wavelet transform 224A component process tile 0, the prefilter 222B component and wavelet transform 224B component process tile 1, the prefilter 222C component and wavelet transform 224C component process tile 2, and the prefilter 222D component and wavelet transform 224D component process tile 3. The prefilter 222 component performs pre-filtering of the tile as described herein, and the wavelet transform 224 component decomposes the tile into frequency bands as described herein. However, in some embodiments, the video encoding system 220 may include a single prefilter 222 component and a single wavelet transform 224 component that processes the tile. In some embodiments, the video encoding system 220 may include multiple (eg, 2) pre-filter 222 components and multiple (eg, 2) wavelet transform 224 components, each wavelet transform component processing multiple (eg, 2) tiles.

[0059] The two encoders 226A and 226B may operate on two tiles from each slice (e.g., encoder 226A operates on tile 0 and tile 1; encoder 226B operates on tile 2 and tile 3). Each encoder 226 may multiplex blocks from different bands between its two tiles (i.e., in Figure 3C ) to allow 16 time units between the processing of blocks from the same band. By multiplexing the processing of blocks between two tiles, dependencies between blocks in the same band can be properly handled.

[0060] Although embodiments are described in which each slice is divided into four tiles and two encoders operate on corresponding tiles from each slice, in some embodiments, the slices may be divided into more tiles (e.g., six or eight tiles), and in some embodiments more encoders may be used (e.g., three, four, or more encoders).

[0061] Figure 3C Performing a wavelet transform on a pixel block storing pixel data to generate frequency band data prior to encoding is shown according to at least some embodiments. In this example, a two-level wavelet decomposition is applied to a 128×128 pixel block 300 by a wavelet transform 324 component to generate sixteen 32×32 frequency data blocks 302 representing seven frequency bands. The frequency blocks 302 are then provided to an encoder 326 for encoding. For example, the frequency blocks 302 can be written to a buffer by the wavelet transform 324 component and read from the buffer by the encoder 326 component.

[0062] In the labels of the frequency blocks 302, the letter L represents a low-pass filter, and the letter H represents a high-pass filter. The blocks 302 marked with two letters represent a first-order (2D) wavelet transform or decomposition. In the blocks 302 marked with two letters (representing three of the seven frequency bands LH, HL and HH), the first letter represents the vertical filter (high or low) performed first, and the second letter represents the horizontal filter (high or low) performed second. The blocks 302 marked with four letters represent a two-level wavelet transform or decomposition. In the blocks 302 marked with four letters, the first two letters (LL) indicate that there is a vertical low-pass filter first, followed by a horizontal low-pass filter; the second two letters indicate that the resulting LL block is then four-way filtered, LL, LH, HL and HH (thereby showing four of the seven frequency bands (LLLL, LLLH, LLHL and LLHH).

[0063] Decompose the pixel data into Figure 3CThe frequency bands shown allow the encoder 326 to buffer and process the frequency bands as separate streams. Processing the frequency bands as separate streams allows the encoder 326 components to multiplex the processing of independent streams. In block-based encoding methods such as HEVC encoding, blocks (called coding tree units (CTUs)) are processed in a block processing pipeline at multiple stages; two or more blocks may be at different stages of the pipeline at a given clock cycle, and the block moves through the pipeline with the clock cycle. The processing of a given block may depend on one or more previously processed neighboring blocks, such as one or more blocks in the row above the given block and / or the block to the left of the given block. By multiplexing the processing of the frequency band data stream, the encoder 326 spaces out the processing of blocks in a given stream, thereby providing additional clock cycles to process the neighboring blocks on which the given block depends. For example, when a given block reaches a stage that depends on a previously processed neighboring block, the block to the left of the given block may be several stages ahead of the given block in the pipeline. This allows the encoder 326 to better handle dependencies on previously processed blocks and reduce or eliminate the need to wait for processing of neighboring blocks in the pipeline to complete before processing a given block at a stage that depends on the neighboring blocks.

[0064] In addition, if Figure 3C As shown, decomposing the pixel data into frequency bands allows the frequency bands to be prioritized by the encoder 326 and the wireless interface. Typically, in image and video transmission, lower frequencies are more important, while higher frequencies are less important. Higher frequencies typically correspond to details in the image and can therefore be considered a lower priority. Higher frequency bands contain a smaller percentage of energy in the image. Most of the energy is contained in lower frequency bands. Therefore, decomposing the pixel data into frequency bands provides a priority sorting for the data stream, which the encoder 326 and the wireless interface can utilize when encoding and transmitting the data stream. For example, in some embodiments, different compression techniques can be used on different frequency bands, where more aggressive compression is applied to lower priority bands and more conservative compression is applied to higher priority bands. As another example, the prioritization of frequency bands can help provide a moderate degradation of the VR / MR system. The performance of the wireless connection can be monitored, and feedback from the device can be considered to track the performance of the entire system. If the system falls behind for some reason, for example, if the wireless connection degrades and the bandwidth capacity of the wireless connection drops below a threshold, the encoder 326 and the wireless interface may prioritize encoding and transmission of one or more lower frequency bands in the lower frequency bands and may reduce or decrease encoding and / or transmission of one or more frequency levels in the frequency levels that have been assigned a lower priority level (e.g., one or more higher frequency bands in the higher frequency bands).

[0065] As mentioned above, the wavelet transform decomposes the image into frequency bands. In some embodiments, this can be used to send the same signal to displays with different resolutions. As an example, it is assumed that the signal is decomposed into seven frequency bands by applying a secondary wavelet decomposition. If four frequency bands (LLLL, LLLH, LLHL and LLHH) in the frequency band are sent, the frequency band can be reconstructed to the initial expected resolution with a smaller visual quality. Alternatively, the frequency band can also be reconstructed with 1 / 4 resolution (1 / 2 in each dimension), which is applicable to display panels with smaller display resolutions.

[0066] Figure 4 According to at least some embodiments, Figure 1 and Figure 2 A high-level flow chart of a method of operation of a VR / MR system including a video encoding system is shown. As indicated at 400, the device sends data to a base station via a wireless connection. As indicated at 410, the base station renders a frame including virtual content based at least in part on the device data. As indicated at 420, the base station compresses the rendered data and sends the compressed data to the device via a wireless connection. As indicated at 430, the device decompresses and displays the virtual content to generate a 3D virtual view for viewing by a user. As indicated by the arrow returning from 430 to 400, the method continues as long as the user is using the VR / MR system.

[0067] In some embodiments, instead of rendering and encoding the entire frame in the base station and transmitting the rendered frame to the device, the base station may render and encode portions of the frame (called slices) and transmit the encoded slices to the device when the encoded slices are ready. A slice may, for example, be a row of 128×128 blocks. In some embodiments, the video encoding system may perform tile-based rendering, encoding, and transmission. In tile-based rendering, encoding, and transmission, each slice may be divided into multiple tiles, each tile including one or more blocks (e.g., four tiles, each tile including four blocks), and the base station may render and encode the tiles and transmit the encoded tiles to the device when they are ready.

[0068] Figure 5 According to at least some embodiments, Figure 1 A flow chart of a method of operation of a video encoding system is shown. Figure 5 The method can be used, for example, in Figure 4 420 executions. Figure 5 The method assumes that slice-based encoding and transmission are being performed. However, in some embodiments, tile-based encoding and transmission may be performed.

[0069] As indicated at 510, the pre-filter component applies a lens warp and / or foveal filter to the pixel blocks in the strip of the frame. In some embodiments, pre-filtering may include performing lens warping on the frame at the base station before wavelet transforming. Lens warping is performed to correct the distortion of the image introduced by the lens on the device (through which the image is viewed), thereby improving the quality of the image. In some embodiments, the device may store lens warp data of the lens, such as generated by a calibration process, and may provide the lens warp data to the base station via a wireless connection. The base station may then perform lens warping on the frame based on the lens warp data of the device. Performing lens warping on the base station in the pre-filter stage may reduce the resolution of the frame before performing wavelet transform and encoding, which may help improve compression and may reduce latency and bandwidth usage on the wireless link. In addition, by performing lens warping on the base station in the pre-filter stage rather than performing lens warping on the device after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on the base station to reduce the resolution before encoding and then performing lens warp filtering on the device.

[0070] In some embodiments, the pre-filtering at 510 may also include filtering to reduce the resolution in the peripheral area while maintaining a higher resolution in the foveal area. In some embodiments, the gaze tracking information obtained from the device can be used to identify the direction the user is currently looking at. The area of ​​the frame corresponding to the fovea (referred to as the foveal area) can be identified based at least in part on the determined gaze direction. The peripheral area (i.e., the portion of the frame outside the foveal area) can be pre-filtered based on understanding of the human visual system to reduce information, such as by filtering high frequency information and / or increasing color compression. Pre-filtering of the peripheral area can help provide improved frame compression.

[0071] As indicated at 520, the wavelet transform component applies a wavelet transform technique to the pixel block to decompose the pixel data into N (e.g., 7) frequency bands. The frequency bands are then organized into blocks, which are provided to a block-based encoder for encoding / compression. As an example, a frame may be divided into 128×128 blocks, and a two-level wavelet decomposition may be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing seven frequency bands, such as Figure 3C shown.

[0072] As indicated at 530, the encoder applies the coding technique to the bands in the block to compress the data. The encoder may be, for example, a high-efficiency video coding (HEVC) encoder. However, in some embodiments, other coding techniques may be used. As indicated at element 520, decomposing the pixel data into bands allows the encoder to buffer and process the bands as separate streams. Processing the bands as separate streams allows the encoder components to multiplex the processing of independent streams. In block-based coding methods such as HEVC coding, blocks (called coding tree units (CTUs)) are processed in the pipeline at multiple stages; two or more blocks may be at different stages of the pipeline at a given clock cycle, and the block moves through the pipeline with the clock cycle. The processing of a given block may depend on one or more previously processed adjacent blocks, such as one or more blocks in a row above a given block and / or a block to the left of a given block. By processing the multiplexed stream, the encoder spaces out the processing of the blocks in a given stream, thereby providing additional clock cycles to process the adjacent blocks on which the given block depends. For example, when a given block arrives at a stage that is dependent on a previously processed neighboring block, blocks to the left of the given block may be several stages ahead of the given block in the pipeline. This allows the encoder to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for processing of neighboring blocks in the pipeline to complete before processing the given block at a stage that is dependent on the neighboring block.

[0073] As indicated at 540, the wireless interface packets the compressed data and sends the packets to the device over the wireless connection.

[0074] As indicated at element 520, decomposing the pixel data into frequency bands allows the frequency bands to be prioritized by the encoder at element 530 and by the wireless interface at element 540. Typically, in image and video transmission, lower frequencies are more important, while higher frequencies are less important. Higher frequencies typically correspond to details in the image and can therefore be considered lower priority. Higher frequency bands contain a smaller percentage of energy in the image. Most of the energy is contained in lower frequency bands. Therefore, decomposing the pixel data into frequency bands provides a priority sorting for the data stream, which the encoder and wireless interface can utilize when encoding and transmitting the data stream. For example, in some embodiments, different compression techniques can be used on different frequency bands, where more aggressive compression is applied to lower priority bands and more conservative compression is applied to higher priority bands. As another example, the prioritization of frequency bands can help provide a moderate degradation of the VR / MR system. The performance of the wireless connection can be monitored, and feedback from the device can be considered to track the performance of the entire system. If the system falls behind for some reason, for example, if the wireless connection degrades and the bandwidth capacity of the wireless connection drops below a threshold, the encoder and the wireless interface may prioritize encoding and transmission of one or more lower frequency bands in the lower frequency bands and may reduce or lower encoding and / or transmission of one or more frequency levels in the frequency levels that have been assigned a lower priority level (e.g., one or more higher frequency bands in the higher frequency bands).

[0075] At 550, if there are more slices to be encoded and transmitted, the method returns to element 510 to process the next slice. Otherwise, at 560, if there are more frames to be encoded and transmitted, the method returns to element 510 to start processing the next frame.

[0076] Figure 6 According to at least some embodiments, Figure 2 A flow chart of a method of operation of a video encoding system is shown. Figure 6 The method can be used, for example, in Figure 4 420 of the execution. Figure 6 In the method, a video encoding system may use multiple encoders to perform tile-based encoding to process corresponding tiles from each slice.

[0077] As indicated at 600, the rendering engine renders a strip comprising a plurality of tiles (four tiles in this example), each tile comprising a plurality of pixel blocks (four 128x128 pixel blocks in this example).

[0078] As indicated at 610, the pre-filter component applies a lens warp and / or a foveal filter to the strip. In some embodiments, pre-filtering may include performing lens warping on the frame at the base station prior to wavelet transform. Lens warping is performed to correct distortion of the image introduced by the lens on the device (through which the image is viewed), thereby improving the quality of the image. In some embodiments, the device may store lens warp data of the lens, such as generated by a calibration process, and may provide the lens warp data to the base station via a wireless connection. The base station may then perform lens warping on the frame based on the lens warp data of the device. Performing lens warping on the base station in the pre-filter stage may reduce the resolution of the frame before performing wavelet transform and encoding, which may help improve compression and may reduce latency and bandwidth usage on the wireless link. In addition, by performing lens warping on the base station in the pre-filter stage rather than performing lens warping on the device after decoding, filtering of the image data may only need to be performed once, as opposed to performing filtering on the base station to reduce resolution before encoding and then performing lens warp filtering on the device.

[0079] In some embodiments, the pre-filtering at 610 may also include filtering to reduce the resolution in the peripheral area while maintaining a higher resolution in the foveal area. In some embodiments, the gaze tracking information obtained by the device can be used to identify the direction the user is currently looking at. The area of ​​the frame corresponding to the fovea (referred to as the foveal area) can be identified based at least in part on the determined gaze direction. The peripheral area (i.e., the portion of the frame outside the foveal area) can be pre-filtered based on understanding of the human visual system to reduce information, such as by filtering high frequency information and / or increasing color compression. Pre-filtering of the peripheral area can help provide improved frame compression.

[0080] In some embodiments, the video encoding system may include a single pre-filter component that processes a tile. In some embodiments, the video encoding system may include a separate pre-filter component for processing each tile. In some embodiments, the video encoding system may include multiple (e.g., 2) pre-filter components that each process multiple (e.g., 2) tiles.

[0081] As indicated at 620, the wavelet transform component applies a wavelet transform technique to the pixel blocks in the strip to decompose the pixel data into N (e.g., 7) frequency bands. The frequency bands are then organized into blocks (e.g., CTUs), which can be provided to a block-based encoder for encoding / compression. As an example, a frame can be divided into 128×128 blocks, and a two-level wavelet decomposition can be applied to each 128×128 block to generate 16 32×32 frequency data blocks representing seven frequency bands, such as Figure 3C shown.

[0082] In some embodiments, the video encoding system may include a single wavelet transform component that processes a tile. In some embodiments, the video encoding system may include a separate wavelet transform component for processing each tile. In some embodiments, the video encoding system may include multiple (e.g., 2) wavelet transform components that each process multiple (e.g., 2) tiles.

[0083] exist Figure 6 In the method, the exemplary video encoding system includes two encoders that are configured to encode blocks of frequency data from different ones of the slices generated in parallel at element 620. For example, a first encoder may be configured to encode blocks from slice 0 and slice 1, and a second encoder may be configured to encode blocks from slice 2 and slice 3. As indicated at 630A, the first encoder applies the encoding technique to the frequency bands in the blocks of tile 0 and tile 1, thereby multiplexing the processing of blocks from different frequency bands between two different tiles. As indicated at 630B, the second encoder applies the encoding technique to the frequency bands in the blocks of tile 2 and tile 3, thereby multiplexing the processing of blocks from different frequency bands between two different tiles.

[0084] The encoder may be, for example, a high-efficiency video coding (HEVC) encoder. However, in some embodiments, other encoding techniques may be used. As indicated at element 620, decomposing the pixel data into bands allows the encoder to buffer and process the bands as separate streams at elements 630A and 630B. Processing the bands as separate streams allows the encoder to multiplex the processing of independent streams. In block-based encoding methods such as HEVC encoding, blocks (called coding tree units (CTUs)) are processed in the pipeline at multiple stages; two or more blocks may be at different stages of the pipeline at a given clock cycle, and the block moves through the pipeline with the clock cycle. The processing of a given block may depend on one or more previously processed adjacent blocks, such as one or more blocks in a row above a given block and / or a block to the left of a given block. By processing multiplexed streams, the encoder spaces out the processing of blocks in a given stream, thereby providing additional clock cycles to process adjacent blocks on which a given block depends. For example, when a given block reaches a stage that depends on previously processed adjacent blocks, the block to the left of a given block may be several stages ahead of the given block in the pipeline. This allows the encoder to better handle dependencies on previously processed blocks and reduces or eliminates the need to wait for processing of neighboring blocks in the pipeline to complete before processing a given block at a stage that depends on the neighboring blocks.

[0085] As indicated at 640, the wireless interface packets the compressed data generated by the encoder at element 530 and sends the packets to the device over the wireless connection. As indicated by the arrow from 650 back to 600, the method continues as long as the user is using the VR / MR system.

[0086] As indicated at element 620, decomposing the pixel data into frequency bands allows the frequency bands to be prioritized by the encoder at elements 630A and 630B, and by the wireless interface at element 640. Typically, in image and video transmission, lower frequencies are more important, while higher frequencies are less important. Higher frequencies typically correspond to details in the image and can therefore be considered lower priority. Higher frequency bands contain a smaller percentage of energy in the image. Most of the energy is contained in lower frequency bands. Therefore, decomposing the pixel data into frequency bands provides a priority sorting for the data stream, which the encoder and wireless interface can utilize when encoding and transmitting the data stream. For example, in some embodiments, different compression techniques can be used on different frequency bands, where more aggressive compression is applied to lower priority bands and more conservative compression is applied to higher priority bands. As another example, the prioritization of frequency bands can help provide a moderate degradation of the VR / MR system. The performance of the wireless connection can be monitored, and feedback from the device can be considered to track the performance of the entire system. If the system falls behind for some reason, for example, if the wireless connection degrades and the bandwidth capacity of the wireless connection drops below a threshold, the encoder and the wireless interface may prioritize encoding and transmission of one or more lower frequency bands in the lower frequency bands and may reduce or lower encoding and / or transmission of one or more frequency levels in the frequency levels that have been assigned a lower priority level (e.g., one or more higher frequency bands in the higher frequency bands).

[0087] Although embodiments are described in which each slice is divided into four tiles and two encoders operate on corresponding tiles from each slice, in some embodiments, the slices may be divided into more tiles (e.g., six or eight tiles) and in some embodiments more encoders may be used (e.g., three or four encoders).

[0088] Rate Control

[0089] The encoder may employ a rate control method or algorithm to dynamically adjust encoder parameters to achieve a target bit rate. The rate control method may estimate the video bit rate based on the available bandwidth of the wireless connection. The estimated bit rate may be used to adjust encoder parameters to ensure that the encoded bit stream can be successfully transmitted while optimizing the use of the available bandwidth. Typically, rate control methods for video streaming systems work on a time constant of one or two frames to estimate the video bit rate and adjust encoder parameters accordingly.

[0090] A goal of a VR / MR system implementing an embodiment of a video encoding system as described herein may be to provide very low latency (e.g., 4-5 milliseconds) between the time virtual content is rendered on a base station and the time the virtual content is displayed on a device, while minimizing errors (e.g., dropped packets) and maximizing bandwidth usage over a wireless connection. The system's target low latency is a challenge for rate control in the encoder when encoding data to be transmitted to a device over a wireless connection.

[0091] In some embodiments, the video encoding system includes: a wavelet transform component that performs a two-layer wavelet decomposition that decomposes a pixel block (e.g., 64x64 or 128x128 block) from a strip of a frame into multiple (e.g., seven) different frequency bands; and a block-based encoder, such as HEVC that encodes the frequency band. In some embodiments, in order to perform rate control under the strict delay requirements of the VR / MR system, the rate control component can adjust the encoder parameters (e.g., the quantization parameter of the frequency band) for each strip, instead of working with a time constant of one or two frames to adjust the encoder parameters. In some embodiments, the frequency band of the strip can be buffered in a local memory after the wavelet transform, so that the complexity statistics on the frequency band can be calculated before encoding, for example, by a complexity analysis component of the video encoding system. In some embodiments, the complexity statistics may include, but are not limited to, one or more of the pixel gradient and size variance of the CTU block. The complexity statistics can then be used by the rate control component to determine the quantization parameter of the frequency band for modulating the rate for the current strip in the encoder. Computing complexity statistics for the buffered frequency bands allows the video encoding system to estimate the level of complexity in a slice prior to encoding the slice.

[0092] In some embodiments, the rate control component may examine the statistics of one or more previously processed frames to determine the actual number of bits used to encode the co-sited blocks or slices of the frame. This information may then be combined with the complexity statistics of the current slice to determine the quantization parameter for the current slice.

[0093] In some embodiments, the quantization parameters of the bands may be calculated jointly to optimize the overall display quality of the frame after decoder reconstruction and wavelet synthesis are performed on the device. The method for rate allocation between the bands is to find operating points in each of the bands that result in equal slopes on the operating rate distortion curve. In some embodiments, the operating rate distortion curve may be estimated using a combination of statistics from the entropy encoding stage of the encoder pipeline, where the fact is that the rate of the coefficients can be closely approximated by the quantization parameter step size 6.

[0094] In some embodiments, there may be a region of interest within an image (e.g., a foveal region) that should have a higher resolution than surrounding areas in the same image (e.g., a peripheral region). In some embodiments, the complexity statistics from a previous frame along with the foveal region from the previous frame may be stored and used to calculate quantization parameters for bands in a strip of the current frame. For example, for each block to be encoded, the foveal region of the previous frame may be used to modulate the complexity statistics calculated for the previous frame. The resulting complexity statistics may then be used in addition to the complexity statistics calculated for the strip of the current frame to determine the quantization parameter.

[0095] In some embodiments, the rate control component may adjust encoder parameters (e.g., quantization parameters for bands or priorities) at the block level rather than at the slice level. In some embodiments, complexity statistics may be calculated for each block of a slice, and the quantization parameter for the block may be determined.

[0096] Figure 7 According to at least some embodiments, Figure 1 or Figure 2 Block diagram of a video coding system shown in , where complexity statistics of frequency bands are calculated before encoding and used to generate quantization parameters for encoding.

[0097] In some embodiments, the rendered strip of the frame including the pixel data is first processed by a pre-filter 1004 component (e.g., an N-channel filter bank). In some embodiments, gaze tracking information obtained from the device can be used to identify the direction in which the user is currently looking. The area of ​​the frame corresponding to the fovea (referred to as the foveal area) can be identified based at least in part on the determined gaze direction. The peripheral area (i.e., the portion of the frame outside the foveal area) can be pre-filtered based on understanding of the human visual system to reduce information, such as by filtering high frequency information and / or increasing color compression.

[0098] The output of the pre-filter 1004 is processed by a wavelet transform 1006 component. In some embodiments, the wavelet transform 1006 component performs a two-layer wavelet decomposition that decomposes a pixel block (e.g., 64x64 or 128x128 block) from a strip of a frame into multiple (e.g., seven) different frequency bands. The frequency bands of the strip can be buffered 1008 after the wavelet transform 1005 so that the complexity statistics on the frequency bands of the strip can be calculated by the complexity analysis 1050 component of the video encoding system before encoding. In some embodiments, the complexity statistics may include, but are not limited to, one or more of the pixel gradient and size variance of the CTU block. The complexity analysis 1050 component provides the complexity statistics of the frequency bands in the strip to the rate control 1018 component of the encoder 1008. The rate control 1018 component then calculates the quantization parameter of each of the frequency bands in the strip based at least in part on the complexity statistics.

[0099] In some embodiments, the quantization parameters of the bands may be calculated jointly by the rate control 1018 component to allocate the rate between the bands after decoder reconstruction and wavelet synthesis are performed on the device while optimizing the overall display quality of the frame. The method for rate allocation between the bands is to find an operating point in each of the bands that results in an equal slope on the operating rate distortion curve. In some embodiments, the operating rate distortion curve may be estimated using a combination of statistics from the entropy encoding stage of the encoder pipeline, where the fact is that the rate of the coefficients can be closely approximated by the quantization parameter step size 6.

[0100] In some embodiments, the rate control component may examine the statistics of one or more previously processed frames to determine the actual number of bits used to encode the co-sited blocks or slices of the frame. This information may then be combined with the complexity statistics of the current slice to determine the quantization parameter for the current slice.

[0101] In some embodiments, there may be a region of interest within an image (e.g., a foveal region) that should have a higher resolution than surrounding areas in the same image (e.g., a peripheral region). In some embodiments, the complexity statistics from a previous frame along with the foveal region from the previous frame may be stored by the rate control 1018 component and used to calculate the quantization parameters for the bands in the strips of the current frame. For example, for each block to be encoded, the foveal region of the previous frame may be used to modulate the complexity statistics calculated for the previous frame. The resulting complexity statistics may then be used in addition to the complexity statistics calculated for the strips of the current frame to determine the quantization parameter.

[0102] The rate control 1018 component provides the quantization parameters of the bands in the slice to the block processing pipeline 1020, which uses the quantization parameters to encode the bands of the current slice. The encoded bands are provided to the wireless interface 1010, which groups and transmits the encoded bands to the device via the wireless connection 1080.

[0103] The goal of the video encoding system is to optimize the quality of the video stream to the device while maximizing bandwidth usage and minimizing the delay on the wireless connection 1080 to the device. In some embodiments, to help achieve this goal, the wireless interface 1010 can provide the current wireless connection information as feedback to the rate control 1018 component of the encoder 1008. The rate control component 1018 can use this information to determine the quantization parameters of the frequency band. In some embodiments, the wireless interface 1010 can track the instantaneous total bit rate of the wireless connection 1080, and can also track the discarded packet rate on the connection 1080. In some embodiments, the wireless interface 1010 feeds the current bit rate of the wireless connection 1080 back to the rate control 1018 component, which can use this information to allocate the available bit rate between different frequency bands. For example, if the bit rate of the wireless connection 1080 is currently low, the rate control 1018 component can adjust the quantization parameters of one or more frequency bands in the higher frequency band so that fewer bits are allocated to those frequency bands. In some embodiments, the wireless interface 1010 will also feed the current dropped packet rate for the wireless connection 1080 back to the rate control 1018 component; the rate control 1018 component can use the current dropped packet rate and the current bit rate to balance the adjustment of the quantization parameters of the frequency band to reduce or minimize the dropped packet rate while maximizing bandwidth usage at the current bit rate to optimize the quality of the video stream.

[0104] In some embodiments, the rate control 1018 component may adjust the quantization parameter of the frequency band at the block level rather than at the slice level.

[0105] Figure 8 According to at least some embodiments, Figure 7Flowchart of the method of operation of the video encoding system shown in . As indicated at 1100, the pre-filter component determines the foveal region of the frame and pre-filters the pixel data in the strip. In some embodiments, the rendered strip of the frame including the pixel data is first processed by the pre-filter component (e.g., an N-channel filter bank). In some embodiments, the gaze tracking information obtained from the device can be used to identify the direction in which the user is currently looking. The area of ​​the frame corresponding to the fovea (referred to as the foveal region) can be identified at least in part based on the determined gaze direction. The peripheral area (i.e., the portion of the frame outside the foveal region) can be pre-filtered based on understanding of the human visual system to reduce information, such as by filtering high-frequency information and / or increasing color compression.

[0106] As indicated at 1110, the wavelet transform component decomposes the pre-filtered pixel data into N (e.g., 7) frequency bands. In some embodiments, the wavelet transform 1006 component performs a two-layer wavelet decomposition that decomposes pixel blocks (e.g., 64x64 or 128x128 blocks) of a strip from a frame into multiple (e.g., seven) different frequency bands. The frequency bands of the strip can be buffered after the wavelet transform so that complexity statistics on the frequency bands of the strip can be calculated by the complexity analysis component of the video encoding system before encoding.

[0107] As indicated at 1120, the complexity analysis component calculates complexity statistics on the frequency band. In some embodiments, the complexity statistics may include, but are not limited to, one or more of pixel gradients and size variances of the CTU block. The complexity analysis component provides the complexity statistics of the frequency band in the slice to the rate control component of the encoder.

[0108] As indicated at 1130, the encoder rate control component determines a quantization parameter for each of the frequency bands in the slice based at least in part on the complexity statistics. In some embodiments, the quantization parameters for the frequency bands can be calculated jointly by the rate control 1018 component to allocate the rates between the frequency bands after performing decoder reconstruction and wavelet synthesis on the device while optimizing the overall display quality of the frame. The method for rate allocation between the frequency bands is to find an operating point in each of the frequency bands that results in an equal slope on the operating rate distortion curve. In some embodiments, the operating rate distortion curve can be estimated using a combination of statistics from the entropy encoding stage of the encoder pipeline, where the fact is that the rate of the coefficients can be closely approximated by the quantization parameter step size of 6.

[0109] In some embodiments, the rate control component may examine the statistics of one or more previously processed frames to determine the actual number of bits used to encode the co-sited blocks or slices of the frame. This information may then be combined with the complexity statistics of the current slice to determine the quantization parameter for the current slice.

[0110] In some embodiments, there may be a region of interest within an image (e.g., a foveal region) that should have a higher resolution than surrounding areas in the same image (e.g., a peripheral region). In some embodiments, the complexity statistics from a previous frame along with the foveal region from the previous frame may be stored by the rate control 1018 component and used to calculate the quantization parameters for the bands in the strips of the current frame. For example, for each block to be encoded, the foveal region of the previous frame may be used to modulate the complexity statistics calculated for the previous frame. The resulting complexity statistics may then be used in addition to the complexity statistics calculated for the strips of the current frame to determine the quantization parameter.

[0111] The encoder (e.g., HEVC encoder) uses the quantization parameter to apply a coding technique to the frequency band to compress the data, as indicated at 1140. The rate control component provides the quantization parameter for the frequency band in the slice to the block processing pipeline of the encoder, which uses the quantization parameter to encode the frequency band of the current slice. The encoded frequency band is provided to the wireless interface of the base station.

[0112] As indicated at 1150, the wireless interface groups the compressed data and sends the compressed data to the device via a wireless connection. In some embodiments, the wireless interface may provide the current wireless connection information as feedback to the rate control component of the encoder. The rate control component may use this information to determine the quantization parameter of the frequency band. In some embodiments, the wireless interface may track the instantaneous total bit rate of the wireless connection, and may also track the discarded packet rate on the connection. In some embodiments, the rate control component may use the current bit rate of the wireless connection to distribute the available bit rate between different frequency bands. In some embodiments, the rate control component may use the current discarded packet rate and the current bit rate to balance the adjustment of the quantization parameter of the frequency band, to reduce or minimize the discarded packet rate, so as to optimize the quality of the video stream under the current bit rate.

[0113] As indicated by the arrow from element 1160 returning to element 1100, the method may continue as long as there is data to be transferred to the device.

[0114] Exemplary VR / MR Systems

[0115] Fig. 9An exemplary VR / MR system 2000 that can implement a video encoding system according to at least some embodiments is shown. The VR / MR system 2000 may include at least one device 2150 (e.g., a laptop or notebook computer, a tablet or tablet device, a smart phone, a handheld computing device, or an HMD such as a headset, helmet, goggles, or glasses that can be worn by a user) and a computing device 2100 (referred to herein as a base station). The base station 2100 renders a VR frame or an MR frame including virtual content, encodes the frame, and transmits the encoded frame to the device 2150 via a wireless connection 2180 for decoding and display by the device 2150.

[0116] Base station 2100 and device 2150 may each include wireless communication technology that allows base station 2100 and device 2150 to communicate and exchange data via wireless connection 2180. In some embodiments, wireless connection 2180 may be implemented according to a proprietary wireless communication technology that provides a highly directional wireless link between device 2150 and base station 2100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0117] In some embodiments, device 2150 may include sensors that collect information about the user's environment (e.g., video, depth information, lighting information, etc.) and / or information about the user (e.g., the user's expression, eye movement, gaze direction, gestures, etc.). Device 2150 may transmit at least some of the information collected by the sensors to base station 2100 via wireless connection 2180. Base station 2100 may render a frame including virtual content for display by device 2150 based at least in part on the various information obtained from the sensors, encode the frame, and transmit the encoded frame to device 2150 via wireless connection 2180 for decoding and display to the user. In order to encode and transmit the frame, base station 2100 may implement the following: Figures 1 to 8 The video encoding system shown.

[0118] Fig.10 is a diagram showing how, according to some embodiments, Fig. 9Block diagram of functional components and processing in an exemplary VR / MR system shown. Device 2150 may be, but is not limited to, a laptop or notebook computer, a tablet or tablet device, a smart phone, a handheld computing device, or an HMD, such as a headset, a helmet, goggles, or glasses that can be worn by a user. Device 2150 may include a display 2156 component or subsystem that can implement any of various types of virtual or augmented reality display technologies. For example, HMD device 2150 may be a near-eye system that displays left and right images (the left and right images are viewed by the subject) on a screen in front of the user's eyes, such as DLP (digital light processing), LCD (liquid crystal display), and LCoS (liquid crystal on silicon) technology VR systems. As another example, HMD device 2150 may be a direct retinal projector system that scans left and right images to the subject's eyes pixel by pixel. To scan the image, the left projector and the right projector generate a light beam that is directed to a left reflective component and a right reflective component (e.g., an ellipsoidal reflector) located in front of the user's eyes; the reflective component reflects the light beam to the user's eyes. To create a three-dimensional (3D) effect, virtual content at different depths or distances in the 3D virtual view is shifted left or right in the two images as a function of triangulation of distance, with closer objects shifted more than farther objects.

[0119] The device 2150 may also include a controller 2154 configured to implement device-side functions of the VR / MR system 2000 as described herein. In some embodiments, the device 2150 may also include a memory 2170 configured to store software (code 2172) of device components of the VR / MR system 2000 that can be executed by the controller 2154, and data 2174 that can be used by the software when executed on the controller 2154. In various embodiments, the controller 2154 may be a single processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number). The controller 2154 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. For example, in various embodiments, the controller 2154 may include a general-purpose processor or an embedded processor that implements any of a variety of instruction set architectures (ISAs) such as x86, PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISAs. In a multi-processor system, each processor may implement the same ISA together, but it is not necessary. The controller 2154 may adopt any micro-architecture, including scalar, superscalar, pipeline, super-pipeline, out-of-order, in-order, speculative, non-speculative, etc., or a combination thereof. The controller 2154 may include a circuit that implements microcode technology. The controller 2154 may include one or more processing cores that are each configured to execute instructions. The controller 2154 may include one or more levels of cache, which may be of any size and any configuration (set association, direct mapping, etc.). In some embodiments, the controller 2154 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuit. Typically, the GPU may be configured to render an object to be displayed into a frame buffer (e.g., a frame buffer including pixel data of an entire frame). The GPU may include one or more graphics processors that may execute graphics software to perform some or all of the graphics operations or hardware acceleration of certain graphics operations. In some embodiments, the controller 2154 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder (codec), etc. In some embodiments, the controller 2154 may include at least one system on a chip (SOC).

[0120] The memory 2170 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single inline memory module (SIMM), a dual inline memory module (DIMM), etc. Alternatively, the device may be mounted in a chip stack configuration, a package stack configuration, or a multi-chip module configuration with an integrated circuit implementing the system.

[0121] In some embodiments, the sensor 2160 may include, but is not limited to, one or more gaze tracking sensors (e.g., an IR camera with an IR illumination source) that can be used to track the position and movement of the user's eyes. In some embodiments, there may be two gaze tracking sensors, each of which tracks a corresponding eye. In some embodiments, the information collected by the gaze tracking sensor can be used to adjust the rendering of the image by the base station 2100, and / or adjust the projection of the image performed by the projection system of the device 2150 based on the direction and angle of the user's eye viewing. For example, in some embodiments, the content of the image in the area around the location currently viewed by the user's eyes can be rendered with more details and can be at a higher resolution than the content in the area not viewed by the user, which allows the available image data processing time to be spent on the content viewed by the foveal region of the eye, rather than on the content viewed by the peripheral area of ​​the eye. Similarly, the content of the image in the area not viewed by the user can be compressed more than the content of the area around the point currently viewed by the user. In some embodiments, there may be two gaze tracking sensors, which are located on the inner surface of the device 2150, at a position where the sensor has a view of the corresponding eye in the user's eyes. However, in various embodiments, more or fewer gaze tracking sensors may be used, and the gaze tracking sensors may be positioned at other locations. In an exemplary non-limiting embodiment, each gaze tracking sensor may include an IR light source and an IR camera, such as a 400×400 pixel camera with a frame rate of 120 FPS or greater, a HFOV of 70 degrees, and a working distance of 10 millimeters (mm) to 80 mm.

[0122] In some embodiments, the device 2150 may include at least one inertial measurement unit (IMU) 2162 that is configured to detect the position, orientation, and / or motion of the device 2150 and provide the detected position, orientation, and / or motion data to a controller 2154 of the device 2150 and / or to the base station 2100.

[0123] The device 2150 may also include a wireless interface 2152 configured to communicate with an external base station 2100 via a wireless connection 2180 to send sensor input to the base station 2100 and receive compressed rendered frames, strips, or tiles from the base station 2100. In some embodiments, the wireless interface 2152 may implement a proprietary wireless communication technology that provides a highly directional wireless link between the device 2150 and the base station 2100. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0124] The base station 2100 may be an external device (e.g., a computing system, a game console, etc.) communicatively coupled to the device 2150 via the wireless interface 2180. The base station 2100 may include one or more of various types of processors (e.g., SOC, CPU, ISP, GPU, codec, and / or other components) for rendering, filtering, encoding, and transmitting video and / or images. The base station 2100 may render frames (each frame including a left image and a right image) including virtual content based at least in part on various inputs obtained from the sensor 2160 via the wireless connection 2180, filter and compress the rendered frames (or strips of frames) using a video encoding system as described herein, and transmit the compressed frames or strips to the device 2150 for display.

[0125] The base station 2100 may be or may include any type of computing system or computing device, such as a desktop computer, a laptop or notebook computer, a tablet or tablet device, a smart phone, a handheld computing device, a game controller, a gaming system, etc. The base station 2100 may include a controller 2110 including one or more processors that implement the base-side functions of the VR / MR system 2000 (including the video encoding system as described herein). The base station 2100 may also include a memory 2120 configured to store software (code 2122) of the base station components of the VR / MR system 2000 that can be executed by the base station controller 2110, and data 2124 that can be used by the software when executed on the controller 2110.

[0126] In various embodiments, the base station controller 2110 may be a single processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight or another suitable number). The controller 2110 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. For example, in various embodiments, the controller 2110 may include a general-purpose processor or embedded processor that implements any instruction set architecture in a variety of instruction set architectures (ISAs) such as x86, PowerPC, SPARC, RISC or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each processor may implement the same ISA together, but it is not necessary. The controller 2110 may adopt any microarchitecture, including scalar, superscalar, pipeline, superpipeline, out of order, in order, speculative, non-speculative, etc., or a combination thereof. The controller 2110 may include a circuit that implements microcode technology. The controller 2110 may include one or more processing cores that are each configured to execute instructions. The controller 2110 may include one or more levels of cache, which may be of any size and any configuration (set associative, direct mapping, etc.). In some embodiments, the controller 2110 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuit. Typically, the GPU may be configured to render an object to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). The GPU may include one or more graphics processors that may execute graphics software to perform some or all of the graphics operations or hardware acceleration of certain graphics operations. In some embodiments, the controller 2110 may include one or more other components for processing, rendering, filtering, and encoding video and / or images as described herein, such as one or more of various types of integrated circuits (ICs), image signal processors (ISPs), encoders / decoders (codecs), etc. In some embodiments, the controller 2110 may include at least one system on a chip (SOC).

[0127] The base station memory 2120 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single inline memory module (SIMM), a dual inline memory module (DIMM), etc. Alternatively, the device may be mounted in a chip stack configuration, a package stack configuration, or a multi-chip module configuration with an integrated circuit implementing the system.

[0128] The base station 2100 may also include one or more wireless technology interfaces 2130 configured to communicate with the device 2150 via the wireless connection 2180 to receive sensor input from the device 2150 and to send compressed frames, strips or tiles from the base station 2100 to the device 2150. In some embodiments, the wireless technology interface 2130 may implement a proprietary wireless communication technology that provides a highly directional wireless link between the device 2150 and the base station 2100. In some embodiments, the directionality and bandwidth of the wireless communication technology may support multiple devices 2150 communicating with the base station 2100 at the same time, thereby enabling multiple users to use the system 2000 in a co-located environment at the same time. However, in some embodiments, other commercial (e.g., Wi-Fi, Bluetooth, etc.) or proprietary wireless communication technologies may be used.

[0129] In some embodiments, the base station 2100 may be configured to render frames based at least in part on sensor 2160 input received from the device 2150 and transmit the frames to the device 2150 to provide a 3D virtual view to the user. In some embodiments, the virtual view may include rendering the user's environment, including rendering real objects in the user's environment based on video captured by one or more scene cameras (e.g., RGB (visible light) cameras) that capture high-quality, high-resolution video of the user's environment in real time for display. In some embodiments, the virtual view may also include virtual content (e.g., virtual objects, virtual labels for real objects, the user's avatar, etc.) rendered by the base station 2100 and composited with the projected 3D view of the user's real environment.

[0130] Although not in Fig. 9 and Fig.10, however, in some embodiments, the VR / MR system 2000 may include one or more other components. For example, the system may include a cursor control device (e.g., a mouse) for moving a virtual cursor in a 3D virtual view to interact with virtual content. Fig. 9 and Fig.10 A single device 2150 is shown, but in some embodiments, the VR / MR system 2000 may support multiple devices 2150 communicating with the base station 2100 simultaneously, thereby enabling multiple users to use the system simultaneously in a co-located environment.

[0131] In different embodiments, the methods described herein can be implemented in software, hardware or a combination thereof. In addition, the order of the frames of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from the present disclosure, various modifications and changes can obviously be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions and improvements are possible. Therefore, multiple examples can be provided for the components described as a single example in this article. The boundaries between various components, operations and data repositories are arbitrary to a certain extent, and specific operations are shown in the context of a specific exemplary configuration. Other allocations of functions are expected, and they may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions and improvements may fall within the scope of the embodiments defined in the following claims.

Claims

1. A video encoding system, comprising: Block-based encoders; a wavelet transform component, the wavelet transform component comprising one or more processors, the one or more processors of the wavelet transform component being configured to apply a wavelet transform to a plurality of pixel blocks from a stripe of a current video frame to decompose the pixel blocks from the stripe into a plurality of frequency bands and output frequency blocks including frequency data of the frequency bands of the stripe to a buffer; a complexity analysis component, the complexity analysis component comprising one or more processors, the one or more processors of the complexity analysis component configured to calculate complexity statistics for the buffered frequency bands across the slices and output the complexity statistics for the frequency bands of the slices to the encoder; Wherein, the encoder is configured as: determining a quantization parameter for each of the frequency bands of the slice based at least in part on the complexity statistics for the frequency bands of the slice; and applying an encoding technique to the buffered frequency blocks of the slice, wherein the encoding technique uses the determined quantization parameter for the frequency band of the slice to compress the frequency data in the frequency blocks; A wireless interface is configured to transmit the compressed frequency data of the strip to a receiving device via a wireless connection.

Images