Method and apparatus for video coding using palette mode

Abstract

To provide a system and method for video encoding and decoding using a palette mode.SOLUTION: The electronic device first receives, from a video bitstream having a hierarchical structure, a first syntax element associated with a first level of the hierarchical structure. In accordance with a determination that the first syntax element indicates that the palette mode is enabled for one or more coding units (CUs) below the first level in the video bitstream, reconstruct, from the video bitstream, pixel values of at least one of the one or more CUs in accordance with the corresponding palette table. On the other hand, in accordance with a determination that the first syntax element indicates that the palette mode is disabled for the one or more CUs, reconstruct, from the video bitstream, pixel values of any of the one or more CUs according to a non-palette scheme.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] This application relates generally to encoding and compressing video data, and more particularly to a method and system for video coding using palette mode. [Background technology]

[0002] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing devices, video streaming devices, etc. Electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression standards such as those defined by the MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC) standards. Video compression typically involves performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in the video data. Block-based video coding partitions a video frame into one or more slices, with each slice having multiple video blocks, sometimes referred to as coding tree units (CTUs). Each CTU may contain one coding unit (CU) or may be recursively divided into smaller CUs until a predetermined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU may be coded in either intra, inter, or IBC mode. Video blocks in an intra-coded (I) slice of a video frame are coded using spatial prediction with respect to reference samples in neighboring blocks within the same video frame. Video blocks in an inter-coded (P or B) slice of a video frame may use spatial prediction with respect to reference samples in neighboring blocks within the same video frame or temporal prediction with respect to reference samples in other earlier and / or later reference video frames.

[0003] Spatial or temporal prediction based on previously coded reference blocks, e.g., neighboring blocks, results in a predicted block for the current video block being coded. The process of finding the reference block may be accomplished by a block matching algorithm. Residual data representing pixel differences between the current block being coded and the predicted block is referred to as a residual block or prediction error. Inter-coded blocks are coded according to motion vectors that point to reference blocks in reference frames that form the predicted block and the residual block. The process of determining the motion vectors is typically referred to as motion estimation. Intra-coded blocks are coded according to an intra-prediction mode and the residual block. For further compression, the residual block may be transformed from the pixel domain to a transform domain, e.g., the frequency domain, resulting in residual transform coefficients that may be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, may be scanned to produce a one-dimensional vector of transform coefficients, which may then be entropy coded into a video bitstream to achieve further compression.

[0004] The encoded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by another electronic device having digital video capabilities or for transmission directly to the electronic device via wired or wireless connection. The electronic device then performs video decompression (the opposite process to video compression described above), for example, by parsing the encoded video bitstream to obtain syntax elements from the bitstream, reconstructing digital video data from the encoded video bitstream into its original form based at least in part on the syntax elements obtained from the bitstream, and rendering the reconstructed digital video data on a display of the electronic device.

[0005] As digital video quality increases from high definition to 4Kx2K or even 8Kx4K, the amount of video data to be encoded / decoded increases exponentially, which poses a constant challenge as to how the video data can be encoded / decoded more efficiently while maintaining the image quality of the decoded video data. Summary of the Invention [Problem to be solved by the invention]

[0006] This application describes implementations related to systems and methods for encoding and decoding video data, and more particularly, video encoding and decoding using palette mode. [Means for solving the problem]

[0007] According to a first aspect of the present application, a method for decoding video data includes receiving, from a video bitstream having a hierarchical structure, a first syntax element associated with a first level of the hierarchical structure; reconstructing, from the video bitstream, pixel values ​​of at least one of the one or more coding units (CUs) according to a corresponding palette table in accordance with a determination that the first syntax element indicates that palette mode is enabled for one or more CUs below the first level in the video bitstream; and reconstructing, from the video bitstream, pixel values ​​of any of the one or more CUs according to a non-palette scheme in accordance with a determination that the first syntax element indicates that palette mode is disabled for the one or more CUs.

[0008] According to a second aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory, the programs, when executed by the one or more processing units, causing the electronic device to perform the method for decoding video data described above.

[0009] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units, the programs, when executed by the one or more processing units, causing the electronic device to perform the method for decoding video data described above.

[0010] According to a fourth aspect of the present application, a method for encoding video data includes generating a first syntax element associated with a first level of the hierarchical structure for inclusion in a video bitstream having a hierarchical structure, the first syntax element indicating that palette mode is enabled for one or more coding units (CUs) below the first level in the video bitstream; encoding pixel values ​​of the one or more CUs (CUs), each CU having a corresponding palette table, and the first syntax element into the video bitstream; and outputting a video bitstream including the encoded one or more CUs and the first syntax element.

[0011] According to a fifth aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory, the programs, when executed by the one or more processing units, causing the electronic device to perform the method for encoding video data described above.

[0012] According to a sixth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units, the programs, when executed by the one or more processing units, causing the electronic device to perform the method for encoding video data described above.

[0013] The accompanying drawings, which are included to provide a further understanding of the implementations and which are incorporated in and constitute a part of this specification, illustrate the described implementations and, together with the description, serve to explain the underlying principles, and like reference numerals refer to corresponding parts. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example video encoding and decoding system according to some implementations of this disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an example video encoder according to some implementations of this disclosure. [Figure 3] FIG. 2 is a block diagram illustrating an example video decoder according to some implementations of this disclosure. [Figure 4A] FIG. 10 is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes, according to some implementations of this disclosure. [Figure 4B] FIG. 10 is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes, according to some implementations of this disclosure. [Figure 4C] FIG. 10 is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes, according to some implementations of this disclosure. [Figure 4D] FIG. 10 is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes, according to some implementations of this disclosure. [Figure 4E] FIG. 10 is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes, according to some implementations of this disclosure. [Figure 5] 1 is a block diagram illustrating an example of determining and using a palette table to encode video data, according to some implementations of the present disclosure. [Figure 6] 1 is a flowchart illustrating an example process by which a video encoder implements a technique for encoding video data using a palette-based scheme, according to some implementations of this disclosure. [Figure 7] 1 is a flowchart illustrating an example process by which a video decoder implements a technique for decoding video data using a palette-based scheme, according to some implementations of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used and the subject matter may be practiced without these specific details without departing from the scope of the claims. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0016] 1 is a block diagram illustrating an example system 10 for encoding and decoding video blocks in parallel, according to some implementations of the present disclosure. As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data that is subsequently decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0017] In some implementations, destination device 14 may receive encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, link 16 may include a communication medium that enables source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from source device 12 to destination device 14.

[0018] In some other implementations, the encoded video data may be transmitted from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by destination device 14 via input interface 28. Storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, storage device 32 may correspond to a file server or another intermediate storage device that may hold the encoded video data generated by source device 12. Destination device 14 may access the stored video data from storage device 32 via streaming or download. The file server may be any type of computer capable of storing encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data through any standard data connection, including a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from storage device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0019] As shown in FIG. 1 , source device 12 includes video source 18, video encoder 20, and output interface 22. Video source 18 may include sources such as a video capture device such as a video camera, a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As an example, if video source 18 is a video camera in a security surveillance system, source device 12 and destination device 14 may form a camera phone or video phone. However, implementations described herein may be applicable to video coding generally and may be applied to wireless and / or wired applications.

[0020] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by destination device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or a transmitter.

[0021] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or modem and receive encoded video data over link 16. The encoded video data communicated over link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communications medium, stored on a storage medium, or stored on a file server.

[0022] In some implementations, destination device 14 may include a display device 34, which may be an integrated display device and an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0023] Video encoder 20 and video decoder 30 may operate in accordance with proprietary or industry standards, such as VVC, HEVC, MPEG-4, Part 10, Advanced Video Coding (AVC), or extensions of such standards. It should be understood that the present application is not limited to a particular video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally assumed that video encoder 20 of source device 12 may be configured to encode video data in accordance with any of these current or future standards. Similarly, it is also generally assumed that video decoder 30 of destination device 14 may be configured to decode video data in accordance with any of these current or future standards.

[0024] Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, an electronic device may store instructions for the software on a suitable non-transitory computer-readable medium and execute those instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in this disclosure. Video encoder 20 and video decoder 30 may each be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device.

[0025] 2 is a block diagram illustrating an example video encoder 20 according to some implementations described in this application. Video encoder 20 may perform intra- and inter-predictive coding of video blocks within a video frame. Intra-predictive coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-predictive coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence.

[0026] As shown in FIG. 2, video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. Prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a partitioning unit 45, an intra-prediction processing unit 46, and an intra-block copy (BC) unit 48. In some implementations, video encoder 20 also includes an inverse quantization unit 58 for video block reconstruction, an inverse transform processing unit 60, and an adder 62. A deblocking filter (not shown) may be disposed between adder 62 and DPB 64 to filter block boundaries to remove block distortion artifacts from the reconstructed video. In addition to the deblocking filter, an in-loop filter (not shown) may be used to filter the output of adder 62. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be divided among one or more of the illustrated fixed or programmable hardware units.

[0027] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data for use in encoding video data by video encoder 20 (e.g., in intra- or inter-prediction coding modes). Video data memory 40 and DPB 64 may be formed by any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20 or off-chip with respect to those components.

[0028] As shown in FIG. 2, after receiving video data, partition unit 45 within prediction processing unit 41 partitions the video data into video blocks. This partitioning may include partitioning a video frame into slices, tiles, or other larger coding units (CUs) according to a predetermined partitioning structure, such as a quadtree structure, associated with the video data. The video frame may be divided into multiple video blocks (or sets of video blocks referred to as tiles). Prediction processing unit 41 may select one of multiple possible predictive coding modes, such as one of multiple intra-predictive coding modes or one of multiple inter-predictive coding modes, for the current video block based on error results (e.g., code rate and distortion level). Prediction processing unit 41 may provide the resulting intra- or inter-predictively coded block to adder 50 to generate a residual block and to adder 62 to reconstruct the coded block for later use as part of a reference frame. Prediction processing unit 41 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy coding unit 56.

[0029] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction processing unit 46 within prediction processing unit 41 may intra-predictively code the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may inter-predictively code the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may, for example, perform a plurality of independent coding modes to select an appropriate coding mode for each block of video data. 84, multiple encoding passes may be performed.

[0030] In some implementations, motion estimation unit 42 determines the inter-prediction mode for a current video frame by generating a motion vector that indicates the displacement of a prediction unit (PU) of a video block in a current video frame relative to a predictive block in a reference video frame according to a predetermined pattern within a sequence of video frames. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate motion for video blocks. The motion vector may indicate, for example, the displacement of a PU of a video block in a current video frame or picture relative to a predictive block in a reference frame (or other coding unit) relative to a current block being coded within the current frame (or other coding unit). The predetermined pattern may designate a video frame in the sequence as a P frame or a B frame. Intra BC unit 48 may determine vectors, such as block vectors, for intra BC coding in a manner similar to the determination of motion vectors by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine block vectors.

[0031] A prediction block is a block of a reference frame that is deemed to closely match the PU of the video block to be encoded, with respect to pixel differences that may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values ​​for sub-integer pixel positions of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional-pixel positions of the reference frame. Thus, motion estimation unit 42 may perform motion search for full-pixel and fractional-pixel positions and output motion vectors with fractional-pixel precision.

[0032] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-predictively coded frame by comparing the position of the PU with the position of a predictive block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy coding unit 56.

[0033] Motion compensation performed by motion compensation unit 44 may involve fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference frame lists, obtain the predictive block from DPB 64, and forward the predictive block to summer 50. Summer 50 then subtracts pixel values ​​of the predictive block provided by motion compensation unit 44 from pixel values ​​of the current video block being coded to form a residual video block of pixel difference values. The pixel difference values ​​forming the residual video block may include luma or chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frame for use by video decoder 30 in decoding the video blocks of the video frame. The syntax elements may include, for example, syntax elements defining the motion vector used to identify the predictive block, any flags indicating a prediction mode, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0034] In some implementations, the intra BC unit 48 may generate a vector and fetch a predictive block in a manner similar to that described above in connection with the motion estimation unit 42 and the motion compensation unit 44, except that the predictive block is in the same frame as the current block being coded, and the vector is referred to as a block vector, as opposed to a motion vector. In particular, the intra BC unit 48 may determine an intra prediction mode to use to code the current block. In some examples, the intra BC unit 48 may code the current block using various intra prediction modes, e.g., during separate coding passes, and test their performance using rate-distortion analysis. The intra BC unit 48 may then select an appropriate intra prediction mode to use from the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values ​​for the various tested intra prediction modes using rate-distortion analysis, and select the intra prediction mode with the best rate-distortion characteristics from the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to create the coded block, along with the bit rate (i.e., number of bits) used to create the coded block. Intra BC unit 48 may calculate ratios from the distortion and rate for various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0035] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction, according to implementations described herein. In either case, for intra block copying, the predictive block may be a block that is deemed to closely match the block to be coded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics, and identifying the predictive block may include calculating values ​​for sub-integer pixel positions.

[0036] Regardless of whether the predictive block is from the same frame via intra prediction or a different frame via inter prediction, video encoder 20 may form a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded, thereby forming pixel difference values. The pixel difference values ​​that form the residual video block may include differences in both luma and chroma components.

[0037] Intra-prediction processing unit 84 may intra-predict the current video block as an alternative to the inter-prediction performed by motion estimation unit 42 and motion compensation unit 44 or the intra-block copy prediction performed by intra BC unit 48, as described above. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. To do so, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or a mode selection unit in some examples) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode in the bitstream.

[0038] After prediction processing unit 41 determines a predictive block for the current video block via either inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and provided to transform processing unit 52. Transform processing unit 52 converts the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0039] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54, which quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix containing the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform this scan.

[0040] Following quantization, entropy coding unit 56 entropy codes the quantized transform coefficients into a video bitstream, using, for example, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The coded bitstream may then be transmitted to video decoder 30 or archived in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy coding unit 56 may entropy code motion vectors and other syntax elements for the current video frame being coded.

[0041] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block into the pixel domain to generate reference blocks for predicting other video blocks. As described above, motion compensation unit 44 may generate motion-compensated prediction blocks from one or more reference blocks of frames stored in DPB 64. Motion compensation unit 44 may apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0042] Adder 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to create a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0043] 3 is a block diagram illustrating an example video decoder 30 according to some implementations of the present application. Video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. Prediction processing unit 81 further includes a motion compensation unit 82, an intra-prediction processing unit 84, and an intra-BC unit 85. Video decoder 30 may perform a decoding process that is generally opposite to the encoding process described above for video encoder 20 in connection with FIG. 2. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, while intra-prediction unit 84 may generate prediction data based on an intra-prediction mode indicator received from entropy decoding unit 80.

[0044] In some examples, a unit of the video decoder 30 may be tasked with performing an implementation of the present application. Also, in some examples, the implementation of the present disclosure may be divided among one or more units of the video decoder 30. For example, the intra BC unit 85 may perform an implementation of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction processing unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0045] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a coded picture buffer (CPB) that stores coded video data from the coded video bitstream. A decoded picture buffer (DPB) 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in intra- or inter-prediction coding modes). Video data memory 79 and DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, video data memory 79 and DPB 92 are shown in Figure 3 as two separate components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30 or off-chip with respect to those components.

[0046] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks and associated syntax elements of encoded video frames. Video decoder 30 may receive the syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors and other syntax elements to prediction processing unit 81.

[0047] If a video frame is coded as an intra-prediction coded (I) frame or for intra-coded predictive blocks in other types of frames, intra-prediction processing unit 84 of prediction processing unit 81 may generate predictive data for video blocks of the current video frame based on a signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0048] If a video frame is encoded as an inter-predictively coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 creates one or more predictive blocks for video blocks of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the predictive blocks may be created from a reference frame in one of the reference frame lists. Video decoder 30 may configure the reference frame lists, List 0 and List 1, using a default configuration technique based on the reference frames stored in DPB 92.

[0049] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 creates a predictive block for the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The predictive block may be within the same reconstructed region of the picture as the current video block as determined by video encoder 20.

[0050] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for video blocks of the current video frame by parsing the motion vectors and other syntax elements, and then use the prediction information to create a predictive block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra- or inter-prediction) used to encode the video blocks of the video frame, the inter-prediction frame type (e.g., B or P), configuration information for one or more of the reference frame lists for the frame, the motion vector for each inter-predictively coded video block of the frame, the inter-prediction status for each inter-predictively coded video block of the frame, and other information for decoding the video blocks in the current video frame.

[0051] Similarly, intra BC unit 85 may use some of the received syntax elements, such as flags, to determine that the current video block was predicted using an intra BC mode, configuration information about which video blocks of the frame are within the reconstructed region and should be stored in DPB 92, block vectors for each intra BC predicted video block of the frame, intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding the video blocks in the current video frame.

[0052] Motion compensation unit 82 may also perform interpolation using an interpolation filter to calculate interpolated values ​​for sub-integer pixels of the reference block, as used by video encoder 20 during encoding of the video block. In this case, motion compensation unit 82 may determine the interpolation filter used by video encoder 20 from the received syntax element and use the interpolation filter to create the predictive block.

[0053] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameter calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct the residual block in the pixel domain.

[0054] After motion compensation unit 82 or intra BC unit 85 generates a predictive block for the current video block based on the vectors and other syntax elements, summer 90 reconstructs a decoded video block for the current video block by adding the residual block from inverse transform processing unit 88 and the corresponding predictive block generated by motion compensation unit 82 and intra BC unit 85. To further process the decoded video block, an in-loop filter (not shown) may be disposed between summer 90 and DPB 92. The decoded video block for a given frame is then stored in DPB 92, which stores reference frames used for subsequent motion compensation of the next video block. DPB 92, or a memory device separate from DPB 92, may store the decoded video for later presentation on a display device, such as display device 34 of FIG. 1.

[0055] In a typical video coding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other cases, a frame may be monochromatic and therefore include only one two-dimensional array of luma samples.

[0056] As shown in FIG. 4A, video encoder 20 (or more specifically, partitioning unit 45) generates a coded representation of a frame by first partitioning the frame into a set of coding tree units (CTUs). A video frame may include an integer number of CTUs, sequentially ordered in raster scan order from left to right and top to bottom. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by video encoder 20 in the sequence parameter set so that all CTUs in a video sequence have the same size, either 128x128, 64x64, 32x32, or 16x16. However, it should be noted that the present application is not necessarily limited to a particular size. As shown in FIG. 4B, each CTU may include one coding tree block (CTB) for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements used to encode the samples in the coding tree blocks. The syntax elements describe the characteristics of different types of units of pixel blocks to be coded, including inter or intra prediction, intra prediction mode, motion vectors, and other parameters, and how the video sequence may be reconstructed at video decoder 30. In a monochrome picture or a picture with three distinct color planes, a CTU may include a single coding tree block and syntax elements used to code samples of the coding tree block. A coding tree block may be an N×N block of samples.

[0057] To achieve better performance, video encoder 20 may recursively perform quadtree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination of both, on the coding tree blocks of a CTU to divide the CTU into smaller coding units (CUs). As shown in FIG. 4C , a 64×64 CTU 400 is first partitioned into four smaller CUs, each with a block size of 32×32. Of the four smaller CUs, CU 410 and CU 420 are each partitioned into four CUs with a block size of 16×16. Two 16×16 CUs, 430 and 440, are further partitioned into four CUs with a block size of 8×8. FIG. 4D illustrates a quadtree data structure showing the final result of the partitioning process of CTU 400 as shown in FIG. 4C , where each leaf node of the quadtree corresponds to one CU, each ranging in size from 32×32 to 8×8. Similar to the CTU illustrated in FIG. 4B, each CU may include a coding block (CB) of luma samples and two corresponding coding blocks (CB) of chroma samples of the same size frame, as well as syntax elements used to encode the samples of the coding block. In a monochrome picture or a picture with three distinct color planes, a CU may include a single coding block and syntax structures used to encode the samples of the coding block. It should be noted that the quadtree partitioning illustrated in FIGS. 4C and 4D is for illustrative purposes only, and a CTU may be divided into CUs to suit various local characteristics based on quadtree / ternary tree / binary tree partitioning. In a multi-tree structure, a CTU is partitioned by a quadtree structure, and each leaf CU of the quadtree may be further partitioned by binary tree and ternary tree structures. As shown in FIG. 4E, there are five partition types: quadtree, horizontal bipartition, vertical bipartition, horizontal tripartition, and vertical tripartition.

[0058] In some implementations, video encoder 20 may further partition the coding blocks of a CU into one or more M×N prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples to which the same inter or intra prediction is applied. A prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to predict the prediction block. In a monochrome picture or a picture with three separate color planes, a PU may include a single prediction block and syntax structures used to predict the prediction block. Video encoder 20 may generate predicted luma, Cb, and Cr blocks for the luma, Cb, and Cr prediction blocks of each PU of a CU.

[0059] Video encoder 20 may generate predictive blocks for a PU using intra prediction or inter prediction. If video encoder 20 generates predictive blocks for a PU using intra prediction, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of a frame associated with the PU. If video encoder 20 generates predictive blocks for the PU using inter prediction, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0060] After video encoder 20 generates predicted luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from its original luma coding block, such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, video encoder 20 may generate Cb and Cr residual blocks for the CU, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU indicates a difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU, respectively.

[0061] Further, as illustrated in FIG. 4C , video encoder 20 may use quadtree partitioning to decompose the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and syntax structures used to transform the samples of the transform block.

[0062] Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to a Cb transform block of the TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0063] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to a process by which transform coefficients are quantized to provide further compression by possibly reducing the amount of data used to represent the transform coefficients. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a sequence of bits forming a representation of the encoded frame and associated data to be stored in storage device 32 or transmitted to destination device 14.

[0064] After receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct frames of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally the opposite of the encoding process performed by video encoder 20. For example, video decoder 30 may inverse transform coefficient blocks associated with TUs of the current CU to reconstruct residual blocks associated with TUs of the current CU. Video decoder 30 also reconstructs coding blocks of the current CU by adding samples of predictive blocks for PUs of the current CU to corresponding samples of transform blocks of TUs of the current CU. After reconstructing the coding blocks for each CU of the frame, video decoder 30 may reconstruct the frame.

[0065] As mentioned above, video coding achieves video compression using two main modes: intra-frame prediction (or intra-prediction) and inter-frame prediction (or inter-prediction). Palette-based coding is another encoding method adopted by many video coding standards. In palette-based coding, which may be particularly suitable for encoding screen-generated content, a video coder (e.g., video encoder 20 or video decoder 30) forms a palette table of colors to represent a given block of video data. The palette table contains the most dominant (e.g., frequently used) pixel values ​​in the given block. Pixel values ​​that do not frequently appear in the video data of a given block are either not included in the palette table or are included in the palette table as escape colors.

[0066] Each entry in the palette table contains an index to a corresponding pixel value in the palette table. The palette index for a sample in a block may be coded to indicate which entry from the palette table should be used to predict or reconstruct which sample. This palette mode begins with the process of generating a palette predictor for the first block of a picture, slice, tile, or other such grouping of video blocks. As described below, palette predictors for subsequent video blocks are typically generated by updating a previously used palette predictor. For illustrative purposes, it is assumed that the palette predictor is defined at the picture level. In other words, a picture may contain multiple coding blocks, each with its own palette table, but there is one palette predictor for the entire picture.

[0067] To reduce the bits required to signal palette entries in a video bitstream, a video decoder may use a palette predictor to determine new palette entries in a palette table used to reconstruct a video block. For example, the palette predictor may include palette entries from a previously used palette table, or may be initialized with the most recently used palette table by including all entries from the most recently used palette table. In some implementations, the palette predictor may include fewer than all entries from the most recently used palette table, while incorporating some entries from other previously used palette tables. The palette predictor may have the same size as the palette table used to encode a different block, or may be larger or smaller than the palette table used to encode a different block. In one example, the palette predictor is implemented as a first-in-first-out (FIFO) table containing 64 palette entries.

[0068] To generate a palette table for a block of video data from the palette predictor, the video decoder may receive a one-bit flag for each entry of the palette predictor from the encoded video bitstream. The one-bit flag may have a first value (e.g., a binary 1) indicating that the associated entry of the palette predictor should be included in the palette table, or a second value (e.g., a binary 0) indicating that the associated entry of the palette predictor should not be included in the palette table. If the size of the palette predictor is larger than the palette table used for the block of video data, the video decoder may stop receiving further flags once a maximum size for the palette table has been reached.

[0069] In some implementations, some entries in the palette table may be directly signaled in the coded video bitstream instead of being determined using a palette predictor. For such entries, the video decoder may receive three separate m-bit values ​​from the coded video bitstream indicating the pixel value for the luma and two chroma components associated with the entry, where m represents the bit depth of the video data. Compared to the multiple m-bit values ​​required for directly signaled palette entries, palette entries derived from a palette predictor require only a one-bit flag. Thus, signaling some or all palette entries using a palette predictor can significantly reduce the number of bits required to signal new palette table entries, thereby improving the overall coding efficiency of palette mode coding.

[0070] In many cases, a palette predictor for a block is determined based on the palette table used to encode one or more previously encoded blocks. However, when encoding the first coding tree unit in a picture, slice, or tile, the palette table of the previously encoded block may not be available. Therefore, a palette predictor cannot be generated using entries in the previously used palette table. In such cases, a set of palette predictor initializers may be signaled in a sequence parameter set (SPS) and / or a picture parameter set (PPS), whose values ​​are used to generate a palette predictor when the previously used palette table is not available. An SPS generally refers to a syntactic structure of syntax elements that apply to a series of consecutive coded video pictures, referred to as a coded video sequence (CVS), determined by the content of syntax elements found in a PPS referenced by syntax elements found in each slice segment header. A PPS generally refers to a syntactic structure of syntax elements that apply to one or more individual pictures in a CVS, determined by the content of syntax elements found in each slice segment header. Thus, an SPS is generally considered a higher level syntactic structure than a PPS, which means that the syntax elements contained in an SPS generally change less frequently and apply to a larger portion of the video data compared to the syntax elements contained in a PPS.

[0071] Figure 5 is a block diagram illustrating an example of determining and using a palette table to encode video data in a picture 500 according to some implementations of the present disclosure. The picture 500 includes a first block 510 associated with a first palette table 520 and a second block 530 associated with a second palette table 540. Because the second block 530 is to the right of the first block 510, the second palette table 540 may be determined based on the first palette table 520. A palette predictor 550 is associated with the picture 500 and is used to collect zero or more palette entries from the first palette table 520 and construct zero or more palette entries in the second palette table 540. It should be noted that the various blocks illustrated in Figure 5 may correspond to CTUs, CUs, PUs, or TUs as described above, and that the blocks are not limited to the block structure of any particular coding standard and may conform to future block-based coding standards.

[0072] Generally, a palette table contains multiple pixel values ​​that are dominant and / or representative for the currently coded block (e.g., block 510 or 530 in FIG. 5). In some examples, a video coder (e.g., video encoder 20 or video decoder 30) may encode a palette table separately for each color component of a block. For example, video encoder 20 may encode a palette table for the luma component of the block, another palette table for the chroma Cb component of the block, and yet another palette table for the chroma Cr component of the block. In this case, first palette table 520 and second palette table 540 may each be multiple palette tables. In other examples, video encoder 20 may encode a single palette table for all color components of the block. In this case, the i-th entry in the palette table is a triplet of (Yi, Cbi, Cri), with each value corresponding to one component of a pixel. Therefore, the representations of first palette table 520 and second palette table 540 are merely exemplary and not intended to be limiting.

[0073] As described herein, rather than directly encoding the actual pixel values ​​of the first block 510, a video coder (such as video encoder 20 or video decoder 30) may encode the pixels of the first block 510 using indices I1,...,IN using a palette-based coding scheme. For example, for each pixel in the first block 510, the video encoder 20 may encode an index value for that pixel, which is associated with a pixel value in the first palette table 520. The video encoder 20 may encode the first palette table 520 and transmit it in the coded video data bitstream for use by the video decoder 30 for palette-based decoding at the decoder side. Generally, one or more palette tables may be transmitted for each block or shared among different blocks. The video decoder 30 may obtain the index values ​​from the video bitstream generated by the video encoder 20 and reconstruct the pixel values ​​using the pixel values ​​in the first palette table 520 corresponding to the index values. In other words, for each index value for the block, video decoder 30 may determine an entry in first palette table 520. Video decoder 30 then replaces each index value in the block with the pixel value specified by the determined entry in first palette table 520.

[0074] In some implementations, a video coder (e.g., video encoder 20 or video decoder 30) determines second palette table 540 based at least in part on palette predictor 550 associated with picture 500. Palette predictor 550 may include some or all of the entries of first palette table 520, and possibly entries from other palette tables. In some examples, palette predictor 550 is implemented using a first-in, first-out table, where adding an entry from first palette table 520 to palette predictor 550 removes the oldest entry currently in palette predictor 550 to keep palette predictor 550 below a maximum size. In other examples, palette predictor 550 may be updated and / or maintained using different techniques.

[0075] In one example, video encoder 20 may encode pred_palette_flag for each block (e.g., second block 530) to indicate whether the palette table for the block is predicted from one or more palette tables associated with one or more other blocks, such as neighboring block 510. For example, if the value of such flag is binary 1, video decoder 30 may determine that second palette table 540 for second block 530 is predicted from one or more previously decoded palette tables, and therefore a new palette table for second block 540 will not be included in the video bitstream that includes pred_palette_flag. If such flag is binary 0, video decoder 30 may determine that second palette table 540 for second block 530 will be included in the video bitstream as a new palette table. In some examples, pred_palette_flag may be coded separately for different color components of the block (e.g., three flags, one for Y, one for Cb, and one for Cr for a video block in YCbCr space). In another example, a single pred_palette_flag may be coded for all color components of a block.

[0076] In the above example, pred_palette_flag is signaled for each block to indicate that all entries in the palette table for the current block are predicted. This means that the second palette table 540 is identical to the first palette table 520, and no additional information is signaled. In other examples, one or more syntax elements may be signaled for each entry. That is, for each entry in the previous palette table, a flag may be signaled to indicate whether the entry is present in the current palette table. If a palette entry is not predicted, the palette entry may be explicitly signaled. In other examples, these two methods may be combined.

[0077] When predicting second palette table 540 according to first palette table 520, video encoder 20 and / or video decoder 30 may identify the location of the block from which the predicted palette table is determined. The predicted palette table may be associated with one or more neighboring blocks of the currently coded block, i.e., second block 530. As illustrated in FIG. 5, video encoder 20 and / or video decoder 30 may identify the location of the left neighboring block, i.e., first block 510, when determining the predicted palette table for second block 530. In other examples, video encoder 20 and / or video decoder 30 may identify the location of one or more blocks at other positions relative to second block 530, such as the block above in picture 500. In another example, the palette table for the last block in scanning order using palette mode may be used as the predicted palette table for second block 530.

[0078] Video encoder 20 and / or video decoder 30 may determine a block for palette prediction according to a predetermined order of block positions. For example, video encoder 20 and / or video decoder 30 may first identify a left neighboring block, i.e., first block 510, for palette prediction. If the left neighboring block is not available for prediction (e.g., the left neighboring block is coded in a mode other than a palette-based coding mode, such as an intra-prediction mode or an inter-prediction mode, or is located at the leftmost edge of the picture or slice), video encoder 20 and / or video decoder 30 may identify an upper neighboring block in picture 500. Video encoder 20 and / or video decoder 30 may continue searching available blocks according to a predetermined order of block positions until it identifies the location of a block having a palette table available for palette prediction. In some examples, video encoder 20 and / or video decoder 30 may determine a prediction palette based on reconstructed samples of multiple blocks and / or neighboring blocks by applying one or more equations, functions, rules, etc. to generate a prediction palette table based on the palette table of one or a combination of multiple neighboring blocks (spatially or in scan order). In one example, a prediction palette table containing palette entries from one or more previously encoded neighboring blocks includes multiple entries N. In this case, video encoder 20 first transmits to video decoder 30 the prediction palette table, i.e., a binary vector V having the same size as N. Each entry in the binary vector indicates whether the corresponding entry in the prediction palette table is reused or copied to the palette table for the current block. For example, V(i)=1 means that the i-th entry in the prediction palette table for a neighboring block, which may have a different index in the current block, is reused or copied to the palette table for the current block.

[0079] In yet another example, video encoder 20 and / or video decoder 30 may construct a candidate list including multiple potential candidates for palette prediction. In such an example, video encoder 20 may encode an index into the candidate list to indicate a candidate block in the list from which a current block is selected for palette prediction. Video decoder 30 may construct the candidate list in a similar manner, decode the index, and use the decoded index to select the palette of the corresponding block for use with the current block. In another example, the palette table of the indicated candidate block in the list may be used as a prediction palette table for prediction for each entry in the palette table for the current block.

[0080] In some implementations, one or more syntax elements may indicate whether an entire palette table, such as the second palette table 540, is predicted from a predicted palette (e.g., the first palette table 520, which may be composed of entries from one or more previously encoded blocks), or whether particular entries in the second palette table 540 are predicted. For example, an initial syntax element may indicate whether all entries in the second palette table 540 are predicted. If the initial syntax element indicates that not all entries are predicted (e.g., a flag having a value of binary 0), one or more additional syntax elements may indicate which entries in the second palette table 540 are predicted from the predicted palette table.

[0081] In some implementations, the size of the palette table, e.g., in terms of the number of pixel values ​​it contains, may be fixed or may be signaled using one or more syntax elements in the encoded bitstream.

[0082] In some implementations, video encoder 20 may encode pixels of a block without exactly matching pixel values ​​in the palette table with actual pixel values ​​in the corresponding block of video data. For example, video encoder 20 and video decoder 30 may combine or combine (i.e., quantize) different entries in the palette table if the pixel values ​​of the entries are within a predetermined range of each other. In other words, if there is already an existing pixel value that is within the error margin of the new pixel value, the new pixel value is not added to the palette table, while the sample in the block that corresponds to the new pixel value is encoded with the index of the existing pixel value. Note that this lossy encoding process does not affect the operation of video decoder 30, which may decode pixel values ​​in the same way regardless of whether a particular palette table is lossless or lossy.

[0083] In some implementations, video encoder 20 may select an entry in the palette table as a predicted pixel value for encoding pixel values ​​in the block. Then, video encoder 20 may determine the difference between the actual pixel value and the selected entry as a residual and encode the residual. Video encoder 20 may generate a residual block including residual values ​​for pixels in the block predicted by the entry in the palette table and then apply transform and quantization to the residual block (as described above in connection with FIG. 2). In this manner, video encoder 20 may generate quantized residual transform coefficients. In another example, the residual block may be encoded losslessly (without transform and quantization) or without transform. Video decoder 30 may inverse transform and inverse quantize the transform coefficients to reconstruct the residual block and then reconstruct the pixel values ​​using the predicted palette entry values ​​and the residual values ​​for the pixel values.

[0084] In some implementations, video encoder 20 may determine an error threshold, referred to as a delta value, to construct the palette table. For example, if the actual pixel value for a location in the block causes an absolute difference between the actual pixel value and an existing pixel value entry in the palette table that is less than or equal to the delta value, video encoder 20 may transmit an index value to identify the corresponding index of a pixel value entry in the palette table for use in reconstructing the actual pixel value for that location. If the actual pixel value for a location in the block causes an absolute difference between the actual pixel value and an existing pixel value entry in the palette table that is greater than the delta value, video encoder 20 may transmit the actual pixel value and add it to the palette table as a new entry. To construct the palette table, video decoder 30 may use the delta value signaled by the encoder, rely on a fixed or known delta value, or estimate or derive the delta value.

[0085] As described above, video encoder 20 and / or video decoder 30 may use coding modes including an intra-prediction mode, an inter-prediction mode, a lossless coding palette mode, and a lossy coding palette mode when encoding video data. Video encoder 20 and video decoder 30 may encode one or more syntax elements that indicate whether palette-based coding is enabled. For example, for each block, video encoder 20 may encode a syntax element that indicates whether a palette-based coding mode should be used for that block (e.g., a CU or PU). For example, this syntax element may be signaled in the coded video bitstream at the block level (e.g., a CU level) and then received by video decoder 30 when decoding the coded video bitstream.

[0086] In some implementations, the above-mentioned syntax elements may be transmitted at a level higher than the block level. For example, video encoder 20 may signal such syntax elements at the slice level, tile level, PPS level, or SPS level. In this case, a value equal to 1 indicates that all blocks at this level or below are coded using palette mode, such that no additional mode information, e.g., palette mode or other modes, is signaled at the block level. A value equal to 0 indicates that none of the blocks at this level or below are coded using palette mode.

[0087] In some implementations, enabling palette mode by a syntax element at a higher level does not mean that each block at or below this higher level must be coded in palette mode. Rather, a separate CU-level or even TU-level syntax element may be needed to indicate whether a CU- or TU-level block is coded in palette mode, and if so, whether a corresponding palette table should be configured. In some implementations, the video coder (e.g., video encoder 20 and video decoder 30) selects a threshold value (e.g., 32) for the number of samples in a block for the minimum block size so that palette mode is not allowed for blocks whose block size is less than the threshold value. In this case, no syntax element is signaled for such blocks. Note that the threshold value for the minimum block size may be explicitly signaled in the bitstream or implicitly set as a default value adapted by both video encoder 20 and video decoder 30.

[0088] A pixel value at one location in a block may be the same (or within a delta value of) a pixel value at another location in the block. For example, it is common for adjacent pixel locations in a block to have the same pixel value or to be mapped to the same index value in a palette table. Thus, video encoder 20 may encode one or more syntax elements that indicate multiple consecutive pixels or index values ​​in a given scanning order that have the same pixel value or index value. A series of similarly valued pixels or index values ​​may be referred to herein as a “run.” For example, if two consecutive pixels or indexes in a given scanning order have different values, the run is equal to 0. If two consecutive pixels or indexes in a given scanning order have the same value, but a third pixel or index in the scanning order has a different value, the run is equal to 1. For three consecutive indices or pixels that have the same value, the run is 2, and so on. Video decoder 30 may obtain the syntax element that indicates the run from the encoded bitstream and use that data to determine the number of consecutive locations that have the same pixel or index value.

[0089] FIG. 6 is a flowchart illustrating an example process 600 for a video encoder to implement a technique for encoding video data using a palette-based scheme according to some implementations of this disclosure. For example, video encoder 20 is configured to encode a video bitstream using a palette mode, where the video bitstream is organized into a hierarchical structure, e.g., as illustrated in FIGS. 4C and 4E , respectively, in which each picture in the video is partitioned into multiple CTUs, and each CTU is further divided into multiple CUs of different shapes and sizes. To implement the palette-based scheme, video encoder 20 generates a first syntax element associated with a first level of the hierarchical structure for inclusion in the video stream (610). As described above, the first level associated with the first syntax element is selected to be a level higher than the CU level, e.g., a tile, slice, or even a picture level. The first syntax element may be stored as part of an SPS, PPS, tile group header, or slice header. When the first syntax element has a value of binary 1, this indicates that palette mode is enabled for one or more coding units (CUs) below the first level in the video bitstream.

[0090] Next, video encoder 20 encodes the pixel values ​​and the first syntax element of one or more CUs into a video bitstream, each CU having a corresponding palette table (630). For example, for each CU to be encoded into the video bitstream, video encoder 20 generates a second syntax element associated with the CU (630-1). As described above, even if the first syntax element indicates that palette mode is enabled for one or more CUs, this does not mean that each individual CU is necessarily coded according to a palette table. Rather, it is the value of the second syntax element that determines whether the video blocks of a particular CU are coded according to palette mode. Assuming that the second syntax element has a value of binary 1, indicating that palette mode is enabled for the CU, video encoder 20 then constructs a palette table for the CU (630-3).

[0091] Various techniques for constructing the palette table are described above in connection with FIG. 5. For example, a palette predictor may be used to construct the palette table; in some implementations, the palette predictor is a FIFO table that holds a set of palette entries most frequently used by the video data. Using the palette table, video encoder 20 then identifies samples in the video blocks of the CU and determines the pixel values ​​of the samples and palette indexes for the samples in the palette table (630-5). As described above, different possibilities exist for samples in a CU. First, there is an existing palette entry in the palette table that corresponds to the sample in the CU. If so, the palette index of this existing palette entry is used to represent the sample in the video bitstream. Second, there is no existing palette entry that matches the pixel value of the sample. If so, video encoder 20 may add a new entry to the palette table and use the palette index of the new entry to represent the sample. In this case, the new entry may be used to represent other samples in the CU that have the same or similar pixel values ​​(within a delta value). In some implementations, video encoder 20 may encode the pixel value of the sample as an escape color entry in the palette table. In either case, video encoder 20 encodes (630-7) the determined palette index corresponding to the sample into the video bitstream.

[0092] In some implementations, video encoder 20 may select a second syntax value of binary 0 for a particular CU, indicating that palette mode is disabled for the CU. In this case, video encoder 20 may select to encode the CU using another prediction scheme, such as intra prediction or inter prediction, and encode the corresponding syntax element accordingly. In other words, the first syntax element indicating that palette mode is enabled may still allow certain CUs below the first level to select a non-palette mode. In contrast, if the first syntax element is set to binary 0, indicating that palette mode is disabled at the first level, none of the CUs below the first level will be coded using palette mode, and therefore the second syntax element or palette table will not be coded into the video bitstream.

[0093] Finally, video encoder 20 outputs an encoded video bitstream including the encoded one or more CUs and the first syntax element at the first level and the second syntax element at the CU level to video decoder 30 or a storage device (650), as shown in FIG. 1. In some implementations, the first level has an associated block size that is equal to or greater than a predetermined threshold associated with one or more CUs below the first level. For example, assume that an ancestor node having a size of 128 samples is trisected into three CUs having sizes of 32, 64, and 32 samples, respectively. If the predetermined threshold used to determine the first level for sharing palette modes is 64, the three CUs are three leaf nodes that share the same palette mode. In some implementations, there is a lower limit (e.g., 32 samples) on the predetermined threshold so that palette mode is not enabled for blocks having 32 or fewer samples for coding efficiency. In some implementations, the first syntax element and the second syntax element are each a one-bit flag.

[0094] In some implementations, under palette mode, a CU is divided into multiple segments, each containing multiple samples (e.g., M samples), where M is a positive number between 16 and 32. For each segment, CABAC parsing and / or encoding of palette-related constructs, such as palette index values, palette index runs, and quantized colors, is independent of those of other segments in the same CU. To achieve this, all CABAC parsing dependencies (e.g., context modeling) and decoding dependencies (e.g., copy-above mode) under palette mode are disabled across adjacent segments.

[0095] In some implementations, different methods may be used to divide a CU into multiple segments under palette mode, for example, based on the traverse scan order: the first M samples along the scan order are grouped into segment 1, the next M samples along the scan order are grouped into segment 2, and so on. In another example, a CU may be divided into multiple segments based on a binary, ternary, or quadtree partitioning structure. Within each segment, the traverse scan order may also be used for palette encoding of the segment. For example, the number of index values ​​for the segment may be signaled first, followed by signaling the actual palette index value for the entire segment using truncated binary coding. Both the index number and palette index value are coded in bypass mode, which groups index-related bypass bins together. Then, an index run is signaled. Finally, component escape values ​​corresponding to escape samples in the segment are grouped together and coded in bypass mode.

[0096] As mentioned above, different block size thresholds may be used to identify shared palette nodes. In one embodiment, one fixed threshold is shared by both the encoder and decoder without signaling. In another embodiment, it is proposed that one syntax element signals the shared palette threshold in the bitstream.

[0097] 7 is a flowchart illustrating an example process by which video decoder 30 implements a technique for decoding video data using a palette-based scheme according to some implementations of this disclosure. For example, video decoder 30 is configured to decode a video bitstream using a palette mode, where the video bitstream is organized into a hierarchical structure, e.g., as illustrated in FIGS. 4C and 4E , respectively, in which each picture in the video is partitioned into multiple CTUs, and each CTU is further divided into multiple CUs of different shapes and sizes. To implement the palette-based scheme, video decoder 30 receives a first syntax element from the video bitstream associated with a first level of the hierarchical structure (710). As described above, the first level associated with the first syntax element is selected to be a level higher than the CU level, e.g., a tile, slice, or even a picture level. The first syntax element is generated by video encoder 20 and may be stored as part of an SPS, PPS, tile group header, or slice header. When the first syntax element has a value of binary 1, this indicates that palette mode is enabled for one or more coding units (CUs) below the first level in the video bitstream.

[0098] Based on the value of 1 of the first syntax element, video decoder 30 reconstructs pixel values ​​of at least one of one or more CUs from the video bitstream according to the corresponding palette table (730). For example, for each CU to be coded into the video bitstream, video decoder 30 receives a second syntax element associated with the CU (730-1). As described above, even if the first syntax element indicates that palette mode is enabled for one or more CUs, this does not mean that each individual CU is necessarily coded according to a palette table. It is the value of the second syntax element that determines whether the video block of a particular CU is coded according to the palette mode. Assuming that the second syntax element has a value of binary 1, indicating that palette mode is enabled for the CU, video decoder 30 reconstructs a palette table for each CU from the video bitstream (730-3).

[0099] Various techniques for constructing the palette table are described above in connection with FIG. 5. For example, a palette predictor may be used to construct the palette table; in some implementations, the palette predictor is a FIFO table that holds a set of palette entries most frequently used by the video data. Using the palette table, video decoder 30 then identifies a sample in the video block of the CU, determines a palette index and then a pixel value for the sample in the palette table, and then reconstructs the pixel value for the sample (730-5). As described above, reconstructing the pixel value may require inverse quantization and inverse transformation of a residual value for the sample, which is added to a pixel value from the palette table as the sample's reconstructed pixel value. In some implementations, video decoder 30 may reconstruct the sample's pixel value from an escape color entry in the palette table.

[0100] If the first syntax element has a value of 0, indicating that palette mode is disabled for the CU, video decoder 30 reconstructs pixel values ​​of one or more CUs from the video bitstream according to a non-paletted scheme (750). As noted above, video decoder 30 may reconstruct the CU using another mode, such as intra-prediction or inter-prediction as described above. Note that all features related to the first and second syntax elements described above in connection with FIG. 6 apply to the palette-based decoding process described herein in connection with FIG. 7.

[0101] In some implementations, a cross-component linear model (CCLM) is used to generate a chroma palette prediction from a luma palette prediction. In one example, the CCLM may be calculated using neighboring luma and chroma samples. After the linear model is determined, a chroma palette prediction may be calculated based on the luma palette table of the same CU along with the linear model. In one example, the chroma palette prediction may be derived as follows: pred C (i,j)=α·rec L '(i,j)+β In the formula, pred C (i,j) represents the predicted saturation palette in CU, and rec L '(i,j) represents the reconstructed luma palette sample of the same CU. The linear model parameters α and β are derived, and different derivation methods may be used. One exemplary method is to use two samples in the luma palette table, namely the minimum luma sample A(x A ,y A ) and maximum brightness sample B(x B ,y B ) is a linear relationship between luminance and saturation values ​​from (x A ,y A ) are the luminance and chroma values ​​for sample A, and (x B ,y B) are the luminance and saturation values ​​for sample B. The linear model parameters α and β are obtained according to the following equations:

[0102]

number

[0103] In some implementations, different contexts based on the shape of the current block are used to signal the traverse scan direction under palette mode. Depending on the shape of the current block, different CABAC contexts may be selected, which results in different CABAC probabilities being used. Such contexts may also depend on the traverse scan direction of neighboring blocks.

[0104] In some implementations, signaling of the cross scan direction may be conditionally omitted depending on the shape of the current block. In this case, video decoder 30 infers the cross scan direction based on the shape of the current block. For example, if the block has an aspect ratio above a certain threshold, its cross scan direction is not signaled under palette mode, but is inferred to be the same as the long side of the block. Alternatively, its cross scan direction is signaled normally. In another example, if the block has an aspect ratio above a certain threshold, its cross scan direction is not signaled under palette mode. Video decoder 30 infers the cross scan direction to be the same as the short side of the block. Alternatively, its cross scan direction is signaled normally.

[0105] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media or communication media, including any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. As such, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the implementations described herein. A computer program product may include computer-readable media.

[0106] The terms used in the description of implementations herein are for the purpose of describing particular implementations only and are not intended to limit the scope of the claims. When used in the description of implementations and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "comprise" and / or "comprising," as used herein, specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0107] Furthermore, although terms such as "first," "second," etc. may be used herein to describe various elements, it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first electrode may be referred to as a second electrode, and a second electrode may be referred to as a first electrode, without departing from the scope of implementation. The first electrode and the second electrode are both electrodes, but they are not the same electrode.

[0108] The description in this application has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications, variations, and alternative implementations will be apparent to those skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. The embodiments have been chosen and described to best explain the principles and practical applications of the invention and to enable others skilled in the art to understand the invention in various implementations and to best utilize the underlying principles and various implementations, with various modifications as suitable for the particular use contemplated. Therefore, it is to be understood that the scope of the claims should not be limited to the specific implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. 1. A method for decoding video data, comprising: receiving, from a video bitstream having a hierarchical structure, a first syntax element associated with a first level of the hierarchical structure; according to determining that the first syntax element indicates that palette mode is enabled for one or more coding units (CUs) below the first level in the video bitstream, reconstructing pixel values ​​of at least one of the one or more CUs from the video bitstream according to a corresponding palette table; In accordance with a determination that the first syntax element indicates that palette mode is disabled for the one or more CUs, reconstructing the pixel values ​​of any of the one or more CUs from the video bitstream according to a non-paletted scheme.

2. The method of claim 1 , wherein the first syntax element is in one of a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, and a slice header.

3. The method of claim 1 , wherein the corresponding palette table is shared by the one or more CUs.

4. The method of claim 1 , wherein the first level has an associated block size greater than a predetermined threshold associated with the one or more CUs below the first level.

5. The method of claim 4 , wherein the predetermined threshold is 32 or greater.

6. The method of claim 4 , wherein the predetermined threshold is greater than 16.

7. The method of claim 1 , wherein the first syntax element comprises a one-bit flag.

8. Reconstructing pixel values ​​of at least one of the one or more coding units (CUs) from the video bitstream according to a corresponding palette table includes: receiving from the video bitstream a second syntax element associated with each CU of the one or more CUs; In response to determining that the second syntax element indicates that the palette mode is enabled for the respective CU, reconstructing a palette table for each of the CUs from the video bitstream; reconstructing the pixel values ​​of each CU from the video bitstream using the reconstructed palette table; 2. The method of claim 1, further comprising: reconstructing the pixel values ​​of the respective CU from the video bitstream according to the non-paletted scheme in accordance with determining that the second syntax element indicates that the palette mode is disabled for the respective CU.

9. 2. The method of claim 1, wherein each of the one or more CUs is divided into a plurality of segments based on a predetermined partition tree structure, and each segment has its own set of palette mode parameters including a total number of palette indexes and a corresponding set of palette indexes associated with the first-level palette table.

10. one or more processing units; a memory coupled to the one or more processing units; and a plurality of programs stored in said memory which, when executed by said one or more processing units, cause the electronic device to perform the method of claims 1 to 9.

11. 10. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, the plurality of programs, when executed by the one or more processing units, causing the electronic device to perform the method of claims 1 to 9.

12. 1. A method for encoding video data, comprising: generating a first syntax element associated with a first level of a hierarchical structure for inclusion in a video bitstream having a hierarchical structure, the first syntax element indicating that palette mode is enabled for one or more coding units (CUs) in the video bitstream below the first level; encoding pixel values ​​of the one or more CUs, each CU having a corresponding palette table, and the first syntax element into the video bitstream; outputting the video bitstream including the encoded one or more CUs and the first syntax element.

13. The method of claim 12 , wherein the first syntax element is in one of a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, and a slice header.

14. The method of claim 12 , wherein the first level has an associated block size that is greater than or equal to a predetermined threshold associated with the one or more CUs below the first level.

15. The method of claim 14 , wherein the predetermined threshold is 32 or greater.

16. The method of claim 14 , wherein the predetermined threshold is greater than 16.

17. The method of claim 12 , wherein the first syntax element comprises a one-bit flag.

18. encoding pixel values ​​of the one or more coding units (CUs), each CU having a corresponding palette table, and the first syntax element into the video bitstream, generating a second syntax element associated with each CU of the one or more CUs for inclusion in the video bitstream; In response to determining that the second syntax element indicates that the palette mode is enabled for the respective CU, constructing a palette table for each CU; determining a palette index for each sample in the respective CU from the palette table; The method of claim 12 , further comprising encoding the determined palette index corresponding to the sample into the video bitstream.

19. 13. The method of claim 12, wherein each of the one or more CUs is divided into a plurality of segments based on a predetermined partition tree structure, and each segment has its own set of palette mode parameters including a total number of palette indices and a corresponding set of palette indices associated with the first-level palette table.

20. one or more processing units; a memory coupled to the one or more processing units; and a plurality of programs stored in said memory which, when executed by said one or more processing units, cause the electronic device to perform the method of claims 12 to 19.

21. 20. A non-transitory computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, the plurality of programs, when executed by the one or more processing units, causing the electronic device to perform the method of claims 12 to 19.

Images