Method, electronic device, non-transitory computer-readable storage medium and computer program for video encoding in 4:4:4 chroma format

Abstract

To provide: an electronic device that performs a method of decoding video data; and the method.SOLUTION: The method according to the invention comprises: receiving, from the bitstream, a first syntax element in a slice header of a slice that indicates whether luma mapping with chroma scaling (LMCS) is applied to a coding unit within the slice; receiving a second syntax element for the coding unit that indicates whether the coding unit is coded using adaptive color-space transformation (ACT); if the second syntax element has a non-zero value, decoding the coding unit by applying inverse ACT to convert luma and chroma residuals of the coding unit from a transformed color space to an original color space of the video data; and if the first syntax element has a non-zero value, decoding the coding unit by performing inverse luma mapping to the luma samples and inverse scaling to the chroma residuals of the coding unit after performing the inverse ACT.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 914,282, filed October 11, 2019, entitled "METHODS AND APPARATUS OF VIDEO CODING IN 4:4:4 CHROMA FORMAT," and U.S. Provisional Application No. 62 / 923,390, filed October 18, 2019, entitled "METHODS AND APPARATUS OF VIDEO CODING IN 4:4:4 CHROMA FORMAT," the entire contents of both of which are incorporated herein by reference.

[0002] This application relates generally to video data encoding and compression, and more particularly to a method and system for performing adaptive color space transformation (ACT) using chroma residual scaling. [Background technology]

[0003] Digital video is supported by a variety of electronic devices, including digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing devices, and video streaming devices. Electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression standards defined in the MPEG-4, ITU-TH.263, ITU-TH.264 / MPEG-4, Part 10, ADVANCED Video Coding (AVC), HIGH EFFICIENCY Video Coding (HEVC), and VERSATILE Video Coding (VVC) standards. Video compression typically involves performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in the video data. In block-based video coding, a video frame is divided into one or more slices, each of which contains multiple video blocks, also known as coding tree units (CTUs). Each CTU contains one coding unit (CU) or may be recursively divided into smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU may be coded in either intra, inter, or IBC mode. Video blocks in an intra-coded (I) slice of a video frame are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same video frame. Video blocks in an inter-coded (P or B) slice of a video frame may use spatial prediction with respect to reference samples in neighboring blocks in the same video frame, or temporal prediction with respect to reference samples in other previous and / or future reference video frames.

[0004] Spatial or temporal prediction based on previously encoded reference blocks, e.g., neighboring blocks, yields a predicted block for the current video block being coded. The process of finding the reference block may be performed by a block matching algorithm. Residual data representing pixel differences between the current block being coded and the predicted block is called a residual block or prediction error. Inter-coded blocks are encoded according to a motion vector and residual block that point to a reference block in a reference frame that forms the predicted block. The process of determining the motion vector is typically called motion estimation. Intra-coded blocks are encoded according to an intra-prediction mode and residual block. For further compression, the residual block is transformed from the pixel domain to a transform domain, e.g., the frequency domain, to obtain residual transform coefficients, which may then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, are scanned to generate a one-dimensional vector of transform coefficients, which are then entropy encoded into a video bitstream to achieve further compression.

[0005] The encoded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by other electronic devices with digital video capabilities, or transmitted directly to the electronic device via wired or wireless connections. The electronic device then performs video decompression (the opposite process to video compression described above), for example, by parsing the encoded video bitstream to obtain syntax elements from the bitstream and reconstructing the digital video data from the encoded video bitstream into its original format based at least in part on the syntax elements obtained from the bitstream, and renders the reconstructed digital video data on a display of the electronic device.

[0006] As digital video quality moves from high definition to 4Kx2K or even 8Kx4K, the amount of video data to be encoded / decoded increases exponentially. There is always a challenge on how to encode / decode the video data more efficiently while maintaining the image quality of the decoded video data.

[0007] Certain video content, e.g., screen content video, is encoded in a 4:4:4 chroma format, in which all three components (luminance and both chroma components) have the same resolution. Although the 4:4:4 chroma format contains more redundancy than the 4:2:0 chroma format and the 4:2:2 chroma format (making it less suitable for achieving good compression efficiency), the 4:4:4 chroma format remains the recommended encoding format for many applications that require high fidelity to preserve color information, such as sharp edges, in the decoded video. Given the redundancy present in 4:4:4 chroma format video, there is evidence that significant coding improvements can be achieved by exploiting the relationships between the three color components of 4:4:4 video (e.g., Y, Cb, Cr in the YCbCr domain, or G, B, R in the RGB domain). Due to these relationships, adaptive color space transformation (ACT) tools are used to exploit the relationships between the three color components during the development of the HEVC Screen Content Coding (SCC) extension. Summary of the Invention

[0008] This application describes implementations for encoding and decoding video data, and more particularly for systems and methods that perform adaptive color space conversion (ACT) using luminance mapping and chroma scaling (LMCS).

[0009] According to a first aspect of the present application, a method for decoding video data includes receiving, from a bitstream, a first syntax element in a slice header of a slice, the first syntax element indicating whether luma mapping with chroma scaling (LMCS) is applied to a coding unit in the slice; receiving a second syntax element for the coding unit, the second syntax element indicating whether the coding unit is coded using adaptive color space transformation (ACT); decoding the coding unit by applying inverse ACT to convert luma and chroma residuals of the coding unit from a transformed color space of the video data to an original color space if the second syntax element has a non-zero value; and after performing the inverse ACT, decoding the coding unit by performing inverse luma mapping to luma samples of the coding unit and inverse scaling to chroma residuals of the coding unit if the first syntax element has a non-zero value.

[0010] According to a second aspect of the present application, a method for decoding video data includes receiving, from a bitstream, video data corresponding to a coding unit, the coding unit being coded using an intra-prediction mode; receiving a first syntax element from the video data, the first syntax element indicating whether the coding unit is coded using adaptive color space transformation (ACT); if the first syntax element has a zero value, receiving one or more syntax elements from the video data, the one or more syntax elements indicating whether chroma components of the coding unit are coded using block differential pulse coding modulation (BDPCM); if the first syntax element has a non-zero value, assigning default values ​​to one or more syntax elements associated with BDPCM; and decoding the coding unit from the video data in accordance with the first syntax element associated with ACT and the one or more syntax elements associated with BDPCM.

[0011] According to a third aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory, which, when executed by the one or more processing units, cause the electronic device to perform the method for decoding video data as described above.

[0012] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units, the programs, when executed by the one or more processing units, causing the electronic device to perform the method for decoding video data as described above. [Brief explanation of the drawings]

[0013] The accompanying drawings, which are included in and incorporated in this specification to provide a further understanding of the implementations, illustrate the described implementations and, together with the description, serve to interpret the underlying principles, with like reference numerals referring to corresponding parts.

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example video encoding and decoding system according to some implementations of this disclosure.

[0015] [Figure 2] FIG. 2 is a block diagram illustrating an example video encoder according to some implementations of this disclosure.

[0016] [Figure 3] FIG. 3 is a block diagram illustrating an example video decoder according to some implementations of this disclosure.

[0017] [Figure 4A] FIG. 4A is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4B]FIG. 4B is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4C] FIG. 4C is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4D] FIG. 4D is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4E] FIG. 4E is a block diagram illustrating how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure.

[0018] [Figure 5A] FIG. 5A is a block diagram illustrating an example of applying an adaptive color space conversion (ACT) technique to convert residuals between RGB color space and YCgCo color space, according to some implementations of this disclosure. [Figure 5B] FIG. 5B is a block diagram illustrating an example of applying an adaptive color space conversion (ACT) technique to convert residuals between RGB color space and YCgCo color space, according to some implementations of this disclosure.

[0019] [Figure 6] FIG. 6 is a block diagram of applying a technique of luma mapping with chroma scaling (LMCS) in an exemplary video data decoding process, in accordance with some implementations of this disclosure.

[0020] [Figure 7] FIG. 7 is a block diagram illustrating an example video decoding process by which a video decoder performs an inverse adaptive color space conversion (ACT) technique, in accordance with some implementations of this disclosure.

[0021] [Figure 8A]FIG. 8A is a block diagram illustrating an example video decoding process by which a video decoder performs techniques of inverse adaptive color space conversion (ACT) and luminance mapping with chroma scaling (LMCS), in accordance with some implementations of this disclosure. [Figure 8B] FIG. 8B is a block diagram illustrating an example video decoding process by which a video decoder performs techniques of inverse adaptive color space conversion (ACT) and luminance mapping with saturation scaling (LMCS), in accordance with some implementations of this disclosure.

[0022] [Figure 9] FIG. 9 is a block diagram illustrating example decoding logic during adaptive color space conversion (ACT) and block differential pulse coded modulation (BDPCM) in accordance with some implementations of this disclosure.

[0023] [Figure 10] FIG. 10 is a flowchart illustrating an example process by which a video decoder decodes video data by performing an inverse adaptive color space conversion with saturation scaling according to some implementations of this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] Reference numbers are given to details of exemplary embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives can be employed without departing from the scope of the claims, and that the subject matter can be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein can be implemented in many types of electronic devices with digital video capabilities.

[0025] 1 is a block diagram illustrating an example system 10 for encoding and decoding video blocks in parallel according to some implementations of the present disclosure. As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data to be subsequently decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0026] In some implementations, destination device 14 can receive encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, link 16 may include a communication medium that enables source device 12 to transmit encoded video data directly to destination 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. The communication medium may include routers, switches, base stations, , or any other equipment that may be useful in facilitating communication from source device 12 to destination device 14 .

[0027] In some other implementations, the encoded video data may be transmitted from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by destination device 14 via input interface 28. Storage device 32 may include any of a variety of distributed or locally accessible data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, storage device 32 may correspond to a file server or another intermediate storage device capable of holding encoded video data generated by source device 12. Destination device 14 may access the stored video data from storage device 32 via streaming or download. The file server may be any type of computer capable of storing encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 can access the encoded video data over any standard data connection, including a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. Transmission of the encoded video data from storage device 32 can be a streaming transmission, a download transmission, or a combination of both.

[0028] 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As an example, if video source 18 is a video camera in a security surveillance system, source device 12 and destination device 14 may form a camera phone or a video phone. However, the implementations described herein are applicable to video encoding generally and can be applied to wireless and / or wired applications.

[0029] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data may (or alternatively) be stored in storage device 32 and later accessed by destination device 14 or another device for decoding and / or playback. Output interface 22 may further include a modem and / or a transmitter.

[0030] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem and may receive encoded video data over link 16. The encoded video data communicated over link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communications medium, stored on a storage medium, or stored on a file server.

[0031] In some implementations, destination device 14 may include a display device 34 that is an integrated display device and an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0032] Video encoder 20 and video decoder 30 may operate in accordance with proprietary or industry standards, such as VVC, HEVC, MPEG-4 Part 10, Advanced Video Coding (AVC), or extensions of such standards. It will be understood that the present application is not limited to a particular video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data in accordance with any of these present or future standards. Similarly, it is generally contemplated that video decoder 30 of destination device 14 may be configured to decode video data in accordance with any of these present or future standards.

[0033] Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, an electronic device can store instructions for the software on a suitable non-transitory computer-readable medium and utilize one or more processors to execute the instructions in hardware to perform the video encoding / decoding operations disclosed in this disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, which may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0034] 2 is a block diagram illustrating an example video encoder 20 according to some implementations described herein. Video encoder 20 can perform intra- and inter-predictive coding of video blocks within video frames. Intra-predictive coding relies on spatial prediction to reduce or remove spatial redundancy of video data within a given video frame or image. Inter-predictive coding relies on temporal prediction to reduce or remove temporal redundancy of video data within adjacent video frames or images of a video sequence.

[0035] As shown in FIG. 2, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, summer 50, transform processing unit 52, quantization unit 54, and entropy encoding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, partitioning unit 45, intra-prediction processing unit 46, and intra-block copy (BC) unit 48. In some implementations, video encoder 20 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62 for video block reconstruction. A deblocking filter (not shown) may be located between summer 62 and DPB 64 to filter block boundaries and remove blockiness artifacts from the reconstructed image. In addition to the deblocking filter, an in-loop filter (not shown) may also be used to filter the output of summer 62. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be divided among one or more of the illustrated fixed or programmable hardware units.

[0036] Video data memory 40 can store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data used in encoding video data by video encoder 20 (e.g., in intra- or inter-prediction coding mode). Video data memory 40 and DPB 64 may be formed by any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20 or off-chip relative to those components.

[0037] As shown in FIG. 2, after receiving the video data, partitioning unit 45 within prediction processing unit 41 partitions the video data into video blocks. This partitioning may include dividing the video frame into slices, tiles, or other larger coding units (CUs) according to a predefined partitioning structure, such as a quadtree structure, associated with the video data. The video frame may be divided into multiple video blocks (or sets of video blocks called tiles). Prediction processing unit 41 may select one of multiple possible predictive coding modes for the current video block based on the error result (e.g., coding rate and distortion level), where the possible predictive coding mode may be one of multiple intra-predictive coding modes or one of multiple inter-predictive coding modes. Prediction processing unit 41 may provide the resulting intra- or inter-predictively coded block to summer 50 to generate a residual block, which may then be provided to summer 62 to reconstruct a block to be encoded for use as part of a reference frame. Prediction processing unit 41 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy encoding unit 56.

[0038] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding on the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding on the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple coding passes on each block of video data to, for example, select an appropriate coding mode.

[0039] In some implementations, motion estimation unit 42 determines the inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, where the motion vector indicates the displacement of a prediction unit (PU) of a video block in the current video frame relative to a predictive block in a reference video frame. Motion estimation performed by motion estimation unit 42 is a motion vector generation process that estimates the motion of a video block. The motion vector may, for example, indicate the displacement of a PU of a video block in a current video frame or image relative to a predictive block in a reference frame (or other coded unit) relative to the current block being coded in the current frame (or other coded unit). The predetermined pattern may designate a video frame in the sequence as a P frame or a B frame. Intra BC unit 48 may determine, for example, a vector of block vectors for intra BC coding in a manner similar to determining motion vectors by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine the block vectors.

[0040] A prediction block is a block of a reference frame that is identified as closely matching a PU of a video block to be encoded with respect to pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values ​​for sub-integer pixel locations of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel locations, eighth-pixel locations, or other fractional pixel locations of the reference frame. Thus, motion estimation unit 42 may perform motion searches for whole and fractional pixel locations and output motion vectors with fractional-pixel accuracy.

[0041] Motion estimation unit 42 calculates a motion vector for a PU by comparing the position of the PU of a video block in an inter-predictively coded frame with the position of a predictive block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), which respectively identify one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.

[0042] The motion compensation performed by motion compensation unit 44 may involve fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector of the PU of the current video block, motion compensation unit 44 may locate the predictive block pointed to by the motion vector in one of the reference frame lists, retrieve the predictive block from DPB 64, and forward the predictive block to summer 50. Summer 50 then forms a residual video block of pixel difference values ​​by subtracting pixel values ​​of the predictive block provided by motion compensation unit 44 from pixel values ​​of the current video block to be coded. The pixel difference values ​​forming the residual video block may include luma or chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video block of the video frame for use by video decoder 30 in decoding the video block of the video frame. The syntax elements may include, for example, syntax elements defining the motion vector used for the predictive block, any flags indicating the prediction mode, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0043] In some implementations, the intra BC unit 48 may generate vectors and fetch predictive blocks in a manner similar to that described above in connection with the motion estimation unit 42 and the motion compensation unit 44, except that the predictive block is in the same frame as the current block to be coded, and the vectors are called block vectors, as opposed to motion vectors. In particular, the intra BC unit 48 may be used to determine an intra prediction mode to encode the current block. In some examples, the intra BC unit 48 may encode the current block using various intra prediction modes, e.g., during separate encoding passes, and test their performance through rate-distortion analysis. The intra BC unit 48 may then select an appropriate intra prediction mode to use from among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values ​​using a rate-distortion analysis of the various tested intra prediction modes and select the intra prediction mode with the best rate-distortion characteristics among the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between an encoded block and the original unencoded block that was encoded to generate the encoded block, and the bitrate (i.e., number of bits) to use to generate the encoded block. Intra BC unit 48 can calculate ratios from the distortion and rate for various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0044] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction according to the implementations described herein. In either case, for intra block copying, the predictive block is the block that is identified as an exact match to the block being coded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metric, and identification of the predictive block may include calculation of values ​​at sub-integer pixel positions.

[0045] Regardless of whether the predictive block is from the same frame via intra prediction or from a different frame via inter prediction, video encoder 20 may form a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel value differences. The pixel difference values ​​that form the residual video block may include both luma and chroma component differences.

[0046] Intra-prediction processing unit 46 may intra-predict the current video block as described above, as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, or intra-block copy prediction performed by intra BC unit 48. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. To do so, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, a mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode in the bitstream.

[0047] After prediction processing unit 41 determines a predictive block for a current video block through either inter- or intra-prediction, summer 50 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and provided to transform processing unit 52. Transform processing unit 52 converts the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0048] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

[0049] Following quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method or technique. The encoded bitstream may then be transmitted to video decoder 30 or archived to storage device 32 for later transmission to or retrieval from video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements of the current video frame being encoded.

[0050] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct residual video blocks in the pixel domain to generate reference blocks for predicting other video blocks. As described above, motion compensation unit 44 may generate motion-compensated prediction blocks from one or more reference blocks of frames stored in DPB 64. Motion compensation unit 44 may apply one or more interpolation filters to the prediction blocks to calculate sub-integer pixel values ​​for use in motion estimation.

[0051] Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0052] 3 is a block diagram illustrating an example video decoder 30 according to some implementations of the present application. Video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, and a DPB 92. Prediction processing unit 81 further includes a motion compensation unit 82, an intra-prediction processing unit 84, and an intra-BC unit 85. Video decoder 30 may perform a decoding process that is generally reverse to the encoding process previously described for video encoder 20 in connection with FIG. 2. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, while intra-prediction unit 84 may generate prediction data based on an intra-prediction mode indicator received from entropy decoding unit 80.

[0053] In some examples, a unit of video decoder 30 may be tasked with performing implementations of the present disclosure. Also, in some examples, implementations of the present disclosure may be divided among one or more units of video decoder 30. For example, intra BC unit 85 may perform implementations of the present disclosure alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra prediction processing unit 84, and entropy decoding unit 80. In some examples, video decoder 30 may not include intra BC unit 85, and the functionality of intra BC unit 85 may be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0054] Video data memory 79 can store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source such as a camera via wired or wireless network communication of the video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a coded picture buffer (CPB) that stores encoded video data from an encoded video bitstream. A decoded picture buffer (DPB) 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in intra- or inter-prediction coding modes). Video data memory 79 and DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For purposes of illustration, video data memory 79 and DPB 92 are shown in Figure 3 as two separate components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30 or off-chip relative to those components.

[0055] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks and associated syntax elements of encoded video frames. Video decoder 30 may receive the syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors and other syntax elements to prediction processing unit 81.

[0056] If a video frame is coded as an intra-predictively coded (I) frame or coded for intra-coded predictive blocks in other types of frames, intra-prediction processing unit 84 of prediction processing unit 81 can generate predictive data for video blocks of the current video frame based on the signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0057] If a video frame is coded as an inter-predictive (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 generates one or more predictive blocks for video blocks of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each predictive block may be generated from a reference frame in one of the reference frame lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference frames stored in DPB 92.

[0058] In some examples, when a video block is coded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block of the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The prediction block may be within the same reconstructed region of the image as the current video block as defined by video encoder 20.

[0059] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for video blocks of a current video frame by analyzing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra- or inter-prediction) to use to encode the video blocks of the video frame, the inter-prediction frame type (e.g., B or P), construction information for one or more reference frame lists for the frame, the motion vectors of each inter-predictively encoded video block of the frame, the inter-prediction status of each inter-predictively encoded video block of the frame, and other information to decode the video blocks in the current video frame.

[0060] Similarly, intra BC unit 85 uses some of the received syntax elements, such as flags, to determine that the current video block was predicted using intra BC mode, construction information about which video blocks of the frame are in the reconstruction domain and stored in DPB 92, block vectors for each intra BC predicted video block of the frame, the intra BC prediction status for each intra BC predicted video block of the frame, and other information to decode the video blocks in the current video frame.

[0061] Motion compensation unit 82 may also perform interpolation using interpolation filters used by video encoder 20 during encoding of the video block to calculate interpolated values ​​for sub-integer pixels of the reference block. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from received syntax elements and use the interpolation filters to generate the predictive block.

[0062] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameter calculated for each video block in a video frame by video encoder 20 to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct residual blocks in the pixel domain.

[0063] After motion compensation unit 82 or intra BC unit 85 generates a prediction block for the current video block based on the vectors and other syntax elements, summer 90 reconstructs a decoded video block for the current video block by summing the residual block from inverse transform processing unit 88 with the corresponding prediction block generated by motion compensation unit 82 and intra BC unit 85. An in-loop filter (not shown) may be located between summer 90 and DPB 92 to further process the decoded video block. The decoded video block for a given frame is then stored in DPB 92, which stores reference frames used for subsequent motion compensation of the next video block. DPB 92, or a memory device separate from DPB 92, may also store decoded video for later presentation on a display device, such as display device 34 of FIG. 1 .

[0064] In a typical video coding process, a video sequence typically includes an ordered set of frames or images. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other examples, a frame may be monochrome, and thus include only one two-dimensional array of luma samples.

[0065] As shown in FIG. 4A, video encoder 20 (or more specifically, division unit 45) generates an encoded representation of a frame by first dividing the frame into a set of coding tree units (CTUs). A video frame may contain an integer number of CTUs arranged consecutively in raster scan order from left to right and top to bottom. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by video encoder 20 in the sequence parameter set so that all CTUs in a video sequence have the same size, which may be one of 128x128, 64x64, 32x32, and 16x16. However, it should be noted that the present application is not necessarily limited to a particular size. As shown in FIG. 4B, each CTU may include one coding tree block (CTB) of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to encode the samples in the coding tree block. The syntax elements describe the properties of different types of units of coding blocks of pixels and how a video sequence can be reconstructed by video decoder 30, including inter or intra prediction, intra prediction mode, motion vectors, and other parameters. For a monochromatic image or an image with three distinct color planes, a CTU may include a single coding tree block and syntax elements used to encode samples of the coding tree block. A coding tree block may be an NxN block of samples.

[0066] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quad tree partitioning, or a combination of both, on the coding tree blocks of a CTU to divide the CTU into smaller coding units (CUs). As shown in FIG. 4C , 64x64 CTU 400 is first partitioned into four smaller CUs, each with a block size of 32x32. Among the four smaller CUs, CU 410 and CU 420 are each partitioned into four CUs with a block size of 16x16. Two 16x16 CUs 430 and 440 are further partitioned into four CUs with a block size of 8x8. FIG. 4D shows a quad tree data structure illustrating the final result of the partitioning process of CTU 400 as shown in FIG. 4C , where each leaf node of the quad tree corresponds to one CU of a corresponding size ranging from 32x32 to 8x8. Similar to the CTU shown in FIG. 4B, each CU may include a coding block (CB) of luma samples, a corresponding coding block of chroma samples for two equal-sized frames, and syntax elements used to encode the samples of the coding block. For monochromatic images or images with three separate color planes, a CU may include a single coding block and syntax elements used to encode the samples of the coding block. Note that the quadtree partitioning shown in FIGS. 4C and 4D is for illustrative purposes only; a CTU can be partitioned into multiple CUs to accommodate various local characteristics based on quadtree / ternary / binary tree partitioning. In a multi-type tree structure, a CTU is partitioned by a quadtree structure, and each quadtree leaf CU may be further partitioned by binary and ternary tree structures. As shown in FIG. 4E, there are five partition types: quad, horizontal bisection, vertical bisection, horizontal trisection, and vertical trisection.

[0067] In some implementations, video encoder 20 may further divide the coding block of a CU into one or more MxN prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples to which the same prediction (inter or intra) is applied. A prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to predict the prediction block. In a monochromatic image or an image with three separate color planes, a PU may include a single prediction block and syntax structures used to predict the prediction block. Video encoder 20 may generate predictive luma, Cb, and Cr blocks for the luma, Cb, and Cr prediction blocks of each PU of the CU.

[0068] Video encoder 20 may generate the predictive blocks of a PU using intra prediction or inter prediction. If video encoder 20 generates the predictive blocks of a PU using intra prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of a frame associated with the PU. If video encoder 20 generates the predictive blocks of a PU using inter prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0069] After video encoder 20 generates predictive luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 may generate a luma residual block for the CU by subtracting the predictive luma block from the original luma coding block of the CU, whereby each sample of the luma residual block of the CU indicates a difference between a luma sample in one of the predictive luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, whereby each sample of the Cb residual block of the CU indicates a difference between a Cb sample in one of the predictive Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample of the Cr residual block of the CU may indicate a difference between a Cr sample in one of the predictive Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0070] Further, as shown in FIG. 4C , video encoder 20 can use quadtree partitioning to decompose the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochromatic image or an image with three separate color planes, a TU may include a single transform block and syntax structures used to transform the transform block samples.

[0071] Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block of the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to a Cb transform block of the TU to generate a Cb coefficient block of the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of the TU to generate a Cr coefficient block of the TU.

[0072] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients and provide further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements that indicate the quantized transform coefficients. For example, video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements that indicate the quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a sequence of bits that form a representation of the encoded frame and associated data, which may be stored on storage device 32 or transmitted to destination device 14.

[0073] After receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct frames of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data generally reverses the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on coefficient blocks associated with TUs of the current CU to reconstruct residual blocks associated with the TUs of the current CU. Video decoder 30 further reconstructs coded blocks of the current CU by adding samples of predictive blocks of PUs of the current CU to corresponding samples of transform blocks of TUs of the current CU. After reconstructing the coded blocks of each CU of the frame, video decoder 30 may reconstruct the frame.

[0074] As described above, video coding achieves video compression using two main modes: intra-frame prediction (or intra-prediction) and inter-frame prediction (or inter-prediction). Palette-based coding is another coding scheme adopted by many video coding standards. In palette-based coding, which is particularly suitable for screen generation content coding, a video encoder (e.g., video encoder 20 or video decoder 30) forms a palette table of colors to represent a given block of video data. The palette table contains the most dominant (e.g., frequently used) pixel values ​​in the given block. Pixel values ​​that are not frequently represented in the video data of the given block are not included in the palette table or are included in the palette table as escape colors.

[0075] Each entry in the palette table contains an index of the corresponding pixel value in the palette table. The palette index for a sample within a block may be coded to indicate which entry from the palette table is used to predict or reconstruct which sample. This palette mode begins with the process of generating a palette predictor for the first block of a picture, slice, tile, or other such grouping of video blocks. As described below, palette predictors for subsequent video blocks are typically generated by updating the previously used palette predictor. For purposes of explanation, it is assumed that the palette predictor is defined at the picture level. In other words, an image may contain multiple coding blocks, each with its own palette table, but there is one palette predictor for the entire image.

[0076] To reduce the bits required to signal palette entries in a video bitstream, a video decoder can utilize a palette predictor to determine new palette entries in a palette table to be used to reconstruct a video block. For example, the palette predictor may include palette entries from a previously used palette table, or may be initialized with a recently used palette table by including all entries from the recently used palette table. In some implementations, the palette predictor may include fewer than all entries from the recently used palette table and then incorporate some entries from other previously used palette tables. The palette predictor may have the same size as the palette table used to encode a different block, or may be larger or smaller than the palette table used to encode a different block. In one example, the palette predictor is implemented as a first-in-first-out (FIFO) table containing 64 palette entries.

[0077] To generate a palette table for a block of video data from the palette predictor, the video decoder may receive a one-bit flag for each entry of the palette predictor from the encoded video bitstream. The one-bit flag may have a first value (e.g., a binary 1) that indicates that the associated entry of the palette predictor is included in the palette table, or a second value (e.g., a binary 0) that indicates that the associated entry of the palette predictor is not included in the palette table. If the size of the palette predictor is larger than the palette table used for the block of video data, the video decoder may stop receiving further flags when the maximum size of the palette table is reached.

[0078] In some implementations, some entries in the palette table may be directly signaled in the encoded video bitstream rather than being determined using a palette predictor. For such entries, a video decoder may receive three separate m-bit values ​​from the encoded video bitstream that indicate the pixel values ​​of the luma and two chroma components associated with the entry, where m represents the bit depth of the video data. Compared to the multiple m-bit values ​​required for directly signaled palette entries, these palette entries derived from the palette predictor require only a one-bit flag. Thus, signaling some or all palette entries using a palette predictor can significantly reduce the number of bits required to signal new palette table entries, thereby improving the overall coding efficiency of palette mode coding.

[0079] In many examples, the palette predictor for a block is determined based on the palette table used to encode one or more previously encoded blocks. However, when encoding the first coding tree unit of a picture, slice, or tile, the palette table of the previously encoded block is unavailable. Therefore, the palette predictor cannot be generated using entries from a previously used palette table. In such cases, the sequence of palette predictor initializers may be signaled in a sequence parameter set (SPS) and / or a picture parameter set (PPS), which are values ​​used to generate the palette predictor when a previously used palette table is unavailable. An SPS generally refers to a syntax structure of syntax elements that apply to a series of subsequent coded video pictures, called a coded video sequence (CVS), as determined by the content of syntax elements found in the PPS pointed to by syntax elements found in each slice segment header. A PPS generally refers to a syntax structure of syntax elements that apply to one or more individual pictures in a CVS, as determined by the content of syntax elements found in each slice segment header. Therefore, an SPS is generally considered to be a higher level syntax structure than a PPS, meaning that the syntax elements contained in an SPS generally change less frequently and apply to the majority of video data compared to the syntax elements contained in a PPS.

[0080] 5A-5B are block diagrams illustrating examples of applying adaptive color space conversion (ACT) techniques to convert residuals between RGB and YCgCo color spaces, according to some implementations of this disclosure.

[0081] In the HEVC screen content coding extension, by applying ACT to adaptively convert the residual from one color space (e.g., RGB) to another color space (e.g., YCgCo), the association (e.g., redundancy) between three color components (e.g., R, G, and B) is significantly reduced in the YCgCo color space. Furthermore, in the existing ACT design, adaptation of different color spaces is performed at the transform unit (TU) level by signaling one flag per TU, tu_act_enabled_flag. When the flag tu_act_enabled_flag is equal to 1, it indicates that the residual of the current TU is coded in the YCgCo space; otherwise (i.e., when the flag is equal to 0), it indicates that the residual of the current TU is coded in the original color space (i.e., no color space conversion). Furthermore, different color space conversion formulas are applied depending on whether the current TU is coded in a lossless mode or a lossy mode. Specifically, the forward and inverse color space conversion formulas between the RGB color space and the YCgCo color space in the lossy mode are defined in FIG. 5A.

[0082] For lossless mode, a lossless version of the RGB-to-YCgCo transform (also known as YCgCo-LS) is used, which is performed based on the lifting operation shown in Figure 5B and the associated description.

[0083] As shown in Figure 5A, the forward and inverse color transformation matrices used in lossy mode are not normalized. Therefore, the width of the YCgCo signal is smaller than the width of the original signal after the color transformation is applied. To compensate for the width reduction caused by the forward color transformation, an adjusted quantization parameter is applied to the residual in the YCgCo domain. Specifically, once the color space transformation is applied, the QP value QP used to quantize the YCgCo domain residual is Y , Q.P. Cg , and QP Co are set to QP-5, QP-5, and QP-3, respectively, where QP is the quantization parameter used in the original color space.

[0084] FIG. 6 is a block diagram of an exemplary video data decoding process that applies the technique of Luminance Mapping with Chroma Scaling (LMCS) in accordance with some implementations of this disclosure.

[0085] In VVC, LMCS is used as a new coding tool applied before in-loop filters (e.g., deblocking filters, SAO, and ALF). Generally, LMCS has two main modules: 1) in-loop mapping of luma components based on an adaptive piecewise linear model; and 2) luma-dependent chroma residual scaling. Figure 6 shows a modified decoding process in which LMCS is applied. In Figure 6, the decoding modules performed in the mapped domain include an entropy decoding module, an inverse quantization module, an inverse transform module, a luma intra prediction module, and a luma sample reconstruction module (i.e., summation of luma prediction samples and luma residual samples). The decoding modules performed in the original (i.e., unmapped) domain include a motion compensation prediction module, a chroma intra prediction module, a chroma sample reconstruction module (i.e., summation of luma prediction samples and luma residual samples), and all in-loop filter modules, such as a deblocking module, SAO module, and ALF module. The new operation modules introduced by LMCS include a forward mapping of luma samples module 610, an inverse mapping of luma samples module 620, and a chroma residual scaling module 630.

[0086] The in-loop mapping of LMCS can adjust the dynamic range of the input signal to improve coding efficiency. The in-loop mapping of luma samples in existing LMCS designs is built on two mapping functions: a forward mapping function FwdMap and a corresponding inverse mapping function InvMap. The forward mapping function is signaled from the encoder to the decoder using a piecewise linear model with 16 equally sized pieces. The inverse mapping function does not need to be signaled because it is directly derived from the forward mapping function.

[0087] The parameters of the luminance mapping model are signaled at the slice level. A presence flag is first signaled to indicate whether an luminance mapping model is signaled for the current slice. If an luminance mapping model is present for the current slice, the corresponding piecewise linear model parameters are further signaled. Furthermore, at the slice level, another LMCS control flag is signaled to enable / disable LMCS for the slice.

[0088] The chroma residual scaling module 630 is designed to compensate for the interaction of quantization precision between the luma signal and its corresponding chroma signal when in-loop mapping is applied to the luma signal. It also signals in the slice header whether chroma residual scaling is enabled or disabled for the current slice. If luma mapping is enabled, an additional flag is signaled to indicate whether luma-dependent chroma residual scaling is applied. If luma mapping is not used, luma-dependent chroma residual scaling is always disabled, and no additional flag is required. Furthermore, chroma residual scaling is always disabled for CUs containing four or fewer chroma samples.

[0089] FIG. 7 is a block diagram illustrating an example video decoding process in which a video decoder performs an inverse adaptive color space conversion (ACT) technique in accordance with some implementations of this disclosure.

[0090] Similar to the ACT design of the HEVC SCC, the ACT of VVC converts the intra / inter prediction residual of one CU in 4:4:4 chroma format from its original color space (e.g., RGB color space) to the YCgCo color space. As a result, redundancy among the three color components can be reduced, resulting in better coding efficiency. FIG. 7 shows a decoding flowchart illustrating how inverse ACT is applied to the VVC framework through the addition of an inverse ACT module 710. When processing an encoded CU with ACT enabled, entropy decoding, inverse quantization, and an inverse DCT / DST-based transform are first applied to the CU. Then, as shown in FIG. 7, the inverse ACT is invoked to convert the decoded residual from the YCgCo color space back to its original color space (e.g., RGB and YCbCr). Furthermore, because the ACT in lossy mode is not normalized, a QP adjustment of (-5, -5, -3) is applied to the Y, Cg, and Co components to compensate for the altered width of the transformed residual.

[0091] In some embodiments, the ACT method reuses the same ACT core transform of HEVC to perform color conversion between different color spaces. Specifically, two different versions of color transform are applied depending on whether the current CU is encoded in a lossy or lossless manner. In the lossy case, the forward and inverse color transforms use the lossy YCgCo transform matrix as shown in FIG. 5A. In the lossless case, the lossless color transform YCgCo-LS is applied as shown in FIG. 5B. Furthermore, unlike existing ACT designs, the following modifications are introduced into the proposed ACT scheme to handle interactions with other coding tools in the VVC standard:

[0092] For example, since the residual of one CU in HEVC may be split into multiple TUs, an ACT control flag is signaled separately for each TU to indicate whether color space conversion needs to be applied. However, as previously described in connection with FIG. 4E, a nested quadtree with bipartition and tripartition structures is applied to VVC to replace the multi-split type concept, eliminating the separate CU, PU, ​​and TU divisions in HEVC. This means that in most cases, a single CU leaf node is used as the unit of prediction and transform processing without further division, unless the maximum supported transform size is smaller than the width or height of one component of the CU. Based on such a division structure, this disclosure proposes adaptively enabling and disabling ACT at the CU level. Specifically, one flag, cu_act_enabled_flag, is signaled for each CU to select between the original color space and the YCgCo color space for encoding the residual of the CU. When the flag is equal to 1, it indicates that the residuals of all TUs in the CU are encoded in the YCgCo color space. Otherwise, if the flag cu_act_enabled_flag is equal to 0, all residuals of the CU are coded in the original color space.

[0093] In some embodiments, a syntax element, e.g., sps_act_enabled_flag, is added to the sequence parameter set (SPS) to indicate whether ACT is enabled at the sequence level. Furthermore, because color space conversion is applied to video content whose luma and chroma components have the same resolution (e.g., 4:4:4 chroma format 4:4:4), ACT can be enabled only for 4:4:4 chroma format by adding one bitstream conformance requirement. Table 1 shows the modified SPS syntax table with the above syntax added. [Table 1]

[0094] Specifically, sps_act_enabled_flag equal to 1 indicates that ACT is enabled, and sps_act_enabled_flag equal to 0 indicates that ACT is disabled, thereby pointing to SPS, and the flag cu_act_enabled_flag is not signaled for the CU, which is inferred to be 0. It is a bitstream conformance requirement that the value of sps_act_enabled_flag must be equal to 0 if ChromaArrayType is not equal to 3.

[0095] In another embodiment, instead of always signaling sps_act_enabled_flag, the signaling of the flag is conditioned on the chroma type of the input signal. Specifically, if ACT is applicable only when the luma and chroma components are of the same resolution, the flag sps_act_enabled_flag is signaled only if the input video is captured in 4:4:4 chroma format. With this change, the modified SPS syntax table is as follows: [Table 2]

[0096] In some embodiments, the syntax design specifications for decoding video data using ACT are shown in the table below. [Table 3] TIFF2025166093000005.tif247170TIFF2025166093000006.tif255168TIFF2025166093000007.tif20416 8TIFF2025166093000008.tif211170TIFF2025166093000009.tif255164TIFF2025166093000010.tif14164

[0097] The flag cu_act_enabled_flag equal to 1 indicates that the residual of the coding unit is coded in the YCgCo color space, and the flag cu_act_enabled_flag equal to 0 indicates that the residual of the coding unit is coded in the original color space (e.g., RGB or YCbCr). If the flag cu_act_enabled_flag is not present, it is inferred to be equal to 0.

[0098] 8A and 8B are block diagrams illustrating an example video decoding process in which a video decoder performs a luma mapping technique with inverse adaptive color space transformation (ACT) and chroma scaling, according to some implementations of this disclosure. In some embodiments, the video bitstream is coded using both ACT (e.g., inverse ACT 710 of FIG. 7) and chroma residual scaling (e.g., chroma residual scaling 630 of FIG. 6). In some other embodiments, the video bitstream is coded using chroma residual scaling but not both ACT, thereby not requiring inverse ACT 710.

[0099] 8A shows an embodiment in which the video encoder performs chroma residual scaling 630 before inverse ACT 710. As a result, the video encoder performs luma mapping using chroma residual scaling 630 in the color-space-transformed domain. For example, assuming the input video is captured in RGB format and converted to YCgCo color space, the video encoder performs chroma residual scaling 630 on the chroma residuals Cg and Co according to the luma residual Y in the YCgCo color space.

[0100] 8B shows an alternative embodiment in which the video encoder performs chroma residual scaling 630 after the inverse ACT 710. As a result, the video encoder performs luma mapping using chroma residual scaling 630 in the original color space domain. For example, assuming the input video is captured in RGB format, the video encoder applies chroma residual scaling to the B and R components.

[0101] FIG. 9 is a block diagram illustrating example decoding logic between adaptive color space conversion (ACT) implementations and block differential pulse coded modulation (BDPCM) according to some implementations of this disclosure.

[0102] BDPCM is a coding tool for screen content coding. In some embodiments, the BDPCM enable flag is signaled at the sequence level in the SPS. The BDPCM enable flag is signaled only if transform skip mode is enabled in the SPS.

[0103] When BDPCM is enabled, if the CU size is less than or equal to MaxTsSize×MaxTsSize for luma samples and the CU is intra-coded, a flag is sent at the CU level, where MaxTsSize is the maximum block size for which transform skip mode is allowed. This flag indicates whether regular intra-coding or BDPCM is used. If BDPCM is used, another BDPCM prediction direction flag is also sent to indicate whether the prediction is horizontal or vertical. The block is then predicted using regular horizontal or vertical intra-prediction processes using unfiltered reference samples. The residuals are quantized, and the difference between each quantized residual and its predictor, i.e., the residual previously coded for its horizontal or vertical (depending on the BDPCM prediction direction) neighbor, is coded.

[0104] TIFF2025166093000011.tif54168

[0105] TIFF2025166093000012.tif27168

[0106] TIFF2025166093000013.tif32168

[0107] Inverse quantization residual Q -1 (Q(r i,j )) is added to the intra-block prediction to generate the reconstructed sample value.

[0108] TIFF2025166093000014.tif58168

[0109] In some embodiments, the video encoder implements different logic to better handle the interaction between ACT and BDPCM. For example, when ACT is applied to one intra CU, BDPCM is enabled for the luma component but disabled for the chroma component (910). A corresponding modified syntax table for the coding unit is shown below: [Table 4] TIFF2025166093000016.tif182163

[0110] In some embodiments, when ACT is applied to one intra CU, BDPCM is enabled for both luma and chroma components (920). The corresponding modified syntax table of the coding unit is shown below: [Table 5] TIFF2025166093000018.tif182165

[0111] In some embodiments, when ACT is applied to one intra CU, BDPCM is disabled for both luma and chroma components (930). In such cases, there is no need to signal BDPCM-related syntax elements. The corresponding modified syntax table for the coding unit is shown below: [Table 6] TIFF2025166093000020.tif198160

[0112] FIG. 10 is a flowchart 1000 illustrating an example process by which a video encoder decodes video data by conditionally performing an inverse adaptive color space conversion (ACT) technique based on interactions between ACT and other encoding tools (e.g., LMCS and BDPCM) in accordance with some implementations of the present disclosure.

[0113] As a first step, the video decoder receives from the bitstream a first syntax element in the slice header of a slice that indicates whether Luminance Mapping with Chroma Scaling (LMCS) is applied to the coding units in the slice (1010).

[0114] Next, the video decoder receives a second syntax element (e.g., cu_act_enabled_flag) for a coding unit in the slice from the bitstream (1020). The second syntax element indicates whether the coding unit is coded using adaptive color space transformation (ACT).

[0115] If the second syntax element has a non-zero value (e.g., indicates that the coding unit was coded using ACT), the video decoder decodes the video data by applying inverse ACT to transform the luma and chroma residuals of the coding unit from the transformed color space of the video data corresponding to the coding unit to the original color space (1030).

[0116] After applying the inverse ACT to transform the luma and chroma residuals of the coding unit, if the first syntax element has a non-zero value (e.g., indicating that the coding unit is coded with LMCS), the video decoder decodes the video data corresponding to the coding unit by performing inverse luma mapping to luma samples and inverse scaling to the chroma residuals of the coding unit (1040).

[0117] In some embodiments, before receiving the second syntax element, the video decoder receives a third syntax element (e.g., chroma_format_idc) from the bitstream, where the third syntax element indicates whether the video data has a predefined chroma format. As noted above, ACT is only valid for certain predefined chroma formats (e.g., 4:4:4 chroma format).

[0118] In some embodiments, a second syntax element (e.g., cu_act_enabled_flag) is present in the bitstream only if the video data has a predefined chroma format (e.g., ACT is disabled if the video data is not in 4:4:4 chroma format).

[0119] In some embodiments, the transformed color space is the YCgCo color space, while the original color space may be the RGB color space or the YCbCr color space.

[0120] In some embodiments, after performing inverse ACT to transform the luma and chroma residuals of the coding unit, the video decoder performs inverse luma mapping to luma samples of the coding unit and inverse scaling to chroma residuals by: generating intermediate luma samples in the mapped domain of the coding unit by adding the luma residuals of the original color space to the predicted luma samples; transforming the intermediate luma samples of the mapped domain into reconstructed luma components in the pre-mapped domain using one or more adaptive piecewise linear models; inverse scaling the chroma residuals according to the corresponding intermediate luma samples in the mapped domain; and reconstructing the chroma components of the coding unit using the scaled chroma residuals and the predicted chroma samples.

[0121] In some embodiments, a video decoder receives, from a bitstream, video data corresponding to a coding unit, where the coding unit is coded using an intra-prediction mode. The video encoder receives a first syntax element (e.g., cu_act_enabled_flag) from the video data, where the first syntax element indicates whether the coding unit is coded using adaptive color space transformation (ACT). If the first syntax element has a zero value, the video encoder receives one or more syntax elements (e.g., intra_bdpcm_chroma_flag, intra_bdpcm_chroma_dir_flag) from the video data, where the one or more syntax elements indicate whether chroma components of the coding unit are coded using block differential pulse coding modulation (BDPCM). If the first syntax element has a non-zero value, the video encoder assigns default values ​​to one or more syntax elements associated with BDPCM. The default values ​​indicate that the chroma components of the coding unit are decoded using inverse BDPCM or otherwise. The video encoder then decodes the coding unit from the video data in accordance with the first syntax element associated with the ACT and one or more syntax elements associated with the BDPCM.

[0122] In some embodiments, before receiving the first syntax element, the video decoder receives a second syntax element from the video data, where the second syntax element indicates whether the video data has a predefined chroma format. As noted above, ACT is only valid for certain predefined chroma formats (e.g., 4:4:4 chroma format).

[0123] In some embodiments, the first syntax element is present in the bitstream only if the video data has a predefined chroma format.

[0124] In some embodiments, when a coding unit is encoded without a color space transformation being applied, the coding unit is decoded using inverse BDPCM.

[0125] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, including any medium that can facilitate transmission of a computer program from one place to another, for example, based on a communications protocol. In this manner, computer-readable media generally may correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures to implement the embodiments described herein. A computer program product may include computer-readable media.

[0126] The terms used in the description of the embodiments herein are intended to describe particular embodiments only and not to limit the scope of the claims. Unless the context clearly dictates otherwise, the singular forms "a kind," "one," and "the" used in the description of the embodiments and the appended claims are intended to include the plural forms as well. It should be further understood that the term "and / or," as used herein, refers to any and all possible combinations of one or more associated listed items, and includes any and all possible combinations of one or more associated listed items. It should be further understood that the terms "comprising" and / or "including," as used herein, specify the presence of said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0127] It is further understood that, although the terms "first," "second," etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one type of element from another type of element. For example, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode, without departing from the scope of the embodiments. A first electrode and a second electrode are both electrodes, but are not the same electrode.

[0128] The descriptions herein are for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the form disclosed. Numerous modifications, variations, and alternative embodiments will be apparent to those skilled in the art upon learning from the above description and the associated drawings. The selection and description of the examples is intended to best interpret the principles and practical applications of the invention and to enable those skilled in the art to understand various embodiments of the invention and to best utilize these basic principles and various modifications, for example, to adapt them to specific applications as anticipated. It is therefore to be understood that the scope of the claims is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims.

Claims

1. 1. A method for decoding video data, comprising: receiving video data corresponding to a coding unit from a bitstream, the coding unit being coded using an intra prediction mode; receiving a first syntax element from the bitstream, the first syntax element indicating whether the coding unit is coded using adaptive color space transformation (ACT); the first syntax element having a zero value indicates that the coding unit is not coded using ACT; receiving one or more syntax elements from the bitstream in accordance with determining that the first syntax element has a zero value, the one or more syntax elements indicating whether a chroma component of a coding unit is coded using block differential pulse coding modulation (BDPCM); a non-zero value of the first syntax element indicates that the coding unit is coded using ACT; assigning default values ​​to the one or more syntax elements associated with the BDPCM in accordance with determining that the first syntax element has a non-zero value; decoding a chroma component of the coding unit from the video data in accordance with a first syntax element associated with ACT and the one or more syntax elements associated with BDPCM; A method comprising:

2. receiving a second syntax element from the bitstream before receiving the first syntax element, the second syntax element indicating whether the video data has a predefined chroma format; The method of claim 1 further comprising:

3. The predefined chroma format is the 4:4:4 chroma format; The method of claim 2.

4. The first syntax element is present in the bitstream only if the video data has a predefined chroma format. The method of claim 2.

5. The default value indicates that the chroma component of the coding unit is decoded without using inverse BDPCM. The method of claim 1.

6. The default value indicates that inverse BDPCM is used to decode the chroma component of the coding unit. The method of claim 1.

7. If no transform is applied and the coding unit is coded, the coding unit is decoded using inverse BDPCM; The method of claim 1.

8. 1. An electronic device comprising: one or more processing units; a memory coupled to the one or more processing units; a plurality of programs stored in the memory; Including, The plurality of programs, when executed by the one or more processing units, cause the electronic device to perform the method of any one of claims 1 to 7. electronic equipment.

9. 1. A non-transitory computer-readable storage medium, comprising: Storing a plurality of programs for execution by an electronic device having one or more processing units, said plurality of programs, when executed by said one or more processing units, causing the electronic device to perform the method of any one of claims 1 to 7. A non-transitory computer-readable storage medium.

10. 1. A method of encoding video data, comprising: determining whether the coding unit is coded using an intra prediction mode using adaptive color space conversion (ACT); determining whether the chroma components of the coding unit are coded using block differential pulse coding modulation (BDPCM) in accordance with determining that the chroma components of the coding unit are not coded using ACT; and decoding the chroma components of the coding unit based on the determination whether the chroma components of the coding unit are coded using BDPCM; Decoding the coding unit by applying an inverse ACT in accordance with determining that the coding unit has been coded using an ACT; A method comprising:

11. signaling a first syntax element, the first syntax element indicating whether the coding unit is coded using ACT; signaling one or more syntax elements in accordance with determining that the first syntax element has a zero value, the one or more syntax elements indicating whether a chroma component of the coding unit is coded using BDPCM; and not signaling one or more syntax elements in accordance with determining that the first syntax element has a non-zero value; and assigning default values ​​to one or more syntax elements if the one or more syntax elements are not signaled; The method of claim 10 further comprising:

12. signaling a second syntax element, the second syntax element indicating whether the video data has a predefined chroma format; signaling the first syntax element if the second syntax element indicates that the video data has a predefined chroma format; and The method of claim 11 further comprising:

13. the predefined chroma format is a 4:4:4 chroma format; The method of claim 12.

14. The first syntax element is signaled only if the video data has a predefined chroma format. The method of claim 12.

15. The default value indicates that the chroma component of the coding unit is decoded without using inverse BDPCM. The method of claim 11.

16. If the coding unit is coded without applying a transform, the coding unit is decoded using inverse BDPCM. The method of claim 10.

17. 1. An electronic device comprising: one or more processing units; a memory coupled to the one or more processing units; a plurality of programs stored in the memory; Including, The plurality of programs, when executed by the one or more processing units, cause the electronic device to perform the method of any one of claims 10 to 16. electronic equipment.

18. 1. A non-transitory computer-readable storage medium, comprising: Storing a plurality of programs for execution by an electronic device having one or more processing units, said plurality of programs, when executed by said one or more processing units, causing the electronic device to perform the method of any one of claims 10 to 16. A non-transitory computer-readable storage medium.

19. 1. A method for storing a bitstream, comprising: generating an encoded bitstream according to the method of any one of claims 10 to 16; storing the generated bitstream; A method comprising:

20. 1. A method for transmitting a bitstream, comprising: generating an encoded bitstream according to the method of any one of claims 10 to 16; transmitting the generated bitstream; A method comprising:

Images