Cross-component determination of chroma and luma components of video data
Generate unavailable brightness samples through sample filling, and use the same downsampling filter to process all brightness samples in the video frame, solving the rate bottleneck problem caused by incompatibility of sampling filters in cross component prediction mode in the prior art, improving the The encoding and decoding efficiency of video data.
Patent Information
- Application Number
- CN202410967265.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-12-30
- Filing Date
- 2020-12-29
- Publication Date
- 2024-11-08
AI Technical Summary
In the existing video encoding and decoding technology, when the cross component prediction mode reduces the cross component redundancy between the brightness and chromaticity samples of the video bitstream, there is downsampling filter incompatibility, resulting in a bottleneck in processing rate and lack of efficient cross component. Prediction mechanism.
Generate brightness samples that are not available during the brightness downsampling process by sample filling, and use the same downsampling filter to generate all downsampled brightness samples across the entire video frame, realizing parallel video processing and enhancing the processing rate of the video data bitstream.
The processing rate of the video data bitstream is improved, the encoding and decoding efficiency of the video data is enhanced, and the bottleneck problem caused by the incompatibility of downsampling filters in the prior art is solved.
Smart Images

Figure CN118921491A_ABST
Abstract
Description
[0001] Related Applications
[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 955,348, filed on December 30, 2019, entitled “Simplifications of Cross-Component Linear Model,” the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present application relates generally to video data encoding and compression, and more particularly, to improved methods and systems for encoding and decoding chrominance and luminance samples of video frames in a video data bitstream. Background Art
[0004] Digital video is supported by a wide variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, and the like. These electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression standards defined by standards such as MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Codec (AVC), High Efficiency Video Codec (HEVC), and Versatile Video Codec (VVC). Video compression typically includes performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data. For block-based video codecs, a video frame is divided into one or more slices, each with multiple video blocks, which may also be referred to as coding tree units (CTUs). Each CTU may contain a coding unit (CU), or may be recursively split into smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf-CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU can be coded or decoded in intra mode, inter mode, or IBC mode. Video blocks in a slice of intra codec (I) of a video frame are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same video frame. Video blocks in a slice of inter codec (P or B) of a video frame can use spatial prediction with respect to reference samples in neighboring blocks in the same video frame, or use temporal prediction with respect to reference samples in other previous and / or future reference video frames.
[0005] A prediction block for the current video block to be encoded is generated based on spatial or temporal prediction of a previously encoded reference block (e.g., a neighboring block). The process of finding a reference block can be accomplished by a block matching algorithm. The residual data representing the pixel differences between the current block to be encoded and the prediction block is called a residual block or prediction error. Inter-coded blocks are encoded according to motion vectors and residual blocks, and motion vectors point to reference blocks in reference frames that form prediction blocks. The process of determining motion vectors is typically called motion estimation. Intra-coded blocks are encoded according to intra-frame prediction modes and residual blocks. For further compression, the residual block is transformed from the pixel domain to the transform domain (e.g., the frequency domain) to generate residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients and then entropy encoded into the video bitstream to achieve greater compression.
[0006] The encoded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by another electronic device with digital video capabilities, or is directly transmitted to the electronic device by wire or wirelessly. The electronic device then performs video decompression (which is the reverse process of video compression described above), such as by parsing the encoded video bitstream to obtain syntax elements from the bitstream, and reconstructing the digital video data into its original format based at least in part on the syntax elements obtained from the bitstream, and presenting the reconstructed digital video data on a display of the electronic device.
[0007] A cross component prediction mode is applied to reduce the cross component redundancy between the brightness and chrominance samples of a video bitstream. Specifically, in this prediction model, the brightness sample is downsampled and used to predict the chrominance samples. However, the brightness sample is not available at some positions of the video frame, and different downsampling filters have been used to generate downsampled brightness samples near these positions. The application of different downsampling filters is incompatible with parallel image processing and may cause a bottleneck to the rate of processing the video bitstream. It would be useful to establish a more efficient cross component prediction mechanism than current practice. Summary of the invention
[0008] The present application describes several implementations, which relate to video data encoding and decoding, and more specifically, the implementations relate to improved methods and systems for encoding and decoding of chrominance and luminance samples of video frames in a video data bitstream. Sample padding is applied to luminance samples to generate unavailable luminance samples used by a luminance downsampling process. These unavailable samples may be outside a video frame or image strip, or have not yet been encoded and decoded and will be subsequently encoded and decoded. These unavailable samples are optionally generated by repeated padding or mirror padding. In this way, the same downsampling filter can be used to generate all downsampled luminance samples across an entire video frame, which enables parallel video processing and enhances the corresponding video processing rate of a video data bitstream.
[0009] In one aspect of the present application, a method for decoding video data is implemented at an electronic device. The method includes: obtaining multiple luminance samples for multiple pixels in a video frame from a bitstream. The multiple pixels belong to a code block and include boundary pixels, and the boundary pixels are inside the code block and adjacent to the boundary of the code block. The method further includes: determining that one or more neighboring pixels of the boundary pixel are unavailable, assigning the luminance sample corresponding to the boundary pixel to the luminance sample corresponding to each of the one or more neighboring pixels, and determining the boundary luminance sample based on at least the luminance samples of the one or more neighboring pixels and the boundary pixel according to a predefined luminance interpolation scheme. Each of the one or more neighboring pixels is outside the code block. The method further includes: determining the boundary chrominance sample from the boundary luminance sample according to a linear mapping model. In some embodiments, one or more neighboring pixels of the boundary pixel are outside the video frame or image strip. Alternatively, in some embodiments, one or more neighboring pixels of the boundary pixel have not been coded and decoded and will be coded and decoded after the code block.
[0010] In another aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When the programs are executed by the one or more processing units, the electronic device performs the method for decoding video data as described above.
[0011] In another aspect, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. When the programs are executed by the one or more processing units, the electronic device executes the method for decoding video data as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings, which are included to provide a further understanding of the implementations and are incorporated in and constitute a part of this specification, illustrate the described implementations and together with the description serve to explain the underlying principles. Like reference numerals refer to corresponding parts.
[0013] Figure 1 A block diagram of an exemplary video encoding and decoding system is shown for illustration purposes according to some implementations of the present disclosure.
[0014] Figure 2 A block diagram of an exemplary video encoder is shown for illustrating some implementations according to the present disclosure.
[0015] Figure 3 A block diagram of an exemplary video decoder is shown for illustration purposes according to some implementations of the present disclosure.
[0016] FIG. 4A to FIG. 4E The block diagram is a diagram illustrating how to recursively partition a frame into a plurality of video blocks of different sizes according to some implementations of the present disclosure.
[0017] Figure 5 The diagram illustrates a process of deriving chroma samples from luma samples of a code block of a video frame of a bitstream according to some embodiments.
[0018] FIG. 6A to FIG. 6D 0046] are four example code blocks according to some embodiments, where each code block includes a plurality of luma samples to be converted into a plurality of chroma samples.
[0019] Figure 7 The present invention is a flowchart of a video data decoding method implemented in an electronic device according to some embodiments. DETAILED DESCRIPTION
[0020] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, many non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be implemented without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.
[0021] Figure 1 1 is a block diagram of an exemplary system 10 for encoding and decoding video blocks in parallel according to some implementations of the present disclosure. Figure 1As shown, system 10 includes a source device 12 that generates and encodes video data to be decoded at a later time by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.
[0022] In some implementations, the target device 14 may receive the encoded video data to be decoded via the link 16. The link 16 may include any type of communication medium or device capable of moving the encoded video data from the source device 12 to the target device 14. In one example, the link 16 may include a communication medium that enables the source device 12 to transmit the encoded video data directly to the target device 14 in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and transmitted to the target device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a portion of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device 12 to the target device 14.
[0023] In some other implementations, the encoded video data may be transferred from the output interface 22 to a storage device 32. Subsequently, the encoded video data in the storage device 32 may be accessed by the target device 14 via the input interface 28. The storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device 32 may correspond to a file server or another intermediate storage device that may hold the encoded video data generated by the source device 12. The target device 14 may access the stored video data from the storage device 32 via streaming or downloading. The file server may be any type of computer capable of storing the encoded video data and transmitting the encoded video data to the target device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. The target device 14 can access the encoded video data through any standard data connection, including a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL, a cable modem, etc.), or a combination of the two, which is suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the storage device 32 can be streaming transmission, download transmission, or a combination of the two.
[0024] like Figure 1 As shown, source device 12 includes video source 18, video encoder 20 and output interface 22. Video source 18 can include sources such as video capture devices, such as cameras, video archives containing previously captured videos, video feed interfaces for receiving videos from video content providers, and / or computer graphics systems for generating computer graphics data such as source video, or combinations of such sources. As an example, if video source 18 is a camera of a security monitoring system, source device 12 and target device 14 can form a camera or video phone. However, the implementation described in the present application can be generally applicable to video encoding and decoding, and can be applicable to wireless and / or wired applications.
[0025] The captured video, pre-captured video, or computer-generated video may be encoded by the video encoder 20. The encoded video data may be transmitted directly to the target device 14 via the output interface 22 of the source device 12. The encoded video data may also (or alternatively) be stored on a storage device 32 for later access by the target device 14 or other device for decoding and / or playback. The output interface 22 may further include a modem and / or a transmitter.
[0026] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included in the encoded video data, which is transmitted over a communication medium, stored on a storage medium, or stored on a file server.
[0027] In some implementations, the target device 14 may include a display device 34, which may be an integrated display device or an external display device configured to communicate with the target device 14. The display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.
[0028] The video encoder 20 and the video decoder 30 can operate according to proprietary or industry standards, such as VVC, HEVC, MPEG-4 Part 10, Advanced Video Codec (AVC), or extensions of such standards. It should be understood that the present application is not limited to specific video encoding / decoding standards, but can be applied to other video encoding / decoding standards. It is generally conceivable that the video encoder 20 of the source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also generally conceivable that the video decoder 30 of the target device 14 can be configured to decode video data according to any of these current or future standards.
[0029] Each of the video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable codec circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When implemented partially in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated into a corresponding device as part of a combined encoder / decoder (CODEC).
[0030] Figure 2 A block diagram of an exemplary video encoder 20 is illustrated in accordance with some implementations described herein. The video encoder 20 may perform intra-frame and inter-frame prediction encoding of video blocks within a video frame. Intra-frame prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence.
[0031] like Figure 2 As shown, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some implementations, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62 for video block reconstruction. A deblocking filter (not shown) can be placed between the summer 62 and the DPB 64 to filter the block boundaries so as to remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, a loop filter (not shown) can also be used to filter the output of the summer 62. The video encoder 20 can take the form of a fixed or programmable hardware unit, or can be divided into one or more hardware units in the fixed or programmable hardware unit shown in the figure.
[0032] The video data memory 40 may store video data to be encoded by components of the video encoder 20. The video data in the video data memory 40 may be obtained, for example, from the video source 18. The DPB 64 is a buffer that stores reference video data for use by the video encoder 20 in encoding the video data (e.g., in intra-frame or inter-frame prediction coding mode). The video data memory 40 and the DPB 64 may be formed by any of a variety of memory devices. In various different examples, the video data memory 40 may be on-chip with other components of the video encoder 20, or off-chip relative to those components.
[0033] like Figure 2As shown, after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles, or other larger coding units (CUs) according to a predefined splitting structure such as a quadtree structure associated with the video data. The video frame may be divided into a plurality of video blocks (or a set of video blocks referred to as tiles). The prediction processing unit 41 may select one of a plurality of possible prediction coding modes for the current video block based on error results (e.g., coding rate and distortion level), such as one of a plurality of intra-frame prediction coding modes or one of a plurality of inter-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame or inter-frame prediction coded block to the summer 50 to generate a residual block, and to the summer 62 to reconstruct the coded block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements such as motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information to the entropy coding unit 56.
[0034] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.
[0035] In some implementations, motion estimation unit 42 determines the inter-prediction mode for a current video frame by generating motion vectors according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a prediction unit (PU) of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors, which estimate the motion of video blocks. The motion vector may, for example, indicate the displacement of a PU of a video block within a current video frame or picture relative to a prediction block within a reference frame (or other coding unit), the prediction block being relative to a current block being encoded within the current frame (or other coding unit). The predetermined pattern may designate video frames in a sequence as P frames or B frames. Intra BC unit 48 may determine vectors, such as block vectors, for intra BC coding in a manner similar to how motion estimation unit 42 determines motion vectors for inter prediction, or may utilize motion estimation unit 42 to determine block vectors.
[0036] A prediction block is a block of a reference frame that is considered to closely match a PU of a video block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, motion estimation unit 42 may perform motion searches relative to full pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.
[0037] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-prediction coded frame by comparing the position of the PU to the position of a prediction block of a reference frame selected from a first reference frame list (list 0) or a second reference frame list (list 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.
[0038] The motion compensation performed by the motion compensation unit 44 may involve fetching or generating a prediction block based on the motion vector determined by the motion estimation unit 42. Upon receiving the motion vector for the PU of the current video block, the motion compensation unit 44 may locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from the DPB 64, and forward the prediction block to the summer 50. Summer 50 then forms a residual video block of pixel difference values by subtracting the pixel values of the prediction block provided by the motion compensation unit 44 from the pixel values of the current video block being encoded. The pixel difference values forming the residual video block may include a luma difference component or a chroma difference component or both. The motion compensation unit 44 may also generate syntax elements associated with the video block of the video frame for use by the video decoder 30 when decoding the video block of the video frame. These syntax elements may include, for example, syntax elements defining a motion vector used to identify the prediction block, any flag indicating a prediction mode, or any other syntax information described herein. Note that the motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.
[0039] In some implementations, the intra BC unit 48 may generate vectors and fetch prediction blocks in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44, but the prediction block is in the same frame as the current block being encoded, and the vector is referred to as a block vector rather than a motion vector. In particular, the intra BC unit 48 may determine the intra prediction mode used to encode the current block. In some examples, the intra BC unit 48 may encode the current block using various different intra prediction modes, for example, during multiple separate encoding passes, and test their performance through rate-distortion analysis. Next, the intra BC unit 48 may select a suitable intra prediction mode to use among the various different tested intra prediction modes, and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values for the various different tested intra prediction modes using rate-distortion analysis, and select the intra prediction mode with the best rate-distortion characteristics among the tested modes as the suitable intra prediction mode to use. The rate-distortion analysis typically determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to produce the coded block, as well as the bit rate (i.e., the number of bits) used to produce the coded block. Intra BC unit 48 may calculate ratios for the distortion and rates for various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.
[0040] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such models for intra BC prediction in accordance with implementations described herein. In either case, for intra block copying, a prediction block may be a block that is considered to closely match a block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics, and identification of the prediction block may include calculating values for sub-integer pixel positions.
[0041] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from different frames according to inter-frame prediction, video encoder 20 can form a residual video block by subtracting the pixel values of the prediction block from the pixel values of the current video block being encoded to form pixel difference values. The pixel difference values forming the residual video block can include both luma component differences and chroma component differences.
[0042] As described above, the intra-prediction processing unit 46 can perform intra-prediction on the current video block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44 or the intra-block copy prediction performed by the intra BC unit 48. In particular, the intra-prediction processing unit 46 can determine the intra-prediction mode used to encode the current block. To do so, the intra-prediction processing unit 46 can encode the current block using a variety of different intra-prediction modes, for example during multiple separate encoding passes, and the intra-prediction processing unit 46 (or in some examples, the mode selection unit) can select a suitable intra-prediction mode to use from the tested intra-prediction modes. The intra-prediction processing unit 46 can provide information indicating the selected intra-prediction mode for the block to the entropy encoding unit 56. The entropy encoding unit 56 can encode the information indicating the selected intra-prediction mode in the bitstream.
[0043] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, summer 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform.
[0044] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.
[0045] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream is then transmitted to video decoder 30 or archived in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy coding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.
[0046] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate a reference block for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values used in motion estimation.
[0047] Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.
[0048] Figure 3 A block diagram of an exemplary video decoder 30 is shown for illustrating some implementations of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction processing unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform operations generally associated with the above-mentioned video encoder 20. Figure 2 For example, the motion compensation unit 82 may generate prediction data based on the motion vector received from the entropy decoding unit 80 , and the intra-prediction unit 84 may generate prediction data based on the intra-prediction mode indicator received from the entropy decoding unit 80 .
[0049] In some examples, units of the video decoder 30 may be tasked to perform implementations of the present application. Likewise, in some examples, implementations of the present disclosure may be divided among one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform implementations of the present application alone or in combination with other units of the video decoder 30 (such as the motion compensation unit 82, the intra prediction processing unit 84, and the entropy decoding unit 80). In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.
[0050] The video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of the video decoder 30. The video data stored in the video data memory 79 may be obtained via a wired or wireless network communication of the video data or by accessing a physical data storage medium (e.g., a flash drive or hard disk), such as from a storage device 32, from a local video source such as a camera. The video data memory 79 may include a coded picture buffer (CPB) that stores encoded video data from an encoded video bitstream. A decoded picture buffer (DPB) 92 of the video decoder 30 stores reference video data for use by the video decoder 30 when decoding video data (e.g., in intra-frame or inter-frame prediction decoding modes). The video data memory 79 and the DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For purposes of illustration, the video data memory 79 and the DPB 92 are shown in FIG. Figure 3 92 are depicted as two different components of the video decoder 30. However, it will be clear to those skilled in the art that the video data memory 79 and the DPB 92 can be provided by the same memory device or a separate memory device. In some examples, the video data memory 79 can be on-chip with other components of the video decoder 30, or off-chip relative to those components.
[0051] During the decoding process, the video decoder 30 receives an encoded video bitstream, which represents video blocks of an encoded video frame and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or at the video block level. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-frame prediction mode indicators and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors and other syntax elements to the prediction processing unit 81.
[0052] When a video frame is encoded as an intra-frame prediction coded (I) frame or a prediction block for intra-frame coding in other types of frames, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-frame prediction mode transmitted by the signal and the reference data from the previously decoded block of the current frame.
[0053] When the video frame is encoded as an inter-frame prediction coded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of these prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists - List 0 and List 1 - using a default construction technique based on the reference frames stored in the DPB 92.
[0054] In some examples, when a video block is decoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates prediction blocks for the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. These prediction blocks may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.
[0055] The motion compensation unit 82 and / or the intra BC unit 85 determine prediction information for the video block of the current video frame by parsing the motion vector and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, the motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra or inter prediction) used to decode the video block of the video frame, the inter prediction frame type (e.g., B or P), the construction information of one or more reference lists in the reference frame list for the frame, the motion vector of each inter prediction decoded video block of the frame, the inter prediction state of each inter prediction decoded video block of the frame, and other information for decoding the video block in the current video frame.
[0056] Similarly, the intra BC unit 85 may use some of the received syntax elements (e.g., flags) to determine whether the current video block is predicted using the intra BC module, construction information of which video blocks of the frame are within the reconstructed region and should be stored in the DPB 92, block vectors for each intra BC predicted video block of the frame, intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.
[0057] Motion compensation unit 82 may also perform interpolation using interpolation filters such as those used by video encoder 20 during encoding of the video block to calculate interpolated values for sub-integer pixels of a reference block. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use those interpolation filters to produce a prediction block.
[0058] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to these transform coefficients to reconstruct the residual block in the pixel domain.
[0059] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the summer 90 reconstructs the decoded video block for the current video block by summing the residual block from the inverse transform processing unit 88 and the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter (not shown) can be placed between the summer 90 and the DPB 92 to further process the decoded video block. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 can also store the decoded video for later use in, for example, Figure 1 The display device 34 is displayed on a display device.
[0060] In a typical video encoding and decoding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luminance samples. SCb is a two-dimensional array of Cb chrominance samples. SCr is a two-dimensional array of Cr chrominance samples. In other cases, a frame may be monochrome and therefore only include a two-dimensional array of luminance samples.
[0061] like Figure 4A As shown, the video encoder 20 (or more specifically, the segmentation unit 45) generates an encoded representation of a frame by first segmenting the frame into a set of coding tree units (CTUs). A video frame may include an integer number of CTUs sequentially ordered in a raster scan order from left to right and from top to bottom. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, which is one of 128×128, 64×64, 32×32, and 16×16. Note, however, that the present application is not necessarily limited to a particular size. Figure 4BAs shown, each CTU may include one coding tree block (CTB) of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to encode samples of the coding tree blocks. These syntax elements describe the properties of different types of units of the coded pixel blocks, and how the video sequence can be reconstructed at the video decoder 30, including inter-frame or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements used to encode samples of the coding tree block. The coding tree block may be an NxN sample block.
[0062] To achieve better performance, the video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or any combination thereof, on the coding tree block of the CTU and divide the CTU into smaller coding units (CUs). Figure 4C As depicted in , the 64x64 CTU 400 is first divided into four smaller CUs, each CU having a block size of 32x32. Among the four smaller CUs, each of CU 410 and CU 420 is divided into four CUs with a block size of 16x16. Each of the two 16x16 CUs 430 and CU440 is further divided into four CUs with a block size of 8x8. Figure 4D Depicts a quadtree data structure that illustrates how Figure 4C As the final result of the partitioning process of the CTU 400 depicted in FIG, each leaf node of the quadtree corresponds to a CU of a corresponding size ranging from 32x32 to 8x8. Figure 4B Like the CTU depicted in , each CU may include one code block (CB) of luma samples and two corresponding code blocks of chroma samples of the same size frame, and syntax elements used to encode the samples of the code blocks. In monochrome pictures or pictures with three separate color planes, a CU may include a single code block and syntax structures used to encode the samples of the code block. Note that Figure 4C and Figure 4D The quadtree partitioning depicted in FIG is for illustration purposes only, and a CTU can be split into CUs based on quadtree / ternary / binary tree partitioning to accommodate varying local characteristics. In a multi-type tree structure, a CTU is partitioned by a quadtree structure, and the leaf CUs of each quadtree can be further partitioned by binary and ternary tree structures. Figure 4E As shown, there are five types of segmentation, namely, quadruple segmentation, horizontal binary segmentation, vertical binary segmentation, horizontal ternary segmentation, and vertical ternary segmentation.
[0063] In some implementations, the video encoder 20 may further partition the code blocks of the CU into one or more MxN prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples on which the same prediction (inter or intra) is applied. The prediction unit (PU) of a CU may include one prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to predict the prediction blocks. In a monochrome picture or a picture with three separate color planes, a PU may include a single prediction block and a syntax structure used to predict the prediction block. The video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma prediction block, the Cb prediction block, and the Cr prediction block of each PU of the CU.
[0064] Video encoder 20 may use intra prediction or inter prediction to generate a prediction block for a PU. If video encoder 20 uses intra prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of a frame associated with the PU. If video encoder 20 uses inter prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of one or more frames other than the frame associated with the PU.
[0065] After the video encoder 20 generates the predicted luma block, Cb block, and Cr block for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting its predicted luma block from the original luma code block of the CU, so that each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma code block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block of the CU, respectively, so that each sample in the Cb residual block of the CU indicates the difference between a Cb sample within one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb code block of the CU, and each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr code block of the CU.
[0066] In addition, if Figure 4CAs shown, the video encoder 20 can use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of the CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks. A transform block is a rectangular (square or non-square) sample block on which the same transform is applied. The transform unit (TU) of the CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Therefore, each TU of the CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with the TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and a syntax structure for transforming the samples of the transform block.
[0067] Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block may be a two-dimensional array of transform coefficients. A transform coefficient may be a scalar. Video encoder 20 may apply one or more transforms to a Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of a TU to generate a Cr coefficient block for the TU.
[0068] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. After the video encoder 20 quantizes the coefficient block, the video encoder 20 may entropy encode the syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform context-adaptive binary arithmetic coding and decoding (CABAC) on the syntax elements indicating the quantized transform coefficients. Finally, the video encoder 20 may output a bitstream including a sequence of bits that form a representation of a coded frame and associated data, and the bitstream is stored in a storage device 32 or transmitted to a target device 14.
[0069] After receiving the bitstream generated by the video encoder 20, the video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. The video decoder 30 can reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 can perform an inverse transform on a coefficient block associated with a TU of the current CU to reconstruct a residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the code block of the current CU by adding samples of the prediction block of the PU of the current CU to corresponding samples of the transform block of the TU of the current CU. After reconstructing the code blocks of each CU of the frame, the video decoder 30 can reconstruct the frame.
[0070] As noted above, video coding primarily uses two modes (i.e., prediction within a frame, or intra-frame prediction, and prediction between frames, or inter-frame prediction) to achieve video compression. Palette-based coding is another coding scheme that has been adopted by many video coding standards. In palette-based coding, which may be particularly suitable for screen-generated content coding, a video codec (e.g., video encoder 20 or video decoder 30) forms a color palette table representing video data for a given block. The palette table includes the most dominant (e.g., frequently used) pixel values in a given block. Pixel values that are not frequently represented in the video data of a given block are not included in the palette table, or are included in the palette table as escape colors.
[0071] Each entry in the palette table includes an index in the palette table for the corresponding pixel value. The palette index for the sample in the block can be encoded and decoded to indicate which entry from the palette table is to be used to predict or reconstruct which sample. The palette mode begins with a process of generating a palette predictor for the first block of a picture, slice, tile, or other such video block grouping. As will be explained below, the palette predictor for subsequent video blocks is typically generated by updating the previously used palette predictor. For the purpose of illustration, it is assumed that the palette predictor is defined at the picture level. In other words, a picture can include multiple code blocks, each with its own palette table, but there is one palette predictor for the entire picture.
[0072] In order to reduce the bits required to signal palette entries in the video bitstream, a video decoder can utilize a palette predictor to determine new palette entries in a palette table for reconstructing a video block. For example, a palette predictor can include palette entries from a previously used palette table, or even be initialized with a most recently used palette table by including all entries of the most recently used palette table. In some implementations, a palette predictor can include fewer entries than all entries from a most recently used palette table, and then incorporate some entries from other previously used palette tables. A palette predictor can have the same size as a palette table used to encode and decode a different block, or can be larger or smaller than a palette table used to encode and decode a different block. In one example, a palette predictor is implemented as a first-in, first-out (FIFO) table that includes 54 palette entries.
[0073] To generate a palette table for a block of video data from a palette predictor, a video decoder may receive a 1-bit flag for each entry of the palette predictor from an encoded video bitstream. The 1-bit flag may have a first value (e.g., binary 1) indicating that the associated entry of the palette predictor is to be included in the palette table, or a second value (e.g., binary 0) indicating that the associated entry of the palette predictor is not to be included in the palette table. If the size of the palette predictor is larger than the palette table for the block of video data, the video decoder may stop receiving more flags once the maximum size of the palette table is reached.
[0074] In some implementations, some entries in a palette table may be signaled directly in the encoded video bitstream, rather than being determined using a palette predictor. For such entries, a video decoder may receive from the encoded video bitstream three separate m-bit values indicating pixel values for the luma and two chroma components associated with the entry, where m represents the bit depth of the video data. Compared to the multiple m-bit values required to directly signal palette entries, those palette entries derived from the palette predictor require only a 1-bit flag. Therefore, using a palette predictor to signal some or all palette entries can significantly reduce the number of bits required to signal new palette table entries, thereby improving the overall codec efficiency of palette mode codecs.
[0075] In many cases, the palette predictor for a block is determined based on the palette table used to encode one or more previously encoded blocks. However, when encoding and decoding the first coding tree unit in a picture, slice, or tile, the palette table of the previously encoded block may not be available. Therefore, the palette predictor cannot be generated using the entries of the previously used palette table. In such a case, a palette predictor initializer sequence can be signaled in a sequence parameter set (SPS) and / or a picture parameter set (PPS), which is the value used to generate the palette predictor when the previously used palette table is not available. SPS generally refers to the syntax structure of syntax elements applicable to a series of consecutively encoded video pictures called a coded video sequence (CVS), as determined by the content of the syntax elements found in the PPS referenced by the syntax elements found in each slice segment header. PPS generally refers to the syntax structure of syntax elements applicable to one or more individual pictures within the CVS, as determined by the syntax elements found in each slice segment header. Therefore, the SPS is generally considered to be a higher-level syntax structure than the PPS, meaning that the syntax elements included in the SPS generally change less frequently and apply to a larger portion of the video data than the syntax elements included in the PPS.
[0076] Figure 5 The figure illustrates a process 500 of deriving chroma samples 502 from luma samples 504 of a code block 506 of a video frame of a bitstream according to some embodiments. The code block 506 of the video frame includes a plurality of pixels, and each pixel is formed by a plurality of color elements (e.g., blue, green, and red). In video encoding and decoding, the brightness and color information of the plurality of pixels are represented by a plurality of luma samples 504 and a plurality of chroma samples 502, respectively. Each of the plurality of pixels corresponds uniquely to a single corresponding luma sample 504. Each chroma sample 502 corresponds to a corresponding set of luma samples 504 according to a subsampling scheme. Each luma sample has a luma component Y', and each chroma sample 502 has a blue difference chroma component Cb and a red difference chroma component Cr. The subsampling scheme (Y':Cb:Cr) of the luma and chroma components has a three-part ratio, such as 4:1:1, 4:2:0, 4:2:2, 4:4:4, and 4:4:0. In Figure 5 , the luma samples 504 and chroma samples 502 of the video frame follow a subsampling scheme with a three-part ratio equal to 4:1:1.
[0077] In some embodiments, a code block 506 of a video frame includes 2M x 2N pixels, corresponding to 2M luminance samples across width and 2N luminance samples across height. Optionally, M and N are equal to or different from each other. Figure 5, the luma samples 504 of the video frame are encoded at a resolution of 2M x 2N, while the chroma samples 502 are encoded at a smaller resolution of M x N. In practice, the chroma samples 502 can be encoded at different chroma resolutions such as 2M x 2N (e.g., 4:4:4 full sampling), 2M xN (e.g., 4:4:0 subsampling), M x 2N (e.g., 4:2:2 subsampling), and 1 / 2M x 2N (e.g., 4:1:1 subsampling).
[0078] In some embodiments, chroma samples 502 and luma samples 504 of the same video frame are separately encoded in the video encoder 20, forwarded from the video encoder 20 to the video decoder 30, and separately decoded in the video decoder 30. In some cases, the luma samples 504 may be used to refine the decoded chroma samples 502. Alternatively, in some embodiments, luma samples 504 of a code block 506 of a video frame are encoded in the video encoder 20 and provided to the video decoder 30 without the chroma samples 502 of the same code block 506 of the video frame. The luma samples 504 of the code block 506 are reconstructed in the video decoder 30, and the chroma samples 502 are derived from the reconstructed luma samples 504 of the code block 506. Assuming that the luma samples 504 have a first luma resolution that is greater than a second chroma resolution of the chroma samples 502, the luma samples 504 of the code block 506 are converted into a set of downsampled luma samples 508 having a third luma resolution equal to the second chroma resolution according to a predefined luma interpolation scheme 510. The set of downsampled luma samples 508 is further converted into chroma samples 502 of a code block 506 of a video frame according to a linear mapping model 512, which is represented as follows:
[0079] Y= α X+ β (1)
[0080] Where X and Y correspond to the luma value of the downsampled luma sample 508 and the chroma value of the corresponding chroma sample 502 , respectively, and α and β are two linear coefficients of the linear mapping model 512 .
[0081] Specifically, the luma value (DLuma) 508 of each downsampled luma sample 508 is derived from the luma values (Luma) 504 of a number of adjacent luma samples according to a luma interpolation scheme 510 as follows:
[0082]
[0083] Where k represents the number of adjacent luma samples, and f(i) represents the filter coefficient for each of the k adjacent luma samples 504. The linear interpolation scheme 510 may be combined with the linear mapping model 512 to determine the luma-chroma cross-component filter model 520 as follows:
[0084]
[0085] Each chroma sample 502 can be derived directly from k adjacent luma samples 504 based on a cross-component linear model, i.e., using the cross-component filter model 520 above. In some embodiments, given two linear coefficients α and β for each code block 506, all chroma samples 502 of a code block 506 are derived directly from luma samples 504 located in the same corresponding code block 506 using the cross-component filter model 520.
[0086] In some embodiments, luma samples 504 and chroma samples 502 of a video frame follow a subsampling scheme in which, on average, every four luma samples 504 correspond to one chroma sample 504 having a blue difference chroma component Cb and a red difference chroma component Cr. Stated another way, a first luma resolution of the luma samples 504 is four times a second chroma resolution of the chroma samples 502. On average, every four luma samples 504 are combined into one downsampled luma sample 508 in a set of luma samples 508 having a third luma resolution equal to the second chroma resolution. In some embodiments, for each luma sample 508, more than four adjacent luma samples 504 are applied to derive a corresponding chroma sample 502. For a subset of luma samples 504, each luma sample 504 is applied more than once to generate more than one downsampled luma sample 508 and / or more than one chroma sample 502. More details about downsampling the plurality of luma samples 504 into luma samples 508 and / or chroma samples 502 based on the predefined interpolation scheme 510 are described below in Figures 6A-6D Have a discussion.
[0087] In some embodiments, the linear mapping model 512 applied to convert the luma sample 508 into the chroma sample 502 is derived using a maximum-minimum method. Specifically, the maximum downsampled luma sample is identified from the downsampled luma sample set, and the minimum downsampled luma sample is identified from the downsampled luma sample set. The maximum chroma sample corresponds to the maximum downsampled luma sample, and the minimum chroma sample corresponds to the minimum downsampled luma sample. The maximum and minimum chroma samples are applied together with the maximum and minimum downsampled luma samples to determine the linear mapping model 512, that is, to determine the two coefficients α and β in equation (1). After deriving the linear mapping model 512, the video codec applies the linear mapping model 512 to the luma sample 508 downsampled from the luma sample 504 to generate the corresponding chroma sample 502 in the code block 506.
[0088] The code blocks of the video frame are reconstructed according to the sequence of sorting. In some cases, the code block 506 is not in the first row or the first column of the video frame, and when the code block 506 is being processed, the selected group of pixels has been reconstructed. For example, the selected group of pixels includes the pixels immediately above and on the left of the pixels in the code block 506. The selected group of chroma samples 516, the selected group of luma samples 518, and the selected group of downsampled luma samples 514 have been reconstructed for the selected group of pixels, and can be used to derive the linear mapping model 512, for example, based on the maximum minimum method. The chroma samples 516, the luma samples 518, and the downsampled luma samples 514 are outside the boundary of the code block 506 and are adjacent to the boundary. Specifically, in some embodiments, the video codec searches for the downsampled luma sample group that has been coded (for example, the selected group of downsampled luma samples 514 corresponding to the top adjacent luma samples and the left adjacent luma samples) to identify the maximum downsampled luma sample 514A-1 and the minimum downsampled luma sample 514B-1. The video codec then identifies previously coded chroma samples 516A-1 and 516B-1 that correspond to the maximum and minimum downsampled luma samples 514A-1 and 514B-1 (e.g., in a selected group of chroma samples 516 that include top-neighboring chroma samples and left-neighboring chroma samples). In this way, a linear mapping model 512 can be derived based on these associated downsampled luma samples (514A-1 and 514B-1) and chroma samples (516A-1 and 516B-1).
[0089] Alternatively, in some embodiments, the video codec searches the luma sample group (e.g., a selected group of luma samples 518 including top and left adjacent luma samples) to identify the luma sample 518A having (i) the maximum luma value and (ii) the minimum luma value in the luma sample group, without performing downsampling on the selected luma sample group to identify the maximum and minimum luma samples 518A and 518B. Next, the video codec performs downsampling in a region associated with the maximum and minimum luma samples 518A and 518B (e.g., in a region having six samples, using a weighted averaging scheme known in the art, including six-tap downsampling, etc.) to generate a downsampled luma sample 514A-2 as a maximum luma sample (which may or may not be exactly the maximum downsampled luma sample 514A-1) and a downsampled luma sample 514B-2 as a minimum luma sample (which may or may not be exactly the minimum downsampled luma sample 514B-1). The video codec then identifies chroma samples 516A-2 and 516B-2 corresponding to the downsampled luma samples 514A-2 and 514B-2 (e.g., in a chroma sample group 516 that includes top and left adjacent chroma samples). In this way, a linear mapping model 512 can be derived based on the downsampled luma samples (514A-2 and 514B-2) and chroma samples (516A-2 and 516B-2).
[0090] Alternatively, in some embodiments, the video codec searches a group of chroma samples (e.g., a group of chroma samples 516 including top and left neighboring chroma samples) to identify maximum and minimum chroma samples 516A-3 and 516B-3 (e.g., chroma samples having maximum and minimum chroma values, respectively). The video codec then identifies downsampled luma samples 514A-3 and 514B-3 (e.g., among a group of downsampled luma samples 514 including top and left neighboring luma samples) that correspond to the maximum and minimum chroma samples 516A-3 and 516B-3. In this way, a linear mapping model 512 is derived based on the downsampled luma samples (514A-3 and 514B-3) and the chroma samples (516A-3 and 516B-3).
[0091] Alternatively, in some embodiments, the video codec searches the group of downsampled luma samples (e.g., the selected group of downsampled luma samples 514) to identify a predefined number (e.g., two) of downsampled luma samples (e.g., 514A-4 and 514A-5) having a maximum luma value and a predefined number (e.g., two) of downsampled luma samples (e.g., 514B-4 and 514B-5) having a minimum luma value. Next, the video codec identifies chroma samples (e.g., 516A-4, 516A-5, 516B-4, and 516B-5) in the selected group of chroma samples 516, each chroma sample corresponding to a corresponding group in the group of maximum downsampled luma samples 514A-4 and 514A-5 and the group of minimum downsampled luma samples 514B-4 and 514B-5, respectively. Next, the video codec performs a weighted average of the values (e.g., chroma or luma values) within each identified group of chroma and luma samples to generate a maximum average chroma value (e.g., of chroma samples 516A-4 and 516A-5), a minimum average chroma value (e.g., of chroma samples 516B-4 and 516B-5), a maximum average downsampled luma value (e.g., of luma samples 514A-4 and 514A-5), and a minimum average downsampled luma value (e.g., of luma samples 514B-4 and 514B-5). In this way, a linear mapping model 512 can be derived based on the downsampled luma samples (514A-4, 514-A-5, 514B-4, and 514B-5) and the chroma samples (516A-4, 516A-5, 516B-4, and 516B-5).
[0092] In addition, in some embodiments, the linear mapping model 512 is derived by creating a linear fit curve for a plurality of downsampled luma samples (e.g., in a selected group of downsampled luma samples 514) and a plurality of corresponding chroma samples (e.g., in a selected group of chroma samples 516). Such a curve fit has a deviation that optionally meets an error tolerance. The plurality of downsampled luma samples includes a predefined number (e.g., greater than 10) of downsampled luma samples corresponding to a predefined number of chroma samples. In some embodiments, for the purpose of deriving the linear mapping model 512, the downsampled luma samples are randomly selected from a group of adjacent downsampled luma samples 514.
[0093] Figures 6A-6D 506A-506D, each of which includes a plurality of luma samples 504 to be converted into a plurality of chroma samples 502. Each cross ("x") represents the position of a luma sample 504, and each circle ("o") represents the position of a chroma sample 502 or a downsampled luma sample 508. Each circled cross 508 overlap. Each code block 506 is marked with a block boundary 602. Each code block 506 complies with a corresponding luma and chroma component subsampling scheme (Y':Cb:Cr), and a corresponding luma interpolation scheme 510 is used to downsample the luma samples 504 of the corresponding code block 506 into corresponding downsampled luma samples 508.
[0094] In some embodiments of the present application, for each code block 506, especially when the downsampled luma samples 508 are adjacent to the block boundary 602, all downsampled luma samples 508 are derived from the adjacent luma samples 504 in the same corresponding code block 506 using a luma interpolation scheme 510. Each downsampled luma sample 508 always overlaps with the corresponding chroma sample 502, and the chroma sample 502 can therefore be derived from the downsampled luma sample 508 based on a linear mapping model 512.
[0095] In some embodiments, the cross-component filter model 520 is a combination of the luma interpolation scheme 510 and the linear mapping model 512. Given the two linear coefficients α and β of the linear mapping model 512 for each code block 506 and the filter coefficients f(i) of the luma interpolation scheme 510, particularly when the downsampled luma samples 508 are adjacent to the block boundary 602, the chroma samples 502 of the code block 506 are directly derived from the luma samples 504 located in the same corresponding code block 506 using the cross-component filter model 520.
[0096] In some embodiments, the code block 506 is the first code block of the video frame, and no other code blocks 506 are reconstructed before the code block 506. Such a code block 506 is optionally located at the upper left corner of the video frame. The linear mapping model 512 is determined independently of the code blocks in the video frame (e.g., based on the previous video frame). Alternatively, in some embodiments, the code block 506 is not the first code block of the video frame, and one or more other code blocks 506 have been reconstructed before the code block 506. For example, the code block 506 is located in the middle of the video frame, and the code blocks immediately above or to the left of the code block 506 have been processed and can be used to determine the linear mapping model 512 or reconstruct the luminance or chrominance samples in the code block 506. In some cases, the code block 506 is adjacent to the upper boundary of the video frame, and at least one code block immediately to the left of the code block 506 has been processed and can be used to determine the linear mapping model 512 or reconstruct the luminance or chrominance samples in the code block 506. In some cases, the code block 506 is adjacent to the left boundary of the video frame, and at least one code block immediately above the code block 506 has been processed and can be used to determine the linear mapping model 512 or to reconstruct the luma or chroma samples in the code block 506. Regardless of the position of the code block 506, the code blocks to the right or below the code block 506 have not been processed and cannot be used to determine the linear mapping model 512 for the code block 506 or to reconstruct the luma or chroma samples in the code block 506.
[0097] In some embodiments, the same predefined luma interpolation scheme 510 or cross component filter model 520 is applied to generate all downsampled luma samples 508 or chroma samples 502, respectively, across the entire code block 506. Alternatively, in some embodiments, the predefined luma interpolation scheme 510 varies with the position of the downsampled luma samples 508 or the position of the boundary pixels associated with the downsampled luma samples 508. Likewise, in some embodiments, the cross component filter model 520 varies with the position of the chroma samples 502 or the position of the boundary pixels associated with the chroma samples 502. For example, each of the predefined luma interpolation scheme 510 and the cross component filter model 520 corresponds to one of a cross, block, or T-shaped filter configuration.
[0098] Reference Fig. 6A, code block 506A is obtained using a bitstream and includes a plurality of luma samples 504 that have been reconstructed from the bitstream. A plurality of corresponding chroma samples 502 are derived from the plurality of luma samples 504. In code block 506A, the subsampling scheme (Y':Cb:Cr) of the luma and chroma components 504 and 502 has a three-part ratio of 4:2:0. Each chroma sample 502 is combined from six adjacent luma samples 504 according to a cross-component filter model 520. The six adjacent luma samples 504 are located at the upper left corner, directly above, upper right corner, lower left corner, directly below, and lower right corner of the corresponding chroma sample 502. In other words, the six adjacent luma samples 504 of each chroma sample 502 are converted into downsampled luma samples 508 overlapping the corresponding chroma samples 502 according to a predefined luma interpolation scheme 510, and then converted into the corresponding chroma samples 502 according to a linear mapping model 512. In one example, the first chroma sample 502A and its corresponding six adjacent luma samples 504A-1 to 504A-6 are completely enclosed within the block boundary 602, and the first chroma sample 502A can be derived from the adjacent luma samples 504A-1 to 504A-6 based on a cross-component filter model 520 or a combination of a luma interpolation scheme 510 and a linear mapping model 512.
[0099] On the contrary, the second chroma sample 502B is next to the left block boundary 602A of code block 506A, and is corresponding to six adjacent luminance samples 504B-1 to 504B-6 according to the cross component filter model 520. In some cases, the left block boundary 602A of code block 506 overlaps with the left boundary of video frame, and two adjacent luminance samples 504B-1 and 504B-6 are unavailable in this or any other code block. Outside code block 506 and unavailable luminance samples 504B-1 and 504B-6 can be copied from luminance samples 504B-2 and 504B-5 next to luminance samples 504B-1 and 504B-6 respectively. In one example, luminance samples 504B-1 and 504B-6 are copied from 504B-4 and 504B-3 symmetrical with luminance samples 504B-1 and 504B-6 about chroma sample 502B. Alternatively, in some cases, the left block boundary 602A of the code block 506 does not overlap with the left boundary of the video frame. The two adjacent luma samples 504B-1 and 504B-6 are not available in the code block 506 itself, however, they can be provided by another adjacent code block that has been reconstructed before the code block 506. The luma samples 504B-1 and 504B-6 outside the code block 506 can still be copied from the luma samples 504B-2 to 504B-5 within the code block 506 without loading the luma samples 504B-1 and 504B-6 from any other code block. In these ways, the second chroma sample 502B and / or the second luma sample 508B are further derived from the adjacent luma samples 504B-1 to 504B-6, wherein the two luma samples 504B-1 and 504B-6 are optionally allocated from two different luma samples (e.g., 504B-2 and 504B-5) included in the code block 506A.
[0100] In other words, when a bitstream having a plurality of luma samples for a plurality of pixels in a video frame is obtained, the plurality of pixels belong to a code block 506A and include boundary pixels located inside the code block 506 and adjacent to the boundary 602 of the code block 506A. The boundary pixel (e.g., corresponding to the luma sample 504B-2) corresponds to one or more neighboring pixels (e.g., corresponding to the luma sample 504B-1) that are unavailable. Each of the one or more neighboring pixels is outside the code block 506A. In some embodiments, one or more neighboring pixels of the boundary pixel are outside the video frame or image strip. In some embodiments, one or more neighboring pixels of the boundary pixel have not been decoded and will be decoded after the code block 506. In some embodiments, the cross-component filter model 520 is limited to the code block 506, and any neighboring pixels outside the code block 506 are considered unavailable. Luma samples 504B-2 and 504B-5 corresponding to the boundary pixels are respectively assigned to luma samples 504B-1 and 504B-6 corresponding to each of the one or more neighboring pixels. The boundary luma samples 508B are determined based on at least the luma samples 504B-1 to 504B-6 of the boundary pixel and one or more neighboring pixels according to a predefined luma interpolation scheme 510, and are used to determine the boundary chroma samples 502B according to a linear mapping model 512. Alternatively, the boundary chroma samples 502B are determined from the luma samples 504B-1 to 504B-6 of the boundary pixel and one or more neighboring pixels according to a cross-component filter model 520.
[0101] Reference Figure 6B , in code block 506B, the subsampling scheme (Y':Cb:Cr) of the luma and chroma components 504 and 502 has a three-part ratio of 4:2:0. Each chroma sample 502 is combined from five adjacent luma samples 504 according to a cross-component filter model 520 having a cross-filter shape. One of the five adjacent luma samples 504 overlaps the chroma sample 502, while the remaining four adjacent luma samples 504 are located directly above, below, to the left, and to the right of the corresponding chroma sample 502. In other words, the five adjacent luma samples 504 of each chroma sample 502 are converted into downsampled luma samples 508 overlapping the corresponding chroma samples 502 according to a predefined luma interpolation scheme 510, and then converted into the corresponding chroma samples 502 according to a linear mapping model 512. In one example, the third chroma sample 502C and its corresponding five adjacent luma samples 504C-1 to 504C-5 are completely enclosed within the block boundary 602, and the third chroma sample 502C can be derived from the adjacent luma samples 504C-1 to 504C-5 based on the cross-component filter model 520 or the combination of the luma interpolation scheme 510 and the linear mapping model 512.
[0102] The fourth chroma sample 502D is adjacent to the left block boundary 602A of code block 506B, and corresponds to five adjacent luminance samples 504D-1 to 504D-5 according to the cross component filter model 520. In some cases, the left block boundary 602A of code block 506B overlaps with the left boundary of the video frame, and adjacent luminance sample 504D-5 is unavailable in this or any other code block. Luminance sample 504D-5 is copied from the luminance sample 504D-1 of adjacent luminance sample 504D-5 or from the luminance sample 504D-3 symmetrical with luminance sample 504D-5 about the center luminance sample 504D-1. In some cases, the left block boundary 602A of code block 506 does not overlap with the left boundary of the video frame. Adjacent luminance sample 504D-5 is unavailable in code block 506 itself, yet, can be provided by another adjacent code block reconstructed before code block 506B. The luma sample 504D-5 outside the code block 506 can still be copied from the luma sample 504D-1 or 504D-3 located within the code block 506 without loading the luma sample 504D-5 from any other code block. In these ways, the fourth chroma sample 502D and / or the fourth luma sample 508D are derived from the adjacent luma samples 504D-1 to 504D-5, wherein the luma sample 504D-5 is optionally allocated based on the luma sample 504D-1 or 504D-3 within the code block 506B.
[0103] The 5th chroma sample 502E is next to the left block boundary 602A and the top block boundary 602B of code block 506B, and is corresponding to five adjacent brightness samples 504E-1 to 504E-5 according to the cross component filter model 520. In certain embodiments, code block 506B is the first code block in the video frame, and two adjacent brightness samples 504E-2 and 504E-5 are unavailable in this or any other code block. Luminance sample 504E-2 and 504E-5 are copied from brightness sample 504E-1. In some cases, code block 506B is not located at the upper left corner of the video frame, and its left side and top adjacent code block are available. Adjacent brightness samples 504E-2 and 504E-5 are unavailable in code block 506 itself, yet, can be provided by another adjacent code block reconstructed before code block 506. Alternatively, luma samples 504E-2 and 504E-5 may still be copied from luma sample 504E-1 located within code block 506 without loading these luma samples from any other code block. In these ways, the fifth chroma sample 502E and / or the fifth luma sample 508E are derived from adjacent luma samples 504E-1 to 504E-5, wherein luma samples 504E-2 and 504E-5 are optionally allocated based on another luma sample 504E-1 adjacent to the block boundary 602 of code block 506B.
[0104] Reference Figure 6C, in code block 506C, the subsampling scheme (Y':Cb:Cr) of luma and chroma components 504 and 502 has a three-part ratio of 4:2:2. Each chroma sample 502 is combined from six adjacent luma samples 504 according to a cross-component filter model 520 having a 2x3 filter shape. One of the six adjacent luma samples 504 overlaps with the chroma sample 502, while the remaining five adjacent luma samples are located to the left, right, bottom left, directly below, and bottom right of the corresponding chroma sample 502. These six adjacent luma samples 504 of each chroma sample 502 are converted into downsampled luma samples 508 overlapping with the corresponding chroma sample 502 according to a predefined luma interpolation scheme 510, and then converted into the corresponding chroma sample 502 according to a linear mapping model 512. In one example, the sixth chroma sample 502F and its corresponding six adjacent luma samples 504F-1 to 504F-6 are completely enclosed within the block boundary 602, and the sixth chroma sample 502F can be derived from the adjacent luma samples 504F-1 to 504F-6 based on the cross-component filter model 520 or the combination of the luma interpolation scheme 510 and the linear mapping model 512.
[0105] The seventh chroma sample 502G is adjacent to the left block boundary 602A and the bottom block boundary 602C of the code block 506C, and corresponds to six adjacent luminance samples 504G-1 to 504G-6 according to the cross component filter model 520. In some embodiments, code block 506C is located at the lower left corner of the video frame, and the left and bottom block boundaries 602A and 602C overlap with the left block boundary and the bottom block boundary of the video frame. In some embodiments, code block 506C is adjacent to the left boundary of the video frame, so luminance samples 504G-1 and 504G-6 do not exist, and luminance samples 504G-4 and 504G-5 have not been decoded. At the lower left corner or left boundary of the video frame, four adjacent luminance samples 504G-1, 504G-4, 504G-5 and 504G-6 are not available in this or any other code block. Each of these luma samples is copied from luma sample 504G-2 or 504G-3, for example, luma sample 504G-4 is copied from luma sample 504G-3, and luma samples 504G-1, 504G-5, and 504G-6 are copied from luma sample 504G-2. In some cases, code block 506C is not located at the lower left corner of the video frame, and its left and bottom adjacent code blocks are available. Adjacent luma samples 504G-1, 504G-4, 504G-5, and 504G-6 are not available in code block 506 itself, however, they can be provided by another adjacent code block that has been reconstructed before code block 506. Alternatively, luma samples 504G-1, 504G-4, 504G-5, and 504G-6 can still be copied from luma samples 504G-2 to 504G-3 located within code block 506 without loading these luma samples from any other code block. In these ways, the seventh chroma sample 502G and / or the seventh luma sample 508G are derived from adjacent luma samples 504G-1 to 504G-6, where the luma samples 504G-1, 504G-4, 504G-5 and 504G-6 are optionally allocated based on another two luma samples 504G-2 and 504G-3 of the block boundary adjacent to the code block 506C.
[0106] Reference Fig.6D, in code block 506D, the subsampling scheme (Y':Cb:Cr) of luma and chroma components 504 and 502 has a three-part ratio of 4:4:4. Each chroma sample 502 is combined from six adjacent luma samples 504 according to a cross-component filter model 520 having a 2x3 filter shape. One of the six adjacent luma samples 504 overlaps with the chroma sample 502, while the remaining five adjacent luma samples are located to the left, right, bottom left, directly below, and bottom right of the corresponding chroma sample 502. These six adjacent luma samples 504 of each chroma sample 502 are converted into downsampled luma samples 508 overlapping with the corresponding chroma sample 502 according to a predefined luma interpolation scheme 510, and then converted into the corresponding chroma sample 502 according to a linear mapping model 512.
[0107] The eighth chroma sample 502H is adjacent to the bottom block boundary 602C and the right block boundary 602D of the code block 506D, and corresponds to the six adjacent luminance samples 504H-1 to 504H-6 according to the cross component filter model 520. In some embodiments, the code block 506D is located at the lower right corner of the video frame, and the right and bottom block boundaries 602D and 602C overlap with the right block boundary and the bottom block boundary of the video frame. The four adjacent luminance samples 504H-3 to 504H-6 are not available in this or any other code block. Alternatively, in some cases, the code block 506D is not located at the lower left corner of the video frame, and its right and bottom adjacent code blocks are still not available because these code blocks have not been reconstructed. Each of these luma samples 504H-3 to 504H-6 is copied from luma sample 504H-1 or 504H-2, for example, luma sample 504H-6 is copied from luma sample 504H-1, and luma samples 504H-3 to 504H-5 are copied from luma sample 504H-2. In these ways, the eighth chroma sample 502H and / or the eighth luma sample 508H are derived from the adjacent luma samples 504H-1 to 504H-6, wherein the luma samples 504H-3 to 504H-6 are optionally allocated based on the other two luma samples 504H-1 and 504H-2 of the block boundary 602 adjacent to the code block 506D.
[0108] Figure 7 Flowchart of a method 700 for decoding video data implemented at an electronic device according to some embodiments. A bitstream having a code block 506 is obtained (702), the code block including a plurality of luma samples 504 for a plurality of pixels in a video frame. The plurality of pixels include a boundary pixel located within the code block 506 and adjacent to a boundary 602 of the code block 506. The electronic device determines (704) that one or more neighboring pixels of the boundary pixel are unavailable. Each of the one or more neighboring pixels is outside the code block 506. Luma samples corresponding to the boundary pixels (e.g., Figure 6C504G-1) is assigned (706) to the brightness samples corresponding to each of the one or more neighboring pixels (e.g., Figure 6C The electronic device determines the boundary chrominance samples (e.g., 504G-1 to 502G-6) according to the cross-component filter model 120 based at least on the brightness samples (e.g., 502G-1 to 502G-6) of the boundary pixel and one or more neighboring pixels. Figure 6C Specifically, the electronic device determines (710) a boundary brightness sample (e.g., 502G-1 to 502G-6) based on at least the brightness samples of the boundary pixel and one or more adjacent pixels according to the predefined brightness interpolation scheme 510. Figure 6C 508G in ). Boundary chroma samples (e.g., Figure 6C 502G) in FIG. 512 is determined from boundary brightness samples according to the linear mapping model 512 (712).
[0109] In some embodiments, the predefined luma interpolation scheme 510 is determined based at least on the location of the boundary pixel, and one or more neighboring pixels are adjacent to the boundary pixel based on the predefined luma interpolation scheme 510. In some cases, the predefined luma interpolation scheme 510 is also determined based on a subsampling scheme, in which luma samples and chroma samples of multiple pixels follow a three-part Y'CbCr ratio. In some cases, the predefined luma interpolation scheme 510 is also determined based on a syntax element derived from the bitstream (e.g., "sps_chroma_collocated_vertical_flag"). Optionally, the predefined luma interpolation scheme 510 corresponds to one of a cross, block, or T-shaped filter configuration.
[0110] In some embodiments, one or more neighboring pixels of the boundary pixel are outside the video frame or image slice. In some embodiments, one or more neighboring pixels of the boundary pixel have not been decoded and will be decoded after the code block.
[0111] In some embodiments, the plurality of pixels includes a first set of pixels that are completely included in the code block. Figures 6A-6C 508A, 508C, and 508F in FIG. 5A , 508C, and 508F in FIG. 5B ) are determined from a set of luma samples corresponding to a first set of pixels according to a predefined luma interpolation scheme 510. Internal chroma samples corresponding to the first set of pixels (e.g., Figures 6A-6C 502A, 502C and 502F) are determined based on the internal brightness samples according to the linear mapping model 512.
[0112] In some embodiments, for example, Fig. 6AIn the embodiment, the plurality of pixels further include an internal pixel, which is inside the code block and corresponds to the internal luma samples (e.g., 504B-3, 504B-4). The boundary luma samples (e.g., 508B) are determined according to a predefined luma interpolation scheme 510 based on the internal luma samples (e.g., 504B-3, 504B-4) of the internal pixels and the luma samples (e.g., 504B-1, 504B-2) of the boundary pixels and one or more neighboring pixels.
[0113] In some embodiments, for example, Fig. 6A , the boundary pixel is a first boundary pixel (e.g., corresponding to luma sample 504B-2), and the one or more neighboring pixels include one or more first neighboring pixels (e.g., corresponding to luma sample 504B-1). The plurality of pixels include a second boundary pixel (e.g., corresponding to luma sample 504B-5) that is inside code block 506 and adjacent to corresponding block boundary 602 of code block 506. The electronic device creates one or more second neighboring pixels (e.g., corresponding to luma sample 504B-6) that are adjacent to the second boundary pixel. Each of the one or more second neighboring pixels is outside code block 506. The luma sample corresponding to the second boundary pixel (e.g., 504B-5) is assigned to the luma sample corresponding to each of the one or more second neighboring pixels (e.g., 504B-6), wherein the boundary luma sample 508B is determined based on luma samples of at least the first and second boundary pixels (e.g., corresponding to luma samples 504B-2 and 504B-5) and the first and second neighboring pixels (e.g., corresponding to luma samples 504B-1 and 504B-6) according to a predefined luma interpolation scheme 510.
[0114] In some embodiments, for example, Figure 6B , the one or more neighboring pixels include only one neighboring pixel (e.g., corresponding to luma sample 504D-5) that is on the opposite side of boundary 602A relative to the boundary pixel (e.g., corresponding to luma sample 504D-1) and has a corresponding luma sample (e.g., 504D-5) assigned from the luma sample corresponding to the boundary pixel (e.g., 504D-1).
[0115] In some embodiments, for example, Figure 6C, the boundary includes a first boundary 602A, and a boundary pixel (e.g., corresponding to luma sample 504G-1) is adjacent to a corner of the code block formed between the first boundary 602A and a second boundary 602C perpendicular to the first boundary 602A. The one or more neighboring pixels include a first neighboring pixel (e.g., corresponding to luma sample 504G-1) on an opposite side of the first boundary 602A relative to the boundary pixel and at least one of: (1) a second neighboring pixel (e.g., corresponding to luma sample 504G-5) on an opposite side of the second boundary relative to the boundary pixel, and (2) a third neighboring pixel (e.g., corresponding to luma sample 504G-6) across a corner of the code block 506C opposite to the boundary pixel.
[0116] In some embodiments, according to the subsampling scheme, luma samples and chroma samples of a plurality of pixels conform to a three-part Y'CbCr ratio having one of the following ratios: (1) 4:1:1, where every four horizontal pixels correspond to four luma samples, one blue difference chroma sample Cb, and one red difference chroma sample Cr; (2) 4:2:0, where every four pixels in each 2x2 pixel block correspond to four luma samples, one blue difference chroma sample Cb, and one red difference chroma sample Cr; (3) 4:2:2, where every four pixels correspond to four luma samples, two blue difference chroma samples Cb, and two red difference chroma samples Cr; and (4) 4:4:4, where every four pixels correspond to four luma samples, four blue difference chroma samples Cb, and four red difference chroma samples Cr.
[0117] In some embodiments, six adjacent luma samples 504 in the 2x3 subarray are downsampled to replaced luma samples 508 according to a predefined luma interpolation scheme, and each of the left and right columns of luma samples in the 2x3 array is used at least twice to generate replaced luma samples on the same row, e.g. Fig. 6A , Figure 6C and Fig.6D In some embodiments, according to a predefined brightness interpolation scheme, five adjacent brightness samples 504 are downsampled into replaced brightness samples 508 and arranged in a cross-shaped sub-array centered on a central pixel, for example, Figure 6B middle.
[0118] In some embodiments, a bitstream is obtained having a second plurality of luma samples 518 and a second plurality of chroma samples 516 for a second plurality of pixels in a video frame. The replaced plurality of luma samples 514 are determined according to a predefined luma interpolation scheme 510 and have the same resolution as the plurality of chroma samples 516. A first parameter α and a second parameter β for a linear mapping model 512 are determined using the replaced plurality of luma samples 514 and the second plurality of chroma samples 516. Further, in some embodiments, a boundary chroma sample Y is determined from the boundary luma sample X according to the linear mapping model 512 as described in equation (1) below. Referring to Figure 5 In some embodiments, two maximum luma samples 514A-4 and 514A-5 are determined among the replaced plurality of luma samples 514, and two minimum luma samples 514B-4 and 514B-5 are determined among the replaced plurality of luma samples 514. The two maximum luma samples 514A-4 and 514A-5 are averaged into a first luma value, and the two minimum luma samples 514B-4 and 514B-5 are averaged into a second luma value. Two first chroma samples 516A-4 and 516A-5 are determined among the second plurality of chroma samples 514 associated with the two maximum luma samples 514A-4 and 514A-5. Two second chroma samples 516B-4 and 516B-5 are determined among the second plurality of chroma samples associated with the two minimum luma samples 514B-4 and 514B-5. The values of the two first chroma samples 516A-4 and 516A-5 are averaged to a first chroma value corresponding to the first luma value, and the values of the two second chroma samples 516B-4 and 516B-5 are averaged to a second chroma value corresponding to the second luma value. A linear mapping model 512 is derived between the chroma samples 502 and the luma samples 508 based on the first and second luma values and the first and second chroma values.
[0119] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium, which includes, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-temporary tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the implementation described in the present application. A computer program product may include a computer-readable medium.
[0120] The terms used in the description of the implementation herein are only for the purpose of describing a specific implementation and are not intended to limit the scope of the claims. When used in the description of the implementation and in the appended claims, the singular forms "one", "an" and "this / that" are expected to also include plural forms, unless the context clearly states otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be further understood that when used in this specification, the terms "include" and / or "comprising" clearly specify the presence of the features, elements and / or parts of the narration, but do not exclude the presence or addition of one or more other features, elements, parts and / or groups thereof.
[0121] It should also be understood that although the terms first, second, etc. may be used herein to describe various different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode without departing from the scope of the implementation. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.
[0122] The description of the present application is given for the purpose of illustration and description, and it is not intended to be exhaustive or limited to the invention of the disclosed form. Many modifications, variations and alternative implementations will be clear to those of ordinary skill in the art who benefit from the teachings given in the foregoing description and the associated drawings. Embodiments are selected and described in order to best explain the principles of the present invention, practical applications, and to enable other persons skilled in the art to understand the present invention with respect to various different implementations and to best utilize potential principles and different implementations with different modifications suitable for the specific purposes envisioned. Therefore, it should be understood that the scope of the claims should not be limited to the specific examples of the disclosed implementations, and modifications and other implementations are expected to be included within the scope of the appended claims.
Claims
1. A method for encoding video data, comprising: Divide the current video frame into one or more coding blocks; Obtaining a plurality of brightness samples for a plurality of pixels in a video frame, wherein the plurality of pixels belong to a code block and include boundary pixels located inside the code block, wherein the boundary pixels are adjacent to a boundary of the code block; Determining that one or more neighboring pixels of the boundary pixel are unavailable, each of the one or more neighboring pixels being outside the code block; assigning a corresponding luminance sample corresponding to the boundary pixel to a luminance sample corresponding to each of the one or more neighboring pixels; determining a boundary luminance sample based at least on luminance samples of the one or more neighboring pixels and the boundary pixel according to a predefined luminance interpolation scheme; Determine a boundary chrominance sample from the boundary luminance sample according to a linear mapping model; as well as A bitstream is formed based on the boundary luma samples and the boundary chroma samples.
2. The method of claim 1, further comprising: The predefined brightness interpolation scheme is determined based on a position of at least one boundary pixel, wherein based on the predefined brightness interpolation scheme, the one or more neighboring pixels are adjacent to the boundary pixel.
3. The method of claim 1, wherein the predefined luma interpolation scheme is determined based on at least one of a subsampling scheme corresponding to the plurality of pixels and a syntax element to be signaled in the bitstream.
4. The method of claim 1, wherein the one or more neighboring pixels of the boundary pixel are outside the video frame or image strip. 5 . The method of claim 1 , wherein the one or more neighboring pixels of the boundary pixel have not yet been decoded and will be decoded after the code block.
6. The method of claim 1, wherein the plurality of pixels comprises a first set of pixels completely included in the code block, the method further comprising: Determining internal brightness samples from a set of brightness samples corresponding to the first set of pixels according to the predefined brightness interpolation scheme; as well as According to the linear mapping model, internal chrominance samples corresponding to the first pixel set are determined based on the internal luminance samples.
7. The method of claim 1, wherein: The plurality of pixels further include interior pixels that are within the code block and correspond to interior luma samples; and The boundary luma samples are determined according to the predefined luma interpolation scheme based on the interior luma samples of the interior pixels and luma samples of the one or more neighboring pixels and the boundary pixels.
8. The method of claim 1, wherein: The boundary pixel is a first boundary pixel, and the one or more adjacent pixels include one or more first adjacent pixels; The plurality of pixels include a second boundary pixel that is inside the code block and is adjacent to a corresponding boundary of the encoding and decoding block; and The method further comprises: Creating one or more second neighboring pixels adjacent to the second boundary pixel, each of the one or more second neighboring pixels being outside the code block; as well as Allocating a luma sample corresponding to the second boundary pixel to a luma sample corresponding to each of the one or more second neighboring pixels, wherein the boundary luma sample is determined based on luma samples of at least both the first and second boundary pixels and both the first and second neighboring pixels according to the predefined luma interpolation scheme.
9. The method of claim 1, wherein the one or more neighboring pixels include only one neighboring pixel that is on an opposite side of the boundary relative to the boundary pixel and has a corresponding luminance sample assigned from the luminance sample corresponding to the boundary pixel.
10. The method of claim 1, wherein: The boundaries include a first boundary; The boundary pixel is adjacent to a corner of the code block formed between the first boundary and a second boundary perpendicular to the first boundary; The one or more neighboring pixels include a first neighboring pixel on an opposite side of the first boundary relative to the boundary pixel and at least one of: (1) a second neighboring pixel on an opposite side of the second boundary relative to the boundary pixel, and (2) a third neighboring pixel opposite to the boundary pixel across the corner of the code block.
11. The method of claim 3, wherein according to the subsampling scheme, the luma samples and the chroma samples of the plurality of pixels conform to a three-part Y'CbCr ratio having one of the following ratios: (1) 4:1:1, where every four horizontal pixels correspond to four luma samples, one blue difference chroma sample Cb, and one red difference chroma sample Cr; (2) 4:2:0, where every four pixels in each 2x2 pixel block correspond to four luma samples, one blue difference chroma sample Cb, and one red difference chroma sample Cr; (3) 4:2:2, where every four pixels correspond to four luma samples, two blue difference chroma samples Cb, and two red difference chroma samples Cr; and (4) 4:4:4, where every four pixels correspond to four luma samples, four blue difference chroma samples Cb, and four red difference chroma samples Cr.
12. The method of claim 1, wherein six luma samples in the 2x3 subarray are downsampled into replaced luma samples according to the predefined luma interpolation scheme, and at least two luma samples in each of the left and right columns in the 2x3 array are used to generate the replaced luma samples.
13. The method of claim 1, wherein according to a predefined luma interpolation scheme, five luma samples are downsampled into replaced luma samples and arranged in a cross-shaped sub-array centered around a central pixel.
14. The method of claim 1, further comprising: obtaining a second plurality of luma samples and a second plurality of chroma samples from the current video frame; determining, according to the predefined luma interpolation scheme, a replacement plurality of luma samples having the same resolution as the plurality of chroma samples; as well as By using the replaced plurality of luma samples and the second plurality of chroma samples, a first parameter α and a second parameter β for the linear mapping model are derived.
15. The method of claim 1, wherein: The boundary chrominance samples Y are determined from the boundary luma samples X according to the linear mapping model described below using the following equation: Y=α*X+β.
16. The method of claim 14, wherein determining the first parameter α and the second parameter β for the linear mapping model further comprises: identifying two maximum luminance samples among the replaced plurality of luminance samples; determining two minimum luma samples among the replaced plurality of luma samples; determining a first brightness value based on the two maximum brightness samples; determining a second brightness value based on the two minimum brightness samples; identifying two first chroma samples among the second plurality of chroma samples associated with the two maximum luma samples; identifying two second chroma samples among the second plurality of chroma samples associated with the two minimum luma samples; determining a first chrominance value corresponding to the first luminance value based on the two first chrominance samples; determining a second chrominance value corresponding to the second luminance value based on the two second chrominance samples; as well as The linear mapping model is derived in chrominance values and luminance values based on the first and second luminance values and the first and second chrominance values.
17. An electronic device comprising: one or more processors; as well as A memory having instructions stored thereon, which, when executed by the one or more processors, cause the processors to perform the method of any one of claims 1-16 to generate a video bitstream.
18. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the encoding method of any one of claims 1 to 16 and store a video bit stream generated by the corresponding encoding method.
19. A computer program product comprising instructions which, when executed by one or more processors, cause the processors to perform the method of any one of claims 1-16 to generate a video bitstream.
