Cross-component determination of chroma and luma components of video data

Abstract

This application addresses the decoding of video data comprising multiple luminance components of multiple pixels in a video frame. The multiple pixels belong to a code block and include a boundary pixel located within and immediately adjacent to a boundary of the code block. One or more neighboring pixels of the boundary pixel are located outside the code block and are determined to be unavailable. The luminance component corresponding to the boundary pixel is assigned to the luminance component corresponding to each of the one or more neighboring pixels. A boundary luminance component is determined based on at least the luminance components of the one or more neighboring pixels and the boundary pixel, according to a predefined luminance interpolation scheme. The boundary luminance component is then converted into a boundary chrominance component according to a linear mapping model.

Description

[0001] Related applications

[0002] This application is a divisional application of Chinese Patent Application 202080096800.1, which claims priority to U.S. Provisional Patent Application No. 62 / 955,348, filed on December 30, 2019, entitled “Simplifications of Cross-Component Linear Model,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application generally relates to video data encoding, decoding and compression, and in particular to improved methods and systems for encoding and decoding chroma and luminance samples of video frames in a video data bitstream. Background Technology

[0004] Digital video is supported by a wide variety of electronic devices, such as digital televisions, laptops or desktop computers, tablets, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing devices, video streaming devices, and so on. These electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression standards defined by standards such as MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Codec (AVC), High Efficiency Video Codec (HEVC), and Universal Video Codec (VVC). Video compression typically involves performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in the video data. For block-based video codecs, video frames are divided into one or more stripes, each strip containing multiple video blocks, also known as coding tree units (CTUs). Each CTU may contain a coding unit (CU), or be recursively subdivided into smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU can be encoded and decoded in intra-frame mode, inter-frame mode, or IBC mode. In intra-frame encoding (I) of a video frame, video blocks in a strip are encoded using spatial prediction with respect to reference samples in adjacent blocks within the same video frame. In inter-frame encoding (P or B) of a video frame, video blocks in a strip can use spatial prediction with respect to reference samples in adjacent blocks within the same video frame, or use temporal prediction with respect to reference samples in other previous and / or future referenced video frames.

[0005] Spatial or temporal predictions based on previously encoded reference blocks (e.g., neighboring blocks) generate a prediction block for the current video block to be encoded. Finding the reference block can be accomplished using a block-matching algorithm. The residual data representing the pixel differences between the current block to be encoded and the prediction block is called the residual block or prediction error. Inter-frame coded blocks are encoded based on motion vectors and the residual block, with the motion vectors pointing to the reference block in the reference frame that forms the prediction block. The process of determining the motion vectors is typically called motion estimation. Intra-frame coded blocks are encoded based on the intra-frame prediction mode and the residual block. For further compression, the residual block is transformed from the pixel domain to the transform domain (e.g., the frequency domain), producing residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, and then entropy-encoded into the video bitstream for greater compression.

[0006] The encoded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by another electronic device with digital video capabilities, or transmitted directly to that electronic device via wired or wireless means. The electronic device then performs video decompression (the reverse of the video compression process described above), by, for example, parsing the encoded video bitstream to obtain syntax elements from it, and reconstructing the digital video data to its original format based at least in part on the syntax elements obtained from the bitstream, and presenting the reconstructed digital video data on the display of the electronic device.

[0007] A cross-component prediction model is applied to reduce cross-component redundancy between luminance and chrominance samples in a video bitstream. Specifically, in this prediction model, luminance samples are downsampled and used to predict chrominance samples. However, luminance samples are unavailable at some locations in the video frame, and different downsampling filters have been used to generate downsampled luminance samples near these locations. The application of different downsampling filters is incompatible with parallel image processing and may bottleneck the processing rate of the video bitstream. Developing a more efficient cross-component prediction mechanism than current practices would be beneficial. Summary of the Invention

[0008] This application describes several implementations relating to video data encoding and decoding, and more specifically, to improved methods and systems for encoding and decoding chroma and luminance samples of video frames in a video data bitstream. Sample padding is applied to the luminance samples to generate unavailable luminance samples used in the luminance downsampling process. These unavailable samples may be outside the video frame or image stripe, or may not have been encoded / decoded and will be subsequently encoded / decoded. These unavailable samples may optionally be generated by repeated padding or mirror padding. In this way, the same downsampling filter can be used to generate all downsampled luminance samples across the entire video frame, enabling parallel video processing and increasing the corresponding video processing rate of the video data bitstream.

[0009] In one aspect of this application, a method for decoding video data is implemented at an electronic device. The method includes: obtaining multiple luminance samples for multiple pixels in a video frame from a bitstream. The multiple pixels belong to a code block and include boundary pixels, with the boundary pixels located inside the code block and adjacent to its boundary. The method further includes: determining that one or more neighboring pixels of the boundary pixel are unavailable; assigning a luminance sample corresponding to the boundary pixel to a luminance sample corresponding to each of the one or more neighboring pixels; and determining a boundary luminance sample based at least on the luminance samples of the one or more neighboring pixels and the boundary pixel according to a predefined luminance interpolation scheme. Each of the one or more neighboring pixels is located outside the code block. The method further includes: determining a boundary chrominance sample from the boundary luminance sample according to a linear mapping model. In some embodiments, one or more neighboring pixels of the boundary pixel are outside the video frame or image stripe. Alternatively, in some embodiments, one or more neighboring pixels of the boundary pixel have not yet been decoded and will be decoded after the code block.

[0010] In another aspect of this application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When executed by the one or more processing units, these programs cause the electronic device to perform the method for decoding video data as described above.

[0011] In another aspect, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. When executed by the one or more processing units, these programs cause the electronic device to perform the method of decoding video data as described above. Attached Figure Description

[0012] The accompanying drawings, included to provide a further understanding of the described implementations and incorporated herein as a part of this specification, illustrate the described implementations and, together with the description, serve to explain the underlying principles. Similar reference numerals denote corresponding parts.

[0013] Figure 1 The diagram illustrates a block diagram of an exemplary video encoding and decoding system according to some implementations of this disclosure.

[0014] Figure 2 A block diagram illustrating an exemplary video encoder is provided to illustrate some implementations of this disclosure.

[0015] Figure 3 A block diagram illustrating an exemplary video decoder is provided to illustrate some implementations of this disclosure.

[0016] Figures 4A to 4E A block diagram is provided to illustrate how, according to some implementations of this disclosure, a frame is recursively divided into multiple video blocks of different sizes.

[0017] Figure 5 The figure illustrates the process of deriving chroma samples from luminance samples of code blocks of video frames in a bitstream, according to some embodiments.

[0018] Figures 6A to 6D Here are four example code blocks according to some embodiments, each code block comprising multiple luminance samples to be converted into multiple chrominance samples.

[0019] Figure 6E The illustration shows an example sample filling scheme according to some embodiments.

[0020] Figure 7 This is a flowchart of a video data decoding method implemented in an electronic device according to some embodiments. Detailed Implementation

[0021] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be readily apparent to those skilled in the art that various different alternatives may be used without departing from the scope of the claims, and that the subject matter may be implemented without these specific details. For example, it will be readily apparent to those skilled in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0022] Figure 1 The diagram illustrates a block diagram of an exemplary system 10 for parallel encoding and decoding of video blocks, according to some implementations of this disclosure. Figure 1As shown, system 10 includes a source device 12 that generates and encodes video data, which is then decoded by a target device 14. The source device 12 and target device 14 can be any electronic device from a wide variety of sources, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and so on. In some implementations, the source device 12 and target device 14 are equipped with wireless communication capabilities.

[0023] In some implementations, target device 14 may receive encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving encoded video data from source device 12 to target device 14. In one example, link 16 may include a communication medium enabling source device 12 to transmit encoded video data directly to target device 14 in real time. The encoded video data may be modulated according to communication standards such as wireless communication protocols and transmitted to target device 14. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network (LAN), a wide area network (WAN), or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful in facilitating communication from source device 12 to target device 14.

[0024] In some other implementations, the encoded video data can be transferred from output interface 22 to storage device 32. The encoded video data in storage device 32 can then be accessed by target device 14 via input interface 28. Storage device 32 can include any data storage medium of a wide variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, storage device 32 can correspond to a file server or another intermediate storage device that can hold the encoded video data generated by source device 12. Target device 14 can access the stored video data from storage device 32 via streaming or downloading. The file server can be any type of computer capable of storing and transferring encoded video data to target device 14. Exemplary file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. Target device 14 can access encoded video data via any standard data connection, including wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or combinations thereof, suitable for accessing encoded video data stored on a file server. Transmission of encoded video data from storage device 32 can be streaming, downloading, or a combination thereof.

[0025] like Figure 1 As shown, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as video capture devices, such as cameras, video archives containing previously captured video, video feed interfaces that receive video from video content providers, and / or computer graphics systems for generating computer graphics data such as source video, or combinations of such sources. As an example, if video source 18 is a camera in a security monitoring system, then source device 12 and target device 14 can form a video phone or video phone. However, the implementation described in this application can be generally applied to video encoding and decoding, and can be applied to wireless and / or wired applications.

[0026] Captured video, pre-captured video, or computer-generated video can be encoded by video encoder 20. The encoded video data can be directly transmitted to target device 14 via output interface 22 of source device 12. The encoded video data can also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or transmitter.

[0027] Target device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem, and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include a variety of syntax elements generated by video encoder 20 and used by video decoder 30 when decoding the video data. Such syntax elements may be included within the encoded video data, which is transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0028] In some implementations, target device 14 may include display device 34, which may be an integrated display device or an external display device configured to communicate with target device 14. Display device 34 displays decoded video data to a user and may include any of a wide variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0029] Video encoder 20 and video decoder 30 can operate according to proprietary or industry standards, such as VVC, HEVC, MPEG-4 Part 10, Advanced Video Codec (AVC), or extensions of such standards. It should be understood that this application is not limited to a specific video encoding / decoding standard, but can be applied to other video encoding / decoding standards. It is generally contemplated that the video encoder 20 of source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is generally contemplated that the video decoder 30 of target device 14 can be configured to decode video data according to any of these current or future standards.

[0030] Each of the video encoder 20 and the video decoder 30 can be implemented as any circuit system in a wide variety of suitable codec circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When implemented in part in software, the electronic device may store instructions for software in a suitable non-transitory computer-readable medium, and execute these instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, wherein either may be integrated into the corresponding device as part of a combined encoder / decoder (CODEC).

[0031] Figure 2 The diagram illustrates a block diagram of an exemplary video encoder 20 according to some implementations described in this application. The video encoder 20 can perform intra-frame and inter-frame predictive coding of video blocks within a video frame. Intra-frame predictive coding relies on spatial prediction to reduce or remove spatial redundancy in the video data within a given video frame or image. Inter-frame predictive coding relies on temporal prediction to reduce or remove temporal redundancy in the video data within neighboring video frames or images of a video sequence.

[0032] like Figure 2 As shown, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copying (BC) unit 48. In some implementations, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62 for video block reconstruction. A deblocking filter (not shown) can be placed between the summer 62 and the DPB 64 to filter block boundaries to remove block artifacts from the reconstructed video. In addition to the deblocking filter, a loop filter (not shown) can be used to filter the output of the summer 62. The video encoder 20 can take the form of a fixed or programmable hardware unit, or can be divided into one or more hardware units among the fixed or programmable hardware units shown in the figure.

[0033] Video data memory 40 can store video data to be encoded by components of video encoder 20. The video data in video data memory 40 can be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data for use by video encoder 20 in encoding the video data (e.g., in intra-frame or inter-frame predictive coding modes). Video data memory 40 and DPB 64 can be formed from any of a wide variety of memory devices. In various examples, video data memory 40 can be on-chip along with other components of video encoder 20, or off-chip relative to those components.

[0034] like Figure 2As shown, after receiving video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include dividing the video frame into stripes, tiles, or other larger coding units (CUs) based on a predefined splitting structure associated with the video data, such as a quadtree structure. The video frame can be divided into multiple video blocks (or a set of video blocks referred to as tiles). The prediction processing unit 41 can select one of several possible predictive coding modes for the current video block based on error results (e.g., coding rate and distortion level), such as one of several intra-frame predictive coding modes or one of several inter-frame predictive coding modes. The prediction processing unit 41 can provide the resulting intra-frame or inter-frame predictive coded block to the summer 50 to generate a residual block, and to the summer 62 to reconstruct the coded block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements, such as motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information, to the entropy coding unit 56.

[0035] To select a suitable intra-predictive coding mode for the current video block, the intra-predictive processing unit 46 within the prediction processing unit 41 can perform intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be encoded, to provide spatial prediction. The motion estimation unit 42 and motion compensation unit 44 within the prediction processing unit 41 perform inter-predictive coding of the current video block relative to one or more prediction blocks in one or more reference frames, to provide temporal prediction. The video encoder 20 can perform multiple coding passes, for example, to select a suitable coding mode for each block of video data.

[0036] In some implementations, motion estimation unit 42 determines the inter-frame prediction mode for the current video frame by generating motion vectors according to a predetermined pattern within the video frame sequence. The motion vectors indicate the displacement of the prediction unit (PU) of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. For example, the motion vectors may indicate the displacement of the PU of a video block within the current video frame or picture relative to a prediction block within a reference frame (or other coded unit), where the prediction block is relative to the currently encoded block within the current frame (or other coded unit). The predetermined pattern may designate the video frames in the sequence as P-frames or B-frames. Intra-frame BC unit 48 may determine vectors for intra-frame BC coding, such as block vectors, in a manner similar to how motion estimation unit 42 determines motion vectors for inter-frame prediction, or the motion estimation unit 42 may determine block vectors.

[0037] A prediction block is a block of a reference frame that is considered to closely match the PU of the video block to be encoded in terms of pixel differences, which can be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference measures. In some implementations, the video encoder 20 can compute values ​​for sub-integer pixel positions of the reference frame stored in the DPB 64. For example, the video encoder 20 can interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Therefore, the motion estimation unit 42 can perform motion search relative to full pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0038] The motion estimation unit 42 calculates motion vectors for the prediction blocks (PUs) of video blocks in inter-frame predictive coding frames by comparing the position of the PU with the position of the prediction block of a reference frame selected from either a first reference frame list (list 0) or a second reference frame list (list 1), each list identifying one or more reference frames stored in the DPB 64. The motion estimation unit 42 sends the calculated motion vectors to the motion compensation unit 44, and then to the entropy coding unit 56.

[0039] Motion compensation performed by motion compensation unit 44 may involve retrieving or generating prediction blocks based on motion vectors determined by motion estimation unit 42. Upon receiving motion vectors from the PU for the current video block, motion compensation unit 44 can locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from DPB 64, and forward the prediction block to summer 50. Summer 50 then forms a residual video block of pixel difference values ​​by subtracting the pixel values ​​of the prediction block provided by motion compensation unit 44 from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include a luminance difference component or a chrominance difference component, or both. Motion compensation unit 44 may also generate syntax elements associated with video blocks of a video frame for use by video decoder 30 when decoding video blocks of a video frame. These syntax elements may include, for example, syntax elements defining motion vectors used to identify prediction blocks, any flags indicating prediction modes, or any other syntax information described herein. Note that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0040] In some implementations, the intra-BC unit 48 can generate vectors and extract prediction blocks in a manner similar to that described above in combination with the motion estimation unit 42 and the motion compensation unit 44, but the prediction blocks are in the same frame as the current block being encoded, and the vectors are called block vectors instead of motion vectors. Specifically, the intra-BC unit 48 can determine the intra-prediction mode to be used for encoding the current block. In some examples, the intra-BC unit 48 can, for example, encode the current block using various different intra-prediction modes during multiple individual encoding passes and test their performance through rate-distortion analysis. Next, the intra-BC unit 48 can select a suitable intra-prediction mode from the various tested intra-prediction modes and generate an intra-prediction mode indicator accordingly. For example, the intra-BC unit 48 can use rate-distortion analysis to calculate rate-distortion values ​​for the various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics from the tested modes as the suitable intra-prediction mode. Rate-distortion analysis typically determines the amount of distortion (or error) between the coded block and the original uncoded block used to generate the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. Intra-frame BC unit 48 can calculate the ratio for the distortion and rate of various coded blocks to determine which intra-frame prediction mode exhibits the best rate-distortion value for the block.

[0041] In other examples, the intra-frame BC unit 48 may use, in whole or in part, the motion estimation unit 42 and the motion compensation unit 44 to perform such a model for intra-frame BC prediction according to the implementation described herein. In either case, for intra-frame block copying, the predicted block may be a block that is considered to closely match the block to be encoded in terms of pixel differences, which can be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference measures, and the identifier of the predicted block may include calculating values ​​for sub-integer pixel locations.

[0042] Regardless of whether the prediction block originates from the same frame based on intra-frame prediction or from different frames based on inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded, thus generating a pixel difference value. The pixel difference value used to form the residual video block can include both luma component difference and chroma component difference.

[0043] As described above, the intra-prediction processing unit 46 can perform intra-prediction on the current video block as an alternative to inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44, or intra-block copy prediction performed by the intra-BC unit 48. Specifically, the intra-prediction processing unit 46 can determine the intra-prediction mode to be used for encoding the current block. To do this, the intra-prediction processing unit 46 can encode the current block using various different intra-prediction modes, for example, during multiple individual encoding passes, and the intra-prediction processing unit 46 (or, in some examples, the mode selection unit) can select a suitable intra-prediction mode from the tested intra-prediction modes for use. The intra-prediction processing unit 46 can provide information indicating the selected intra-prediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 can encode the information indicating the selected intra-prediction mode in the bitstream.

[0044] After prediction processing unit 41 determines the prediction block for the current video block via inter-frame prediction or intra-frame prediction, summer 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as discrete cosine transform (DCT) or a conceptually similar transform.

[0045] The transform processing unit 52 can send the obtained transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, the quantization unit 54 can then perform a scan of the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 can perform the scan.

[0046] After quantization, entropy coding unit 56 entropy-encodes the quantized transform coefficients into the video bitstream using, for example, context-adaptive variable-length codec (CAVLC), context-adaptive binary arithmetic codec (CABAC), syntax-based context-adaptive binary arithmetic codec (SBAC), probabilistic interval segmented entropy (PIPE) codec, or another entropy coding method or technique. The encoded bitstream is then transmitted to video decoder 30 or archived in storage device 32 for later transmission to or retrieval from video decoder 30. Entropy coding unit 56 can also entropy-encode motion vectors and other syntax elements used for the current video frame being encoded.

[0047] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate a reference block for predicting other video blocks. As noted above, motion compensation unit 44 can generate motion-compensated prediction blocks from one or more reference blocks of frames stored in DPB 64. Motion compensation unit 44 can also apply one or more interpolation filters to the prediction block to compute sub-integer pixel values ​​used in motion estimation.

[0048] The summer 62 adds the reconstructed residual block to the motion-compensated prediction block generated by the motion compensation unit 44 to produce a reference block for storage in the DPB 64. The reference block can then be used as a prediction block by the intra-frame BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 for inter-frame prediction of another video block in a subsequent video frame.

[0049] Figure 3 The diagram illustrates a block diagram of an exemplary video decoder 30 according to some implementations of this application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction processing unit 84, and an intra-frame BC unit 85. The video decoder 30 can perform operations generally combined with those described above with respect to the video encoder 20. Figure 2 The encoding process described is the inverse of the decoding process. For example, motion compensation unit 82 can generate prediction data based on motion vectors received from entropy decoding unit 80, while intra-frame prediction unit 84 can generate prediction data based on intra-frame prediction mode indicators received from entropy decoding unit 80.

[0050] In some examples, the units of the video decoder 30 can be assigned tasks to perform the implementation of this application. Similarly, in some examples, the implementation of this disclosure can be divided among one or more units of the video decoder 30. For example, the intra-frame BC unit 85 can perform the implementation of this application alone or in combination with other units of the video decoder 30 (such as the motion compensation unit 82, the intra-frame prediction processing unit 84, and the entropy decoding unit 80). In some examples, the video decoder 30 may not include the intra-frame BC unit 85, and the function of the intra-frame BC unit 85 can be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0051] Video data memory 79 can store video data, such as encoded video bitstreams, to be decoded by other components of video decoder 30. The video data stored in video data memory 79 can be obtained via wired or wireless network communication of video data or by accessing physical data storage media (e.g., flash drives or hard disks), such as from storage device 32 or from a local video source such as a camera. Video data memory 79 may include an encoded picture buffer (CPB) storing encoded video data from the encoded video bitstream. Decoded picture buffer (DPB) 92 of video decoder 30 stores reference video data used by video decoder 30 when decoding video data (e.g., in intra-frame or inter-frame prediction decoding modes). Video data memory 79 and DPB 92 can be formed of any of a wide variety of memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, video data memory 79 and DPB 92 are... Figure 3 The video data memory 79 and DPB 92 are depicted as two distinct components of the video decoder 30. However, it will be readily apparent to those skilled in the art that the video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, the video data memory 79 may be on-chip along with other components of the video decoder 30, or off-chip relative to those components.

[0052] During the decoding process, the video decoder 30 receives an encoded video bitstream, which represents video blocks of encoded video frames and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or video block level. The entropy decoding unit 80 of the video decoder 30 performs entropy decoding on the bitstream to generate quantized coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors and other syntax elements to the prediction processing unit 81.

[0053] When a video frame is encoded as an intra-predictive coding (I) frame or a prediction block for intra-coding in other types of frames, the intra-predictive processing unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video frame based on the intra-predictive mode transmitted by the signal and reference data from the previously decoded block of the current frame.

[0054] When a video frame is encoded as an inter-frame predictive coded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of these prediction blocks can be generated from reference frames within one of the reference frame lists. The video decoder 30 can construct the reference frame lists—list 0 and list 1—using the default construction technique based on the reference frames stored in the DPB 92.

[0055] In some examples, when decoding a video block according to the intra-frame BC mode described herein, the intra-frame BC unit 85 of the prediction processing unit 81 generates prediction blocks for the current video block based on the block vector and other syntax elements received from the entropy decoding unit 80. These prediction blocks may lie within the reconstructed region of the same image as the current video block, as defined by the video encoder 20.

[0056] Motion compensation unit 82 and / or intra-frame prediction (BC) unit 85 determine prediction information for video blocks in the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate prediction blocks for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction) used to decode video blocks in the video frame, the inter-frame prediction frame type (e.g., B or P), the construction information of one or more reference lists in the reference frame list for the frame, the motion vectors for video blocks decoded for each inter-frame prediction of the frame, the inter-frame prediction state for video blocks decoded for each inter-frame prediction of the frame, and other information for decoding video blocks in the current video frame.

[0057] Similarly, the intra-BC unit 85 can use some of the syntax elements (e.g., flags) in the received syntax elements to determine whether the current video block was predicted using the intra-BC module, which video blocks of the frame are in the reconstructed region and the build information that should be stored in the DPB 92, the block vector of the video block for each intra-BC prediction of the frame, the intra-BC prediction state of the video block for each intra-BC prediction of the frame, and other information for decoding the video blocks in the current video frame.

[0058] The motion compensation unit 82 may also perform interpolation using interpolation filters, such as those used by the video encoder 20 during encoding of video blocks, to calculate interpolated values ​​for sub-integer pixels of the reference block. In this case, the motion compensation unit 82 can determine the interpolation filters used by the video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction block.

[0059] The inverse quantization unit 86 uses the same quantization parameters calculated by the video encoder 20 for each video block in the video frame to determine the degree of quantization to inverse quantize the quantized transform coefficients provided in the bitstream and entropy decoded by the entropy decoding unit 80. The inverse transform processing unit 88 applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to these transform coefficients to reconstruct the residual block in the pixel domain.

[0060] After the motion compensation unit 82 or the intra-frame BC unit 85 generates a prediction block for the current video block based on vectors and other syntax elements, the summer 90 reconstructs the decoded video block for the current video block by summing the residual block from the inverse transform processing unit 88 and the corresponding prediction block generated by the motion compensation unit 82 and the intra-frame BC unit 85. A loop filter (not shown) can be placed between the summer 90 and the DPB 92 to further process the decoded video block. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92, or a separate memory device, can also store the decoded video for later use, such as... Figure 1 It is displayed on display devices such as display device 34.

[0061] In a typical video encoding and decoding process, a video sequence typically consists of an ordered set of frames or images. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luminance samples. SCb is a two-dimensional array of Cb chrominance samples. SCr is a two-dimensional array of Cr chrominance samples. In other cases, a frame may be monochrome and therefore consist of only a two-dimensional array of luminance samples.

[0062] like Figure 4A As shown, the video encoder 20 (or more specifically, the segmentation unit 45) generates a coded representation of the frame by first segmenting the frame into a set of Code Tree Units (CTUs). A video frame may include an integer number of CTUs ordered consecutively in a raster scan order from left to right and from top to bottom. Each CTU is the largest logical coding unit, and the width and height of the CTU are represented by signals in a sequence parameter set by the video encoder 20, such that all CTUs in the video sequence have the same size, which is one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that this application is not necessarily limited to a specific size. Figure 4BAs shown, each CTU may include a coding tree block (CTB) for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements for encoding the samples of the coding tree blocks. These syntax elements describe the properties of different types of units in the encoded pixel block, and how the video sequence can be reconstructed at the video decoder 30, including inter-frame or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome image or an image with three separate color planes, the CTU may include a single coding tree block and syntax elements for encoding the samples of the coding tree block. The coding tree block may be an NxN sample block.

[0063] To achieve better performance, the video encoder 20 can recursively perform tree splitting on the coding tree blocks of the CTU, such as binary tree splitting, ternary tree splitting, quadtree splitting, or any combination thereof, and divide the CTU into smaller coding units (CUs). Figure 4C As depicted, the 64x64 CTU 400 was first divided into four smaller CUs, each with a block size of 32x32. Of these four smaller CUs, each of CU 410 and CU 420 was further divided into four CUs with a block size of 16x16. Each of the two 16x16 CUs, 430 and CU 440, was further divided into four CUs with a block size of 8x8. Figure 4D The quadtree data structure is described, and it is illustrated as follows: Figure 4C The final result of the CTU 400 partitioning process depicted in the diagram is that each leaf node of the quadtree corresponds to a CU of a size ranging from 32x32 to 8x8. Like... Figure 4B As depicted in the CTU, each CU can include a code block (CB) of luma samples and two corresponding code blocks of chroma samples of the same size frame, as well as syntax elements for encoding the samples of the code blocks. In monochrome images or images with three separate color planes, the CU can include a single code block and a syntax structure for encoding the samples of the code block. Note that... Figure 4C and Figure 4D The quadtree partitioning depicted is for illustrative purposes only, and a CTU can be split into CUs based on quadtree / ternary / binary tree partitioning to accommodate varying local characteristics. In multi-type tree structures, a CTU is partitioned by a quadtree structure, and the leaf CUs of each quadtree can be further partitioned by binary and ternary tree structures. Figure 4E As shown, there are five types of segmentation: quadrilateral segmentation, horizontal binary segmentation, vertical binary segmentation, horizontal ternary segmentation, and vertical ternary segmentation.

[0064] In some implementations, the video encoder 20 may further segment the code blocks of the CU into one or more MxN prediction blocks (PBs). The prediction blocks are rectangular (square or non-square) sample blocks on which the same prediction (inter-frame or intra-frame) is applied. A prediction unit (PU) of the CU may include a prediction block for luma samples, two corresponding prediction blocks for chroma samples, and syntax elements for predicting the prediction blocks. In monochrome images or images with three separate color planes, a PU may include a single prediction block and syntax structures for predicting the prediction blocks. The video encoder 20 may generate predicted luma blocks, predicted Cb blocks, and predicted Cr blocks for each PU of the CU.

[0065] Video encoder 20 can use intra-frame prediction or inter-frame prediction to generate prediction blocks for the PU. If video encoder 20 uses intra-frame prediction to generate prediction blocks for the PU, then video encoder 20 can generate prediction blocks for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter-frame prediction to generate prediction blocks for the PU, then video encoder 20 can generate prediction blocks for the PU based on decoded samples of one or more frames different from the frame associated with the PU.

[0066] After the video encoder 20 generates a predicted luminance block, Cb block, and Cr block for one or more PUs of the CU, the video encoder 20 can generate a luminance residual block for the CU by subtracting its predicted luminance block from the original luminance code block of the CU, such that each sample in the luminance residual block of the CU indicates the difference between a luminance sample in one of the predicted luminance blocks of the CU and a corresponding sample in the original luminance code block of the CU. Similarly, the video encoder 20 can generate Cb residual blocks and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb code block of the CU, and each sample in the Cr residual block of the CU indicates the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr code block of the CU.

[0067] In addition, such as Figure 4CAs illustrated, the video encoder 20 can use quadtree partitioning to decompose the luminance residual block, Cb residual block, and Cr residual block of the CU into one or more luminance transform blocks, Cb transform blocks, and Cr transform blocks. A transform block is a rectangular (square or non-square) sample block on which the same transform is applied. A transform unit (TU) of the CU can include a transform block for luminance samples, two corresponding transform blocks for chrominance samples, and syntax elements for transforming the transform block samples. Therefore, each TU of the CU can be associated with a luminance transform block, a Cb transform block, and a Cr transform block. In some examples, the luminance transform block associated with a TU can be a sub-block of the CU's luminance residual block. A Cb transform block can be a sub-block of the CU's Cb residual block. A Cr transform block can be a sub-block of the CU's Cr residual block. In a monochrome image or an image with three separate color planes, a TU can include a single transform block and syntax structures for transforming the samples of the transform block.

[0068] Video encoder 20 can apply one or more transforms to the luminance transform block of TU to generate a luminance coefficient block for TU. The coefficient block can be a two-dimensional array of transform coefficients. The transform coefficients can be scalars. Video encoder 20 can apply one or more transforms to the Cb transform block of TU to generate a Cb coefficient block for TU. Video encoder 20 can apply one or more transforms to the Cr transform block of TU to generate a Cr coefficient block for TU.

[0069] After generating coefficient blocks (e.g., luminance coefficient blocks, Cb coefficient blocks, or Cr coefficient blocks), video encoder 20 can quantize the coefficient blocks. Quantization generally refers to a process in which transform coefficients are quantized to reduce the amount of data used to represent the transform coefficients as much as possible, providing further compression. After quantizing the coefficient blocks, video encoder 20 can entropy encode the syntax elements indicating the quantized transform coefficients. For example, video encoder 20 can perform context-adaptive binary arithmetic encoding / decoding (CABAC) on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 can output a bitstream comprising bit sequences that form a representation of coded frames and associated data; the bitstream is stored in storage device 32 or transmitted to target device 14.

[0070] After receiving the bitstream generated by the video encoder 20, the video decoder 30 can parse the bitstream to obtain syntax elements. The video decoder 30 can reconstruct frames of video data, at least in part, based on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 can perform an inverse transform on the coefficient block associated with the TU of the current CU to reconstruct the residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the code blocks of the current CU by adding samples of the prediction block of the PU used for the current CU to the corresponding samples of the transform block of the TU of the current CU. After reconstructing the code blocks of each CU of the frame, the video decoder 30 can reconstruct the frame.

[0071] As noted above, video coding primarily uses two modes (i.e., intra-frame prediction, or intra-prediction, and inter-frame prediction, or inter-prediction) to achieve video compression. Palette-based coding and decoding is another coding and decoding scheme adopted by many video coding and decoding standards. In palette-based coding and decoding, which may be particularly suitable for encoding and decoding screen-generated content, the video codec (e.g., video encoder 20 or video decoder 30) forms a color palette table representing a given block of video data. The palette table includes the most dominant (e.g., frequently used) pixel values ​​in a given block. Pixel values ​​that are not frequently represented in the video data of a given block are not included in the palette table, or are included in the palette table as escape colors.

[0072] Each entry in the palette table includes an index from the palette table for the corresponding pixel value. The palette indexes used for samples within a block can be encoded or decoded to indicate which entry from the palette table should be used to predict or reconstruct which sample. The palette pattern begins with the process of generating a palette predictor for the first block of a group of video blocks, such as pictures, stripes, tiles, or others. As explained below, palette predictors for subsequent video blocks are typically generated by updating previously used palette predictors. For illustrative purposes, it is assumed that the palette predictor is defined at the picture level. In other words, a picture may include multiple code blocks, each with its own palette table, but there is only one palette predictor for the entire picture.

[0073] To reduce the number of bits required to signal palette entries in the video bitstream, the video decoder can utilize a palette predictor to determine new palette entries in the palette table used to reconstruct the video block. For example, the palette predictor can include palette entries from previously used palette tables, or even be initialized using the most recently used palette table by including all entries from the most recently used palette table. In some implementations, the palette predictor can include fewer entries than all entries from the most recently used palette table, and then incorporate some entries from other previously used palette tables. The palette predictor can have the same size as the palette table used to encode and decode different blocks, or it can be larger or smaller than the palette table used to encode and decode different blocks. In one example, the palette predictor is implemented as a first-in-first-out (FIFO) table containing 54 palette entries.

[0074] To generate a palette table for video data blocks from palette predictors, the video decoder can receive a 1-bit flag for each entry of the palette predictor from the encoded video bitstream. The 1-bit flag can have a first value (e.g., binary 1) indicating whether the associated entry of the palette predictor should be included in the palette table, or a second value (e.g., binary 0) indicating whether the associated entry of the palette predictor should not be included in the palette table. If the size of the palette predictor is larger than the palette table for video data blocks, the video decoder can stop receiving more flags once the maximum size of the palette table is reached.

[0075] In some implementations, certain entries in the palette table can be signaled directly into the encoded video bitstream, rather than determined using a palette predictor. For such entries, the video decoder can receive from the encoded video bitstream three separate m-bit values ​​indicating the pixel values ​​associated with the entry's luma and two chroma components, where m represents the bit depth of the video data. Those palette entries derived from the palette predictor require only a 1-bit flag, compared to the multiple m-bit values ​​needed to signal the palette entries directly. Therefore, signaling some or all palette entries using a palette predictor can significantly reduce the number of bits required to signal new palette table entries, thereby improving the overall encoding / decoding efficiency of palette pattern encoding / decoding.

[0076] In many cases, the palette predictor for a block is determined based on the palette table used to encode or decode one or more previously encoded blocks. However, when encoding or decoding the first coding tree unit in a picture, strip, or tile, the palette table for previously encoded blocks may be unavailable. Therefore, the palette predictor cannot be generated using entries from previously used palette tables. In such cases, a palette predictor initializer sequence can be signaled in the Sequence Parameter Set (SPS) and / or Picture Parameter Set (PPS), which are values ​​used to generate the palette predictor when the previously used palette table is unavailable. The SPS typically refers to the syntax structure of the syntax elements applicable to a series of consecutively encoded video pictures called a Coded Video Sequence (CVS), as determined by the contents of the syntax elements found in the PPS referenced by the syntax elements found in the syntax elements found in each strip segment header. The PPS typically refers to the syntax structure of the syntax elements applicable to one or more individual pictures within a CVS, as determined by the syntax elements found in the syntax elements found in the syntax elements found in each strip segment header. Therefore, SPS is generally considered to be a higher-level grammatical structure than PPS, meaning that the grammatical elements included in SPS usually change less frequently compared to those included in PPS, and are applicable to a larger portion of video data.

[0077] Figure 5 The figure illustrates a process 500 of deriving chroma samples 502 from luminance samples 504 of a code block 506 of a video frame from a bitstream, according to some embodiments. The code block 506 of the video frame comprises multiple pixels, and each pixel is formed by multiple color elements (e.g., blue, green, and red). In video encoding and decoding, the brightness and color information of the multiple pixels are represented by multiple luminance samples 504 and multiple chroma samples 502, respectively. Each pixel among the multiple pixels uniquely corresponds to a single corresponding luminance sample 504. Each chroma sample 502 corresponds to a corresponding set of luminance samples 504 according to a subsampling scheme. Each luminance sample has a luminance component Y', and each chroma sample 502 has a blue differential chroma component Cb and a red differential chroma component Cr. The subsampling scheme (Y':Cb:Cr) of the luminance and chroma components has a three-part ratio, such as 4:1:1, 4:2:0, 4:2:2, 4:4:4, and 4:4:0. Figure 5 In the video frame, the luminance sample 504 and chrominance sample 502 follow a subsampling scheme with a three-part ratio of 4:1:1.

[0078] In some embodiments, the code block 506 of the video frame includes 2M x 2N pixels, corresponding to 2M luminance samples across the width and 2N luminance samples across the height. Optionally, M and N may be equal to or unequal to each other. Figure 5The example sampling scheme for the luminance and chrominance components (Y':Cb:Cr = 4:1:1) encodes the luminance sample 504 of the video frame at a resolution of 2M x 2N, while the chrominance sample 502 is encoded at a smaller resolution of M x N. In practice, the chrominance sample 502 can be encoded at different chrominance resolutions such as 2M x 2N (e.g., 4:4:4 full sampling), 2M x N (e.g., 4:4:0 subsampling), M x 2N (e.g., 4:2:2 subsampling), and 1 / 2M x 2N (e.g., 4:1:1 subsampling).

[0079] In some embodiments, chroma samples 502 and luminance samples 504 of the same video frame are encoded separately in video encoder 20, transferred from video encoder 20 to video decoder 30, and decoded separately in video decoder 30. In some cases, luminance sample 504 can be used to refine the decoded chroma sample 502. Alternatively, in some embodiments, luminance sample 504 of code block 506 of a video frame is encoded in video encoder 20 and provided to video decoder 30 in the absence of chroma sample 502 of the same code block 506 of the video frame. Luminance sample 504 of code block 506 is reconstructed in video decoder 30, and chroma sample 502 is derived from the reconstructed luminance sample 504 of code block 506. Assuming that luminance sample 504 has a first luminance resolution—which is greater than the second chroma resolution of chroma sample 502—luminance sample 504 of code block 506 is converted into a set of downsampled luminance samples 508 having a third luminance resolution equal to the second chroma resolution according to a predefined luminance interpolation scheme 510. The set of downsampled luminance samples 508 is further converted into chrominance samples 502 of code block 506 of the video frame according to the linear mapping model 512. The linear mapping model is expressed as follows:

[0080] Y = αX + β (1)

[0081] Where X and Y correspond to the luminance value of the downsampled luminance sample 508 and the chrominance value of the corresponding chrominance sample 502, respectively, and α and β are two linear coefficients of the linear mapping model 512.

[0082] Specifically, the luminance value (DLuma) 508 of each downsampled luminance sample 508 is derived from the luminance values ​​(Luma) 504 of a certain number of adjacent luminance samples according to the following luminance interpolation scheme 510:

[0083]

[0084] Where k represents the number of adjacent luminance samples, and f(i) represents the filter coefficients used for each of the k adjacent luminance samples 504. The linear interpolation scheme 510 can be combined with the linear mapping model 512 to determine the luminance-chrominance cross-component filter model 520 as follows:

[0085]

[0086] Each chroma sample 502 can be derived directly from k adjacent luminance samples 504 using a cross-component linear model, i.e., using the cross-component filter model 520 described above. In some embodiments, given two linear coefficients α and β for each code block 506, all chroma samples 502 of code block 506 are derived directly from luminance samples 504 located in the same corresponding code block 506 using the cross-component filter model 520.

[0087] In some embodiments, the luminance samples 504 and chrominance samples 502 of the video frame follow a subsampling scheme in which, on average, every four luminance samples 504 correspond to one chrominance sample 504 having a blue differential chrominance component Cb and a red differential chrominance component Cr. Alternatively, the first luminance resolution of the luminance samples 504 is four times the second chrominance resolution of the chrominance samples 502. On average, every four luminance samples 504 are combined into one downsampled luminance sample 508 in a set of luminance samples 508 having a third luminance resolution equal to the second chrominance resolution. In some embodiments, for each luminance sample 508, more than four adjacent luminance samples 504 are applied to derive the corresponding chrominance sample 502. For a subset of luminance samples 504, each luminance sample 504 is applied more than once to generate more than one downsampled luminance sample 508 and / or more than one chrominance sample 502. Further details regarding downsampling multiple luminance samples 504 into luminance sample 508 and / or chrominance sample 502 based on a predefined interpolation scheme 510 are provided below. Figures 6A-6D Let's have a discussion.

[0088] In some embodiments, the linear mapping model 512 applied to convert luminance sample 508 into chrominance sample 502 is derived using a max-min method. Specifically, the maximum downsampled luminance sample is identified from the set of downsampled luminance samples, and the minimum downsampled luminance sample is identified from the set of downsampled luminance samples. The maximum chrominance sample corresponds to the maximum downsampled luminance sample, and the minimum chrominance sample corresponds to the minimum downsampled luminance sample. The maximum and minimum chrominance samples, together with the maximum and minimum downsampled luminance samples, are applied to determine the linear mapping model 512, i.e., to determine the two coefficients α and β in equation (1). After deriving the linear mapping model 512, the video codec applies the linear mapping model 512 to the luminance sample 508 downsampled from luminance sample 504 to generate the corresponding chrominance sample 502 in code block 506.

[0089] The code blocks of the video frame are reconstructed according to the ordered sequence. In some cases, code block 506 is not in the first row or first column of the video frame, and a selected group of pixels has already been reconstructed while code block 506 is being processed. For example, the selected group of pixels includes the pixels immediately above and to the left of the pixels in code block 506. The selected groups of chroma samples 516, luminance samples 518, and downsampled luminance samples 514 have been reconstructed for the selected groups of pixels and can be used, for example, to derive a linear mapping model 512 based on a minimax method. Chroma samples 516, luminance samples 518, and downsampled luminance samples 514 are located outside and immediately adjacent to the boundary of code block 506. Specifically, in some embodiments, the video codec searches for already encoded downsampled luminance sample groups (e.g., the selected group of downsampled luminance samples 514 corresponding to the top adjacent luminance sample and the left adjacent luminance sample) to identify the maximum downsampled luminance sample 514A-1 and the minimum downsampled luminance sample 514B-1. The video codec then identifies (e.g., from a selected group of chroma samples 516, including the top adjacent chroma sample and the left adjacent chroma sample) the previously encoded chroma samples 516A-1 and 516B-1 corresponding to the maximum and minimum downsampled luminance samples 514A-1 and 514B-1. Thus, a linear mapping model 512 can be derived based on these associated downsampled luminance samples (514A-1 and 514B-1) and chroma samples (516A-1 and 516B-1).

[0090] Alternatively, in some embodiments, the video codec searches a group of luminance samples (e.g., a selected group of luminance samples 518 including the top and left adjacent luminance samples) to identify (i) a luminance sample 518A with the maximum luminance value and (ii) a luminance sample 518B with the minimum luminance value within the luminance sample group, without performing downsampling on the selected luminance sample group to identify the maximum and minimum luminance samples 518A and 518B. The video codec then performs downsampling in the regions associated with the maximum and minimum luminance samples 518A and 518B (e.g., in regions with six samples, using a weighted averaging scheme known in the art, including six-tap downsampling, etc.) to generate a downsampled luminance sample 514A-2 as the maximum luminance sample (which may or may not be exactly the maximum downsampled luminance sample 514A-1) and a downsampled luminance sample 514B-2 as the minimum luminance sample (which may or may not be exactly the minimum downsampled luminance sample 514B-1). Then, the video codec identifies chroma samples 516A-2 and 516B-2 corresponding to the downsampled luminance samples 514A-2 and 514B-2 (e.g., in the chroma sample group 516 including the top and left adjacent chroma samples). In this way, a linear mapping model 512 can be derived based on the downsampled luminance samples (514A-2 and 514B-2) and the chroma samples (516A-2 and 516B-2).

[0091] Alternatively, in some embodiments, the video codec searches a group of chroma samples (e.g., a group 516 of chroma samples including the top and left adjacent chroma samples) to identify the maximum and minimum chroma samples 516A-3 and 516B-3 (e.g., chroma samples with the maximum and minimum chroma values, respectively). The video codec then identifies downsampled luminance samples 514A-3 and 514B-3 corresponding to the maximum and minimum chroma samples 516A-3 and 516B-3 (e.g., in the group of downsampled luminance samples 514 including the top and left adjacent luminance samples). Thus, a linear mapping model 512 is derived based on the downsampled luminance samples (514A-3 and 514B-3) and the chroma samples (516A-3 and 516B-3).

[0092] Alternatively, in some embodiments, the video codec searches a group of downsampled luminance samples (e.g., a selected group of downsampled luminance samples 514) to identify a predefined number (e.g., two) of downsampled luminance samples (e.g., 514A-4 and 514A-5) with the maximum luminance value and a predefined number (e.g., two) of downsampled luminance samples (e.g., 514B-4 and 514B-5) with the minimum luminance value. The video codec then identifies chroma samples (e.g., 516A-4, 516A-5, 516B-4, and 516B-5) in a selected group of chroma samples 516, each chroma sample corresponding to one of the groups of the maximum downsampled luminance samples 514A-4 and 514A-5 and the groups of the minimum downsampled luminance samples 514B-4 and 514B-5, respectively. Next, the video codec performs a weighted average of values ​​(e.g., chroma or luminance values) within each identified chroma and luminance sample group to generate (e.g., for chroma samples 516A-4 and 516A-5) the maximum average chroma value, (e.g., for chroma samples 516B-4 and 516B-5) the minimum average chroma value, (e.g., for luminance samples 514A-4 and 514A-5) the maximum average downsampled luminance value, and (e.g., for luminance samples 514B-4 and 514B-5) the minimum average downsampled luminance value. Thus, a linear mapping model 512 can be derived based on the downsampled luminance samples (514A-4, 514A-5, 514B-4, and 514B-5) and the chroma samples (516A-4, 516A-5, 516B-4, and 516B-5).

[0093] Furthermore, in some embodiments, a linear mapping model 512 is derived by creating linear fitting curves for multiple downsampled luminance samples (e.g., in a selected group of downsampled luminance samples 514) and multiple corresponding chrominance samples (e.g., in a selected group of chrominance samples 516). Such curve fitting has a bias that optionally satisfies an error tolerance. The multiple downsampled luminance samples comprise a predefined number (e.g., greater than 10) of downsampled luminance samples corresponding to a predefined number of chrominance samples. In some embodiments, for the purpose of deriving the linear mapping model 512, the downsampled luminance samples are randomly selected from groups of adjacent downsampled luminance samples 514.

[0094] Figures 6A-6D Here are four example code blocks 506A-506D according to some embodiments, each of which includes multiple luminance samples 504 to be converted into multiple chroma samples 502. Each cross (“x”) represents the position of a luminance sample 504, and each circle (“o”) represents the position of a chroma sample 502 or a downsampled luminance sample 508. Each circled cross... This indicates the overlapping position of chroma sample 502, luminance sample 504, and downsampled luminance sample 508. Each code block 506 is marked with a block boundary 602. Each code block 506 follows the corresponding luminance and chroma component quantum sampling scheme (Y':Cb:Cr), and uses the corresponding luminance interpolation scheme 510 to downsample the luminance sample 504 of the corresponding code block 506 into the corresponding downsampled luminance sample 508.

[0095] In some embodiments of this application, for each code block 506, particularly when the downsampled luminance sample 508 is adjacent to the block boundary 602, a luminance interpolation scheme 510 is used to derive all downsampled luminance samples 508 from adjacent luminance samples 504 located in the same corresponding code block 506. Each downsampled luminance sample 508 always overlaps with the corresponding chrominance sample 502, and the chrominance sample 502 can therefore be derived from the downsampled luminance sample 508 based on a linear mapping model 512.

[0096] In some embodiments, the cross-component filter model 520 is a combination of the luminance interpolation scheme 510 and the linear mapping model 512. Given two linear coefficients α and β of the linear mapping model 512 for each code block 506 and the filter coefficients f(i) of the luminance interpolation scheme 510, particularly when the downsampled luminance sample 508 is adjacent to the block boundary 602, the chrominance sample 502 of code block 506 is directly derived from the luminance sample 504 located in the same corresponding code block 506 using the cross-component filter model 520.

[0097] In some embodiments, code block 506 is the first code block of a video frame, and no other code blocks 506 are reconstructed before code block 506. Such a code block 506 may optionally be located at the top left corner of the video frame. The linear mapping model 512 is determined independently of the code blocks in the video frame (e.g., based on previous video frames). Alternatively, in some embodiments, code block 506 is not the first code block of the video frame, and one or more other code blocks 506 have been reconstructed before code block 506. For example, code block 506 is located in the middle of the video frame, and the code blocks immediately above or to the left of code block 506 have been processed and can be used to determine the linear mapping model 512 or reconstruct the luminance or chrominance samples in code block 506. In some cases, code block 506 is adjacent to the top boundary of the video frame, and at least one code block immediately to the left of code block 506 has been processed and can be used to determine the linear mapping model 512 or reconstruct the luminance or chrominance samples in code block 506. In some cases, code block 506 is immediately adjacent to the left boundary of the video frame, and at least one code block immediately above code block 506 has been processed and can be used to determine the linear mapping model 512 or reconstruct the luminance or chrominance samples in code block 506. Regardless of the position of code block 506, the code blocks to the right or below code block 506 are not processed and cannot be used to determine the linear mapping model 512 for code block 506 or reconstruct the luminance or chrominance samples in code block 506.

[0098] In some embodiments, the same predefined luminance interpolation scheme 510 or cross component filter model 520 is applied to generate all downsampled luminance samples 508 or chrominance samples 502 across the entire code block 506. Alternatively, in some embodiments, the predefined luminance interpolation scheme 510 varies with the position of the downsampled luminance sample 508 or the position of the boundary pixel associated with the downsampled luminance sample 508. Similarly, in some embodiments, the cross component filter model 520 varies with the position of the chrominance sample 502 or the position of the boundary pixel associated with the chrominance sample 502. For example, each of the predefined luminance interpolation scheme 510 and the cross component filter model 520 corresponds to one of a cross, block, or T-shaped filter configuration.

[0099] Reference Figure 6ACode block 506A utilizes a bitstream and includes multiple luminance samples 504 reconstructed from the bitstream. Multiple corresponding chrominance samples 502 are derived from the multiple luminance samples 504. In code block 506A, the subsampling scheme (Y':Cb:Cr) of the luminance and chrominance components 504 and 502 has a three-part ratio of 4:2:0. Each chrominance sample 502 is combined from six adjacent luminance samples 504 according to a cross-component filter model 520. The six adjacent luminance samples 504 are located at the upper left, upper right, lower left, lower right, and lower right corners of the corresponding chrominance sample 502. Alternatively, the six adjacent luminance samples 504 of each chrominance sample 502 are converted into downsampled luminance samples 508 overlapping with the corresponding chrominance sample 502 according to a predefined luminance interpolation scheme 510, and then converted back into the corresponding chrominance sample 502 according to a linear mapping model 512. In one example, the first chromaticity sample 502A and its six corresponding adjacent luminance samples 504A-1 to 504A-6 are completely enclosed within the block boundary 602, and the first chromaticity sample 502A can be derived from the adjacent luminance samples 504A-1 to 504A-6 based on the cross component filter model 520 or a combination of luminance interpolation scheme 510 and linear mapping model 512.

[0100] Conversely, the second chroma sample 502B is immediately adjacent to the left block boundary 602A of code block 506A and corresponds to six adjacent luma samples 504B-1 to 504B-6 according to the cross-component filter model 520. In some cases, the left block boundary 602A of code block 506 overlaps with the left boundary of the video frame, and the two adjacent luma samples 504B-1 and 504B-6 are unavailable within this or any other code block. The luma samples 504B-1 and 504B-6 that are outside code block 506 and unavailable can be copied from luma samples 504B-2 and 504B-5 that are immediately adjacent to luma samples 504B-1 and 504B-6, respectively. In one example, luma samples 504B-1 and 504B-6 are copied from 504B-4 and 504B-3, which are symmetrical to luma samples 504B-1 and 504B-6 about chroma sample 502B. Alternatively, in some cases, the left block boundary 602A of code block 506 does not overlap with the left boundary of the video frame. Two adjacent luminance samples 504B-1 and 504B-6 are not available in code block 506 itself; however, they can be provided by another adjacent code block that has been reconstructed prior to code block 506. Luminance samples 504B-1 and 504B-6 located outside code block 506 can still be copied from luminance samples 504B-2 to 504B-5 within code block 506 without loading luminance samples 504B-1 and 504B-6 from any other code block. In these ways, the second chroma sample 502B and / or the second luminance sample 508B are further derived from adjacent luminance samples 504B-1 to 504B-6, wherein the two luminance samples 504B-1 and 504B-6 are optionally allocated from two different luminance samples (e.g., 504B-2 and 504B-5) included in code block 506A.

[0101] In other words, when a bitstream capable of deriving multiple luminance samples for multiple pixels in a video frame is obtained, the multiple pixels belong to code block 506A and include boundary pixels located inside code block 506A and immediately adjacent to the boundary 602 of code block 506A. A boundary pixel (e.g., corresponding to luminance sample 504B-2) corresponds to one or more unavailable neighboring pixels (e.g., corresponding to luminance sample 504B-1). Each of the one or more neighboring pixels is outside code block 506A. In some embodiments, one or more neighboring pixels of a boundary pixel are outside the video frame or image stripe. In some embodiments, one or more neighboring pixels of a boundary pixel have not yet been decoded and will be decoded after code block 506. In some embodiments, the cross-component filter model 520 is limited to code block 506, and any neighboring pixels outside code block 506 are considered unavailable. Luminance samples 504B-2 and 504B-5 corresponding to the boundary pixels are assigned to luminance samples 504B-1 and 504B-6 corresponding to each of the one or more neighboring pixels, respectively. Boundary luminance sample 508B is determined based at least on luminance samples 504B-1 to 504B-6 of the boundary pixel and one or more neighboring pixels according to a predefined luminance interpolation scheme 510, and is used to determine boundary chrominance sample 502B according to linear mapping model 512. Alternatively, boundary chrominance sample 502B is determined from luminance samples 504B-1 to 504B-6 of the boundary pixel and one or more neighboring pixels according to cross-component filter model 520.

[0102] Reference Figure 6B In code block 506B, the subsampling scheme (Y':Cb:Cr) of the luminance and chrominance components 504 and 502 has a three-part ratio of 4:2:0. Each chrominance sample 502 is combined from five adjacent luminance samples 504 according to a cross-component filter model 520 with a cross-filter shape. One of these five adjacent luminance samples 504 overlaps with the chrominance sample 502, while the other four adjacent luminance samples 504 are located directly above, below, to the left, and to the right of the corresponding chrominance sample 502. Alternatively, the five adjacent luminance samples 504 of each chrominance sample 502 are transformed into a downsampled luminance sample 508 that overlaps with the corresponding chrominance sample 502 according to a predefined luminance interpolation scheme 510, and then transformed into the corresponding chrominance sample 502 according to a linear mapping model 512. In one example, the third chromaticity sample 502C and its five corresponding adjacent luminance samples 504C-1 to 504C-5 are completely enclosed within the block boundary 602, and the third chromaticity sample 502C can be derived from the adjacent luminance samples 504C-1 to 504C-5 based on the cross component filter model 520 or a combination of luminance interpolation scheme 510 and linear mapping model 512.

[0103] The fourth chroma sample 502D is immediately adjacent to the left block boundary 602A of code block 506B and corresponds to five adjacent luma samples 504D-1 to 504D-5 according to the cross-component filter model 520. In some cases, the left block boundary 602A of code block 506B overlaps with the left boundary of the video frame, and the adjacent luma sample 504D-5 is not available in this or any other code block. Luma sample 504D-5 is copied from luma sample 504D-1, which is immediately adjacent to luma sample 504D-5, or from luma sample 504D-3, which is symmetrical to luma sample 504D-5 about the center luma sample 504D-1. In some cases, the left block boundary 602A of code block 506 does not overlap with the left boundary of the video frame. The adjacent luma sample 504D-5 is not available in code block 506 itself; however, it can be provided by another adjacent code block that has been reconstructed before code block 506B. Luminance sample 504D-5 located outside code block 506 can still be copied from luminance samples 504D-1 or 504D-3 located within code block 506 without loading luminance sample 504D-5 from any other code block. In these ways, fourth chroma sample 502D and / or fourth luminance sample 508D are derived from adjacent luminance samples 504D-1 to 504D-5, wherein luminance sample 504D-5 is optionally assigned based on luminance sample 504D-1 or 504D-3 within code block 506B.

[0104] The fifth chroma sample 502E is adjacent to the left block boundary 602A and top block boundary 602B of code block 506B, and corresponds to five adjacent luma samples 504E-1 to 504E-5 according to the cross-component filter model 520. In some embodiments, code block 506B is the first code block in the video frame, and the two adjacent luma samples 504E-2 and 504E-5 are not available in this or any other code block. Luma samples 504E-2 and 504E-5 are copied from luma sample 504E-1. In some cases, code block 506B is not located at the top left corner of the video frame, and its left and top adjacent code blocks are available. Adjacent luma samples 504E-2 and 504E-5 are not available within code block 506B itself; however, they can be provided by another adjacent code block that has been reconstructed before code block 506B. Alternatively, luminance samples 504E-2 and 504E-5 can still be copied from luminance sample 504E-1 located within code block 506 without loading these luminance samples from any other code block. In these ways, the fifth chroma sample 502E and / or the fifth luminance sample 508E are derived from adjacent luminance samples 504E-1 to 504E-5, wherein luminance samples 504E-2 and 504E-5 are optionally assigned based on another luminance sample 504E-1 at the block boundary 602 of the adjacent code block 506B.

[0105] Reference Figure 6CIn code block 506C, the subsampling scheme (Y':Cb:Cr) of the luminance and chrominance components 504 and 502 has a three-part ratio of 4:2:2. Each chrominance sample 502 is combined from six adjacent luminance samples 504 according to a cross-component filter model 520 with a 2x3 filter shape. One of the six adjacent luminance samples 504 overlaps with the chrominance sample 502, while the other five adjacent luminance samples are located to the left, right, lower left corner, directly below, and lower right corner of the corresponding chrominance sample 502. These six adjacent luminance samples 504 of each chrominance sample 502 are transformed into downsampled luminance samples 508 that overlap with the corresponding chrominance sample 502 according to a predefined luminance interpolation scheme 510, and then transformed into the corresponding chrominance sample 502 according to a linear mapping model 512. In one example, the sixth chroma sample 502F and its six corresponding adjacent luminance samples 504F-1 to 504F-6 are completely enclosed within the block boundary 602, and the sixth chroma sample 502F can be derived from the adjacent luminance samples 504F-1 to 504F-6 based on the cross component filter model 520 or a combination of luminance interpolation scheme 510 and linear mapping model 512.

[0106] The seventh chroma sample 502G is adjacent to the left block boundary 602A and the bottom block boundary 602C of code block 506C, and corresponds to six adjacent luma samples 504G-1 to 504G-6 according to the cross-component filter model 520. In some embodiments, code block 506C is located at the lower left corner of the video frame, and the left and bottom block boundaries 602A and 602C overlap with the left and bottom block boundaries of the video frame. In some embodiments, code block 506C is adjacent to the left boundary of the video frame, so luma samples 504G-1 and 504G-6 are absent, and luma samples 504G-4 and 504G-5 have not yet been decoded. At the lower left corner or left boundary of the video frame, the four adjacent luma samples 504G-1, 504G-4, 504G-5, and 504G-6 are unavailable in this or any other code block. Each of these luminance samples is copied from luminance sample 504G-2 or 504G-3. For example, luminance sample 504G-4 is copied from luminance sample 504G-3, and luminance samples 504G-1, 504G-5, and 504G-6 are copied from luminance sample 504G-2. Specifically, luminance samples 504G-1 and 504G-5 are copied from luminance sample 504G-2 in the same way that luminance sample 504G-4 is copied from luminance sample 504G-3, and luminance sample 504B-1 is copied from luminance sample 504B-2. Then, luminance sample 504G-6 is copied from either luminance sample 504G-1 or 504G-5. In some cases, code block 506C is not located at the lower left corner of the video frame, and its left and bottom adjacent code blocks are available. Adjacent luminance samples 504G-1, 504G-4, 504G-5, and 504G-6 are not available in code block 506 itself; however, they can be provided by another adjacent code block that has been reconstructed prior to code block 506. Alternatively, luminance samples 504G-1, 504G-4, 504G-5, and 504G-6 can still be copied from luminance samples 504G-2 to 504G-3 located within code block 506 without loading these luminance samples from any other code block. In these ways, the seventh chroma sample 502G and / or the seventh luminance sample 508G are derived from adjacent luminance samples 504G-1 to 504G-6, wherein luminance samples 504G-1, 504G-4, 504G-5, and 504G-6 are optionally allocated based on two other luminance samples 504G-2 and 504G-3 at the block boundary of adjacent code block 506C.

[0107] Reference Figure 6DIn code block 506D, the subsampling scheme (Y':Cb:Cr) of the luminance and chrominance components 504 and 502 has a three-part ratio of 4:4:4. Each chrominance sample 502 is combined from six adjacent luminance samples 504 according to a cross-component filter model 520 with a 2x3 filter shape. One of the six adjacent luminance samples 504 overlaps with the chrominance sample 502, while the other five adjacent luminance samples are located to the left, right, lower left corner, directly below, and lower right corner of the corresponding chrominance sample 502. These six adjacent luminance samples 504 of each chrominance sample 502 are transformed into downsampled luminance samples 508 that overlap with the corresponding chrominance sample 502 according to a predefined luminance interpolation scheme 510, and then transformed into the corresponding chrominance sample 502 according to a linear mapping model 512.

[0108] The eighth chroma sample 502H is adjacent to the bottom block boundary 602C and the right block boundary 602D of code block 506D, and corresponds to six adjacent luma samples 504H-1 to 504H-6 according to the cross-component filter model 520. In some embodiments, code block 506D is located at the lower right corner of the video frame, and the right and bottom block boundaries 602D and 602C overlap with the right and bottom block boundaries of the video frame. The four adjacent luma samples 504H-3 to 504H-6 are unavailable in this or any other code block. Alternatively, in some cases, code block 506D is not located at the lower left corner of the video frame, and its right and bottom adjacent code blocks remain unavailable because these code blocks have not yet been reconstructed. Each of these luminance samples 504H-3 to 504H-6 is copied from luminance sample 504H-1 or 504H-2. For example, luminance sample 504H-6 is copied from luminance sample 504H-1, and luminance samples 504H-3 to 504H-5 are copied from luminance sample 504H-2. Specifically, luminance samples 504H-3 and 504H-5 are copied from luminance sample 504H-2 in the same way that luminance sample 504H-6 is copied from luminance sample 504H-1, and luminance sample 504B-1 is copied from luminance sample 504B-2. Then, luminance sample 504H-4 is copied from either luminance sample 504H-3 or 504H-5. In these ways, the eighth chromaticity sample 502H and / or the eighth luminance sample 508H are derived from the adjacent luminance samples 504H-1 to 504H-6, wherein the luminance samples 504H-3 to 504H-6 are optionally allocated based on two other luminance samples 504H-1 and 504H-2 of the block boundary 602 of the adjacent code block 506D.

[0109] According to the embodiments described above in this disclosure, sample padding is applied to luminance samples to generate those unavailable samples used in the downsampling process. It is understood that the same filter can be used to generate downsampled luminance samples regardless of their location through the padding process. Note that various padding methods can be used, such as repeated padding (e.g., ...). Figure 6C Copying the luminance sample 504G-2 to 504G-1, 504G-5, and 504G-6) or mirror filling (e.g.) Figure 6B The brightness sample 504D-3 is copied to 504D-5. The most straightforward padding is repeated padding, which directly uses the most recent available sample values ​​as the values ​​for the samples to be filled.

[0110] Based on the above Figures 6A-6D The description of adjacent pixel padding in the text indicates that adjacent pixels of the boundary pixels of the current coding block can be padded using boundary pixels. (Refer to...) Figure 6E Each of the reference luminance samples pY[-1,-1], pY[-1,0], ..., pY[-1,2N-1] is copied from the luminance samples pY[0,-1], pY[0,0], ..., pY[0,2N-1] within the current coding block if it is unavailable. Note that the sample to be copied is not necessarily a sample within the current coding block; for example, pY[0,-1]. Since pY[0,-1] is considered unavailable, it is copied from the luminance sample pY[0,0]. The two-dimensional array pY[x][y] represents the luminance samples. It should also be noted that adjacent pixels include not only pixels on the opposite side of the boundary pixel relative to the boundary pixel, but also adjacent pixels at the corners of the coding block opposite the boundary pixel. By padding the samples, a 6-tap downsampling filter can be applied to generate the downsampling sample pDsY[M][N]. Using the scheme proposed in this disclosure, all downsampling filters used to generate all downsampling samples can be the same.

[0111] Figure 7 This is a flowchart of a video data decoding method 700 implemented at an electronic device according to some embodiments. Multiple luminance samples 504, including multiple pixels in a video frame, are obtained (702) from the bitstream. The multiple pixels include boundary pixels located inside code block 506 and immediately adjacent to the boundary 602 of code block 506. The electronic device determines (704) that one or more neighboring pixels of the boundary pixels are unavailable. Each of the one or more neighboring pixels is outside code block 506. The luminance samples corresponding to the boundary pixels (e.g., ...) Figure 6C 504G-2) is assigned (706) to the corresponding luminance sample of each of one or more neighboring pixels (e.g., Figure 6C(504G-6 in the example). The electronic device determines the boundary chromaticity sample (e.g., based on the cross component filter model 120 at least based on the luminance samples of the boundary pixel and one or more neighboring pixels (e.g., 502G-1 to 502G-6). Figure 6C In the case of 502G). Specifically, the electronic device determines (710) boundary luminance samples (e.g., based on luminance samples of the boundary pixel and one or more adjacent pixels (e.g., 502G-1 to 502G-6) according to a predefined luminance interpolation scheme 510. Figure 6C 508G in the middle). Boundary chromaticity samples (e.g., Figure 6C 502G in the model is determined from the boundary brightness sample according to the linear mapping model 512 (712).

[0112] In some embodiments, the predefined luminance interpolation scheme 510 is determined at least based on the position of the boundary pixels, and based on the predefined luminance interpolation scheme 510, one or more adjacent pixels are adjacent to the boundary pixels. In some cases, the predefined luminance interpolation scheme 510 is also determined based on a subsampling scheme, in which luminance and chrominance samples of multiple pixels conform to a three-part Y'CbCr ratio. In some cases, the predefined luminance interpolation scheme 510 is also determined based on syntax elements derived from the bitstream (e.g., "sps_chroma_collocated_vertical_flag"). Optionally, the predefined luminance interpolation scheme 510 corresponds to one of a cross, block, or T-shaped filter configuration.

[0113] In some embodiments, one or more neighboring pixels of a boundary pixel are outside a video frame or image strip. In some embodiments, one or more neighboring pixels of a boundary pixel have not yet been decoded and will be decoded after a code block.

[0114] In some embodiments, the plurality of pixels comprises a first set of pixels that are entirely included in the code block. Internal brightness samples (e.g., Figures 6A-6C The 508A, 508C, and 508F in the dataset are determined from the luminance sample set corresponding to the first pixel set according to a predefined luminance interpolation scheme 510. The internal chrominance samples corresponding to the first pixel set (e.g., ...) Figures 6A-6C The values ​​502A, 502C, and 502F in the model are determined based on internal luminance samples according to the linear mapping model 512.

[0115] In some embodiments, such as in Figure 6AIn this context, multiple pixels further include internal pixels, which are located within the code block and correspond to internal luminance samples (e.g., 504B-3, 504B-4). Boundary luminance samples (e.g., 508B) are determined according to a predefined luminance interpolation scheme 510 based on the internal luminance samples of the internal pixels (e.g., 504B-3, 504B-4) and the luminance samples of the boundary pixels and one or more adjacent pixels (e.g., 504B-1, 504B-2).

[0116] In some embodiments, such as in Figure 6A In this context, the boundary pixel is a first boundary pixel (e.g., corresponding to luminance sample 504B-2), and one or more adjacent pixels include one or more first adjacent pixels (e.g., corresponding to luminance sample 504B-1). Multiple pixels include second boundary pixels (e.g., corresponding to luminance sample 504B-5) that are located inside code block 506 and immediately adjacent to the corresponding block boundary 602 of code block 506. The electronic device creates one or more second adjacent pixels (e.g., corresponding to luminance sample 504B-6) immediately adjacent to the second boundary pixels. Each of the one or more second adjacent pixels is located outside code block 506. A luminance sample (e.g., 504B-5) corresponding to a second boundary pixel is assigned to a luminance sample (e.g., 504B-6) corresponding to each of one or more second neighboring pixels, wherein the boundary luminance sample 508B is determined based on the luminance samples of at least the first and second boundary pixels (e.g., corresponding to luminance samples 504B-2 and 504B-5) and the first and second neighboring pixels (e.g., corresponding to luminance samples 504B-1 and 504B-6) according to a predefined luminance interpolation scheme 510.

[0117] In some embodiments, such as in Figure 6B In this context, one or more adjacent pixels include only one adjacent pixel (e.g., corresponding to luminance sample 504D-5), which is located on the opposite side of boundary 602A relative to the boundary pixel (e.g., corresponding to luminance sample 504D-1) and has a corresponding luminance sample (e.g., 504D-5) assigned from the luminance sample (e.g., 504D-1) corresponding to the boundary pixel.

[0118] In some embodiments, such as in Figure 6CIn the block 506C, the boundary includes a first boundary 602A, and the boundary pixel (e.g., corresponding to luminance sample 504G-2) is adjacent to the corner of the code block formed between the first boundary 602A and the second boundary 602C perpendicular to the first boundary 602A. One or more adjacent pixels include a first adjacent pixel (e.g., corresponding to luminance sample 504G-1) located on the opposite side of the first boundary 602A relative to the boundary pixel, and at least one of the following: (1) a second adjacent pixel (e.g., corresponding to luminance sample 504G-5) located on the opposite side of the second boundary relative to the boundary pixel, and (2) a third adjacent pixel (e.g., corresponding to luminance sample 504G-6) with its corner opposite the boundary pixel across the code block 506C.

[0119] In some embodiments, according to the subsampling scheme, the luminance samples and chrominance samples of a plurality of pixels follow a three-part Y'CbCr ratio having one of the following ratios: (1) 4:1:1, wherein every four horizontal pixels correspond to four luminance samples, one blue differential chrominance sample Cb, and one red differential chrominance sample Cr; (2) 4:2:0, wherein every four pixels in each 2x2 pixel block correspond to four luminance samples, one blue differential chrominance sample Cb, and one red differential chrominance sample Cr; (3) 4:2:2, wherein every four pixels correspond to four luminance samples, two blue differential chrominance samples Cb, and two red differential chrominance samples Cr; and (4) 4:4:4, wherein every four pixels correspond to four luminance samples, four blue differential chrominance samples Cb, and four red differential chrominance samples Cr.

[0120] In some embodiments, according to a predefined brightness interpolation scheme, six adjacent brightness samples 504 in the 2x3 subarray are downsampled into replacement brightness samples 508, and each column of the left and right columns of brightness samples in the 2x3 array is used at least twice to generate replacement brightness samples on the same row, for example in Figure 6A , Figure 6C and Figure 6D In some embodiments, according to a predefined brightness interpolation scheme, five adjacent brightness samples 504 are downsampled into replacement brightness samples 508 and arranged in a cross-shaped subarray centered on a central pixel, for example in... Figure 6B middle.

[0121] In some embodiments, a bitstream is obtained, which can derive a second plurality of luminance samples 518 and a second plurality of chrominance samples 516 for a second plurality of pixels in a video frame. Replacement plurality of luminance samples 514 are determined according to a predefined luminance interpolation scheme 510 and have the same resolution as the plurality of chrominance samples 516. A first parameter α and a second parameter β for a linear mapping model 512 are determined using the replaced plurality of luminance samples 514 and the second plurality of chrominance samples 516. Further, in some embodiments, a boundary chrominance sample Y is determined from a boundary luminance sample X according to a linear mapping model 512 as described below in equation (1). (Refer to...) Figure 5 In some embodiments, two maximum luminance samples 514A-4 and 514A-5 are determined among the replaced plurality of luminance samples 514, and two minimum luminance samples 514B-4 and 514B-5 are determined among the replaced plurality of luminance samples 514. The two maximum luminance samples 514A-4 and 514A-5 are averaged to obtain a first luminance value, and the two minimum luminance samples 514B-4 and 514B-5 are averaged to obtain a second luminance value. Two first chrominance samples 516A-4 and 516A-5 are determined among the second plurality of chrominance samples 514 associated with the two maximum luminance samples 514A-4 and 514A-5. Two second chrominance samples 516B-4 and 516B-5 are determined among the second plurality of chrominance samples associated with the two minimum luminance samples 514B-4 and 514B-5. The values ​​of the two first chromaticity samples 516A-4 and 516A-5 are averaged to obtain the first chromaticity value corresponding to the first luminance value, and the values ​​of the two second chromaticity samples 516B-4 and 516B-5 are averaged to obtain the second chromaticity value corresponding to the second luminance value. Based on the first and second luminance values ​​and the first and second chromaticity values, a linear mapping model 512 is derived between chromaticity sample 502 and luminance sample 508.

[0122] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code, and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium, including any medium that, for example, facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures used to implement the implementations described in this application. Computer program products may include computer-readable media.

[0123] The terminology used in the description of implementations herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. When used in the description of implementations and the appended claims, the singular forms “a,” “an,” and “this / that” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more of the associated listed items. It should be further understood that, when used in this specification, the terms “comprising” and / or “including” expressly specify the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0124] It should also be understood that although the terms "first," "second," etc., may be used herein to describe various different elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first electrode may be called a second electrode, and similarly, a second electrode may be called a first electrode, without departing from the scope of the implementation. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.

[0125] The description of this application has been given for illustrative and descriptive purposes and is not intended to be exhaustive or limited to the disclosed forms of the invention. Many modifications, variations, and alternative implementations will be apparent to those skilled in the art who benefit from the teachings given in the foregoing description and associated drawings. Embodiments have been chosen and described to best explain the principles of the invention, its practical application, and to enable others skilled in the art to understand the invention in various different implementations and to best utilize the underlying principles and different implementations with different modifications suitable for the contemplated particular uses. Therefore, it should be understood that the scope of the claims should not be limited to the specific examples of the disclosed implementations, and modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. A method for decoding video data, comprising: Multiple luminance samples of a coded block are obtained from the bit stream, wherein the multiple luminance samples include luminance samples corresponding to the boundary luminance position adjacent to the boundary of the coded block or an extension of the boundary. In response to determining that one or more adjacent luminance samples of the coding block are unavailable, luminance samples corresponding to the boundary luminance position are assigned to the one or more adjacent luminance samples, wherein each of the one or more adjacent luminance samples corresponds to an adjacent luminance position outside the coding block and close to the boundary luminance position; According to a predefined brightness interpolation scheme, the boundary brightness sample points are determined at least based on the one or more adjacent brightness sample points and the brightness sample points corresponding to the boundary brightness positions. Obtain a second plurality of luminance samples and a second plurality of chrominance samples from the bit stream; Based on the predefined luminance interpolation scheme, a plurality of replacement luminance samples with the same resolution as the second plurality of chrominance samples are determined; By using a plurality of replaced luminance samples and a plurality of replaced chrominance samples, the first parameter α and the second parameter β for the linear mapping model are derived, including: Among the replaced multiple brightness samples, identify the two largest brightness samples; Among the multiple replaced brightness samples, determine the two smallest brightness samples; The first brightness value is determined based on the two maximum brightness samples; The second brightness value is determined based on the two minimum brightness samples. Two first chromaticity samples are identified among the second plurality of chromaticity samples associated with the two maximum luminance samples; Two second chromaticity samples are identified among the second plurality of chromaticity samples associated with the two minimum luminance samples; Based on the two first chromaticity sample points, a first chromaticity value corresponding to the first luminance value is determined; Based on the two second chromaticity samples, determine the second chromaticity value corresponding to the second luminance value; and Based on the first luminance value and the second luminance value, and the first chrominance value and the second chrominance value, the linear mapping model is derived between the chrominance value and the luminance value; and Based on the linear mapping model, boundary chromaticity samples are determined from the boundary luminance samples.

2. The method according to claim 1, wherein the adjacent brightness position includes the adjacent brightness position above, and the adjacent brightness position above includes the position directly above the coding block and / or the first upper-left position above the left of the coding block.

3. The method according to claim 1, wherein the adjacent brightness position includes the adjacent brightness position on the left, and the adjacent brightness position on the left includes the position directly to the left of the coding block and / or the second upper-left position above the left of the coding block.

4. The method of claim 2, wherein the boundary brightness position is located inside the coding block and adjacent to the boundary above the coding block, or the boundary brightness position is located inside the adjacent coding block on the left and adjacent to the left extension of the boundary above.

5. The method of claim 3, wherein the boundary brightness position is located inside the coding block and adjacent to the left boundary of the coding block, or the boundary brightness position is located inside the adjacent coding block above and adjacent to the upper extension of the left boundary.

6. The method according to claim 1, wherein, The predefined luminance interpolation scheme is determined based on at least one of the subsampling scheme corresponding to the plurality of luminance samples and the syntax elements to be acquired in the bitstream.

7. The method of claim 1, wherein the one or more adjacent luminance samples are located outside the image stripe covering the current coding block.

8. The method according to claim 1, wherein the plurality of luminance samples includes a first luminance sample set, the method further comprising: According to the predefined brightness interpolation scheme, internal brightness samples are determined from the first brightness sample set; as well as Based on the linear mapping model, and using the internal luminance samples, determine the internal chrominance samples corresponding to the first set of luminance samples.

9. The method according to claim 1, wherein: The plurality of brightness samples includes internal brightness samples; and The boundary brightness sample points are determined based on the predefined brightness interpolation scheme, the internal brightness sample points, the one or more adjacent brightness sample points, and the brightness sample points corresponding to the boundary brightness positions.

10. The method of claim 1, wherein the one or more adjacent luminance samples comprise only one adjacent luminance sample, the adjacent luminance sample being located on the opposite side of the boundary relative to the boundary luminance position, and having a luminance sample assigned from the luminance sample corresponding to the boundary luminance position.

11. The method according to claim 1, wherein: The boundary includes a first boundary; The boundary brightness position is adjacent to the corner of the coded block formed between the first boundary and the second boundary perpendicular to the first boundary; The one or more adjacent brightness samples include: a first adjacent brightness sample, which is located on the opposite side of the first boundary relative to the brightness position of the boundary; And at least one of the following: (1) a second adjacent luminance sample point located on the opposite side of the second boundary relative to the boundary luminance position; and (2) a third adjacent luminance sample point located at an angle across the coding block relative to the boundary luminance position.

12. The method of claim 6, wherein, according to the subsample scheme, the luminance samples and chrominance samples of the coded block conform to a three-part Y'CbCr ratio having one of the following ratio values: (1) 4:2:0, where each of the four luminance samples in the 2x2 luminance sample block corresponds to a blue difference chromaticity sample Cb and a red difference chromaticity sample Cr; (2) 4:2:2, where every four luminance samples correspond to two blue difference chromaticity samples Cb and two red difference chromaticity samples Cr; and (3) 4:4:4, where each of the four brightness samples corresponds to four blue difference chromaticity samples Cb and four red difference chromaticity samples Cr.

13. The method of claim 1, wherein six luminance samples in the 2x3 subarray are downsampled into a plurality of replacement luminance samples according to the predefined luminance interpolation scheme, and at least two luminance samples in each of the left and right columns of the 2x3 array are used to generate the plurality of replacement luminance samples.

14. The method of claim 1, wherein five luminance samples are downsampled into a plurality of replacement luminance samples according to a predefined luminance interpolation scheme and arranged in a cross-shaped subarray centered on a central pixel.

15. The method of claim 1, wherein the boundary chromaticity sample Y is determined from the boundary luminance sample X according to the linear mapping model described using the following equation: ， Where α and β are parameters.

16. An electronic device comprising: Non-volatile computer-readable storage medium; as well as One or more processors are configured to perform the following encoding method to generate a bitstream and store the bitstream: Multiple luminance samples of a coding block are obtained from video data, wherein the multiple luminance samples include luminance samples corresponding to the luminance positions of the boundary adjacent to the boundary of the coding block or an extension of the boundary. In response to determining that one or more adjacent luminance samples of the coding block are unavailable, luminance samples corresponding to the boundary luminance position are assigned to the one or more adjacent luminance samples, wherein each of the one or more adjacent luminance samples corresponds to an adjacent luminance position outside the coding block and close to the boundary luminance position; According to a predefined brightness interpolation scheme, the boundary brightness sample points are determined at least based on the one or more adjacent brightness sample points and the brightness sample points corresponding to the boundary brightness positions. Obtain a second plurality of luminance samples and a second plurality of chrominance samples from the bit stream; Based on the predefined luminance interpolation scheme, a plurality of replacement luminance samples with the same resolution as the second plurality of chrominance samples are determined; By using a plurality of replaced luminance samples and a plurality of replaced chrominance samples, the first parameter α and the second parameter β for the linear mapping model are derived, including: Among the replaced multiple brightness samples, identify the two largest brightness samples; Among the multiple replaced brightness samples, determine the two smallest brightness samples; The first brightness value is determined based on the two maximum brightness samples; The second brightness value is determined based on the two minimum brightness samples. Two first chromaticity samples are identified among the second plurality of chromaticity samples associated with the two maximum luminance samples; Two second chromaticity samples are identified among the second plurality of chromaticity samples associated with the two minimum luminance samples; Based on the two first chromaticity sample points, a first chromaticity value corresponding to the first luminance value is determined; Based on the two second chromaticity samples, determine the second chromaticity value corresponding to the second luminance value; and Based on the first luminance value and the second luminance value, and the first chrominance value and the second chrominance value, the linear mapping model is derived between the chrominance value and the luminance value; and Based on the linear mapping model, boundary chromaticity samples are determined from the boundary luminance samples. The bitstream will be decoded by the method according to any one of claims 1-15.

17. A non-transitory computer-readable storage medium storing computer-executable instructions and a bit stream thereon, wherein the computer-executable instructions, when executed by a computing device having one or more processors, cause the one or more processors to perform an encoding method to generate the bit stream: Multiple luminance samples are obtained from a coded block of video data, where, The plurality of luminance samples include luminance samples corresponding to the luminance positions of the boundary adjacent to the boundary of the coding block or an extension of the boundary. In response to determining that one or more adjacent luminance samples of the coding block are unavailable, luminance samples corresponding to the boundary luminance position are assigned to the one or more adjacent luminance samples, wherein each of the one or more adjacent luminance samples corresponds to an adjacent luminance position outside the coding block and close to the boundary luminance position; According to a predefined brightness interpolation scheme, the boundary brightness sample points are determined at least based on the one or more adjacent brightness sample points and the brightness sample points corresponding to the boundary brightness positions. Obtain a second plurality of luminance samples and a second plurality of chrominance samples from the bit stream; Based on the predefined luminance interpolation scheme, a plurality of replacement luminance samples with the same resolution as the second plurality of chrominance samples are determined; By using a plurality of replaced luminance samples and a plurality of replaced chrominance samples, the first parameter α and the second parameter β for the linear mapping model are derived, including: Among the replaced multiple brightness samples, identify the two largest brightness samples; Among the multiple replaced brightness samples, determine the two smallest brightness samples; The first brightness value is determined based on the two maximum brightness samples; The second brightness value is determined based on the two minimum brightness samples. Two first chromaticity samples are identified among the second plurality of chromaticity samples associated with the two maximum luminance samples; Two second chromaticity samples are identified among the second plurality of chromaticity samples associated with the two minimum luminance samples; Based on the two first chromaticity sample points, a first chromaticity value corresponding to the first luminance value is determined; Based on the two second chromaticity samples, determine the second chromaticity value corresponding to the second luminance value; and Based on the first luminance value and the second luminance value, and the first chrominance value and the second chrominance value, the linear mapping model is derived between the chrominance value and the luminance value; and Based on the linear mapping model, boundary chromaticity samples are determined from the boundary luminance samples. The bitstream will be decoded by the method according to any one of claims 1-15.

18. A computer program product comprising computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1-15.

19. A method for transmitting a bit stream, comprising performing the following encoding method to generate the bit stream and transmitting the bit stream: Multiple luminance samples are obtained from a coded block of video data, where, The plurality of luminance samples include luminance samples corresponding to the luminance positions of the boundary adjacent to the boundary of the coding block or an extension of the boundary. In response to determining that one or more adjacent luminance samples of the coding block are unavailable, luminance samples corresponding to the boundary luminance position are assigned to the one or more adjacent luminance samples, wherein each of the one or more adjacent luminance samples corresponds to an adjacent luminance position outside the coding block and close to the boundary luminance position; According to a predefined brightness interpolation scheme, the boundary brightness sample points are determined at least based on the one or more adjacent brightness sample points and the brightness sample points corresponding to the boundary brightness positions. Obtain a second plurality of luminance samples and a second plurality of chrominance samples from the bit stream; Based on the predefined luminance interpolation scheme, a plurality of replacement luminance samples with the same resolution as the second plurality of chrominance samples are determined; By using a plurality of replaced luminance samples and a plurality of replaced chrominance samples, the first parameter α and the second parameter β for the linear mapping model are derived, including: Among the replaced multiple brightness samples, identify the two largest brightness samples; Among the multiple replaced brightness samples, determine the two smallest brightness samples; The first brightness value is determined based on the two maximum brightness samples; The second brightness value is determined based on the two minimum brightness samples. Two first chromaticity samples are identified among the second plurality of chromaticity samples associated with the two maximum luminance samples; Two second chromaticity samples are identified among the second plurality of chromaticity samples associated with the two minimum luminance samples; Based on the two first chromaticity sample points, a first chromaticity value corresponding to the first luminance value is determined; Based on the two second chromaticity samples, determine the second chromaticity value corresponding to the second luminance value; and Based on the first luminance value and the second luminance value, and the first chrominance value and the second chrominance value, the linear mapping model is derived between the chrominance value and the luminance value; and Based on the linear mapping model, boundary chromaticity samples are determined from the boundary luminance samples. The bitstream will be decoded by the method according to any one of claims 1-15.

Images