Candidate lists of multiple reference lines for video coding

Abstract

The video transcoder and video decoder are configured to decode video data using multi-auxiliary-line processing. The video transcoder and video decoder can determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the in-frame prediction mode, and use multi-auxiliary-line processing to decode video data blocks based on the number of auxiliary lines.

Description

Technical Field

[0001] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 250,080, filed September 29, 2021, and U.S. Provisional Patent Application No. 63 / 255,270, filed October 13, 2021, the entire contents of which are incorporated herein by reference.

[0002] This case concerns video encoding and video decoding. Prior Technology

[0003] Digital video capabilities can be integrated into a wide variety of devices, including digital televisions, digital live broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite wireless phones, so-called "smartphones," video conferencing equipment, video streaming devices, etc. Digital video devices implement video decoding technologies, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Decoding (AVC), ITU-T H.265 / High-Efficiency Video Decoding (HEVC), ITU-T H.266 / Variety Video Decoding (VVC), and extensions thereof, as well as proprietary video transcoders / formats, such as AOMedia Video 1 (AV1) developed by the Open Media Alliance. Video devices can more efficiently send, receive, encode, decode, and / or store digital video information by implementing such video decoding technologies.

[0004] Video decoding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or eliminate inherent redundancy in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be segmented into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in an intra-frame decoding (I) slice of a picture are encoded using spatial prediction with reference samples from adjacent blocks within the same picture. Video blocks in an inter-frame decoding (P or B) slice of a picture can use spatial prediction with reference samples from adjacent blocks within the same picture or temporal prediction with reference samples from other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0005] Generally speaking, this application describes techniques for improving the decoding efficiency and performance of multiple auxiliary lines (MRL) processing. These techniques can be used in any video transcoder, including instances of enhanced compression models (ECM) beyond VVC.

[0006] MRL processing can be used with various types of intra-frame prediction modes, including general intra-frame prediction, Intra-Frame Pattern Derivation (DIMD), and Template-Based Intra-Frame Pattern Derivation (TIMD) decoding modes. In DIMD and TIMD, the video decoder determines the intra-frame prediction direction. In general intra-frame prediction, the video transcoder can send the intra-frame prediction direction to the video decoder using a signal. In this example, the number of auxiliary lines and the actual use of auxiliary lines may differ for different types of intra-frame prediction. As an example, the video transcoder and video decoder can use a different number of auxiliary lines for MRL processing with a general intra-frame prediction mode compared to the number of auxiliary lines used for DIMD or TIMD modes. In this way, the management burden of signal transmission can be reduced for some intra-frame prediction types, and the auxiliary lines used can be better customized for the intra-frame prediction type.

[0007] In one instance, this case describes a method for decoding video data, which includes determining the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction pattern, and using multi-auxiliary line processing to decode video data blocks based on the number of auxiliary lines.

[0008] In another instance, this case describes an apparatus configured to decode video data, the apparatus comprising: a memory configured to store blocks of video data, and one or more processors implemented in a circuit and communicating with the memory, the one or more processors being configured to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on an in-frame prediction mode, and to use multi-auxiliary-line processing to decode the video data based on the number of auxiliary lines.

[0009] In another instance, this case describes an apparatus configured to decode video data, the apparatus comprising: a unit for determining the number of auxiliary lines to be used for multi-auxiliary line processing based on an in-frame prediction mode, and a unit for decoding a block of video data using multi-auxiliary line processing based on the number of auxiliary lines.

[0010] In another instance, this case describes a non-transitory computer-readable storage medium that stores instructions, when executed, causing one or more processors of a device configured to decode video data to determine, based on an in-frame prediction mode, the number of auxiliary lines to be used for multi-auxiliary line processing, and to use multi-auxiliary line processing to decode blocks of video data based on the number of auxiliary lines.

[0011] In another instance, this case describes an apparatus configured to encode video data, the apparatus comprising: a memory configured to store blocks of video data, and one or more processors implemented in a circuit and communicating with the memory, the one or more processors being configured to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on an in-frame prediction mode, and to encode the blocks of video data using multi-auxiliary-line processing based on the number of auxiliary lines.

[0012] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Simple Explanation of the Diagram

[0013] Figure 1 is a block diagram illustrating an example video encoding and decoding system that can implement the technology described in this case.

[0014] Figure 2 is a conceptual diagram illustrating an example of multi-auxiliary-line processing.

[0015] Figure 3 is a block diagram illustrating an example video transcoder that can perform the technology described in this case.

[0016] Figure 4 is a block diagram illustrating an example video decoder that can implement the technology in this case.

[0017] Figure 5 is a flowchart illustrating the instance method for encoding the current block according to the technology of this case.

[0018] Figure 6 is a flowchart illustrating an instance method for decoding the current block according to the technology of this case.

[0019] Figure 7 is a flowchart illustrating another instance method for decoding the current block according to the technology in this case. Implementation

[0020] In in-frame prediction, the prediction for the current decoding unit (CU) is generated from samples taken from auxiliary lines. In some instances, the video decoder can be configured to perform in-frame prediction from one of multiple auxiliary lines (MRLs), such as MRL processing. The default auxiliary lines are the auxiliary lines above and to the left of the current decoding unit (line 0). In addition to line 0, the auxiliary lines used for MRL processing may also include one or more auxiliary lines that are more than one sample away from the top and left boundaries of the current decoding unit.

[0021] In some instances, a video decoder can be configured to construct and use candidate lists for MRL processing. In one instance, the video decoder can be configured to use both decoder-side Intra-Frame Pattern Derivation (DIMD) and Template-Based Intra-Frame Pattern Derivation (TIMD) decoding modes. DIMD and TIMD decoding modes are intra-frame prediction decoding modes in which the video decoder implicitly determines the intra-frame prediction direction without decoding syntax elements that directly indicate the intra-frame prediction direction.

[0022] In one implementation, when operating according to DIMD and TIMD decoding modes, the video decoder is configured to use the same MRL candidate list as used for general intra-frame prediction (e.g., intra-frame prediction modes where the intra-frame prediction direction is determined by a method other than DIMD or TIMD). That is, DIMD and TIMD decoding modes share the same MRL candidate list as general intra-frame prediction. This implementation, using the same MRL candidate list for multiple intra-frame prediction modes, lacks flexibility and may result in suboptimal intra-frame prediction decoding using MRL processing, since the video content used for DIMD and TIMD modes may differ.

[0023] This case describes several techniques that can solve the above problems. For example, a video decoder can be configured to determine the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction mode, and to use multi-auxiliary line processing to decode video data blocks based on the number of auxiliary lines.

[0024] Figure 1 is a block diagram illustrating an example video encoding and decoding system 100 that can perform the technology of this invention. The technology of this invention is generally aimed at decoding (encoding and / or decoding) video data. Typically, video data includes any data used to process video. Therefore, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video relay data, such as signal transmission data.

[0025] As shown in Figure 1, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, the source device 102 provides the video data to the destination device 116 via computer-readable media 110. The source device 102 and the destination device 116 can include any of a wide range of devices, including desktop computers, laptops, mobile devices, tablets, set-top boxes, handsets (e.g., smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receivers, etc. In some cases, the source device 102 and the destination device 116 can be equipped for wireless communication and therefore can be referred to as wireless communication devices.

[0026] In the example of Figure 1, source device 102 includes a video source 104, memory 106, a video transcoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a video decoder 300, memory 120, and a display device 118. According to this invention, the video transcoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for multi-auxiliary-line video decoding. Therefore, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or arrangements. For example, source device 102 may receive video data from an external video source (e.g., an external camera). Similarly, destination device 116 may be coupled to an external display device, rather than including an integrated display device.

[0027] The system 100 shown in Figure 1 is merely an example. Generally, any digital video encoding and / or decoding device can perform techniques for multi-line video decoding. Source device 102 and destination device 116 are simply examples of such decoding devices, where source device 102 generates decoded video data for transmission to destination device 116. In this document, a "decoding" device is referred to as a device that performs the decoding (encoding and / or decoding) of data. Therefore, video transcoder 200 and video decoder 300 represent examples of decoding devices, specifically a video transcoder and a video decoder, respectively. In some instances, source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video replay, video broadcasting, or video telephony.

[0028] Typically, video source 104 represents a source of video data (i.e., raw, unencoded video data) and a series of consecutive images (also called "frames") that provide video data to video transcoder 200, which encodes the data in the images. The video source 104 of source device 102 may include video capturing devices such as a camera, a video file containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. Alternatively, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video transcoder 200 encodes the captured, pre-captured, or computer-generated video data. Video transcoder 200 may rearrange the images from the order of reception (sometimes called "display order") into a decoding order for decoding. The video transcoder 200 can generate a bitstream including encoded video data. The source device 102 can then output the encoded video data to a computer-readable medium 110 via an output interface 108 for reception and / or retrieval by an input interface 122 of a destination device 116, for example.

[0029] The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general-purpose memory. In some instances, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Alternatively or additionally, memories 106 and 120 may store software instructions executable by, for example, video transcoder 200 and video decoder 300. Although memories 106 and 120 are shown separately from video transcoder 200 and video decoder 300 in this example, it should be understood that video transcoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, for example, output from video transcoder 200 and input to video decoder 300. In some instances, portions of memory 106 and 120 may be allocated as one or more video buffers, for example, to store raw, decoded, and / or encoded video data.

[0030] Computer-readable media 110 can refer to any type of media or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable media 110 refers to a communication medium that enables source device 102 to transmit encoded video data directly and instantaneously to destination device 116, for example via a radio frequency network or a computer-based network. According to communication standards such as wireless communication protocols, output interface 108 can modulate the transmitted signal including the encoded video data, and input interface 122 can demodulate the received transmitted signal. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, wide area network, or global area network such as the Internet. The communication medium can include a router, switch, base station, or any other device that can be used to facilitate communication from source device 102 to destination device 116.

[0031] In some instances, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disks, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0032] In some instances, source device 102 can output encoded video data to file server 114 or another intermediate storage device that can store the encoded video data generated by source device 102. Destination device 116 can access the stored video data from file server 114 via data streaming or download.

[0033] File server 114 can be any type of server device capable of storing encoded video data and transmitting such encoded video data to destination device 116. File server 114 can represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (e.g., File Transfer Protocol (FTP) or One-Way File Transfer Protocol (FLUTE)), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and / or a Network Additional Storage (NAS) device. File server 114 can supplementarily or alternatively implement one or more HTTP data streaming protocols, such as Dynamic Self-Adjusting Data Streaming over HTTP (DASH), HTTP Live Data Streaming (HLS), Real-Time Data Streaming Protocol (RTSP), HTTP Dynamic Data Streaming, etc.

[0034] Destination device 116 can access encoded video data from file server 114 via any standard data connection, including an Internet connection. This can include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., Digital Subscriber Line (DSL), Cable Modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. Input interface 122 can be configured to operate according to any one or more of the protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data.

[0035] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, modem, wired networking component (e.g., Ethernet card), wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In instances where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured to transmit data, such as encoded video data, according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some instances where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to transmit data, such as encoded video data, according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), Bluetooth™ standard, etc. In some instances, source device 102 and / or destination device 116 may include corresponding system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing functions belonging to video transcoder 200 and / or output interface 108, and destination device 116 may include a SoC device for performing functions belonging to video decoder 300 and / or input interface 122.

[0036] The technology in this case can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet data streaming video transmission, such as Dynamic Self-Adjusting Data Streaming (DASH) over HTTP, digital video encoded onto data storage media, decoding digital video stored on data storage media, or other applications.

[0037] The input interface 122 of the destination device 116 receives an encoded video bitstream from computer-readable media 110 (e.g., communication media, storage device 112, file server 114, etc.). The encoded video bitstream may include signal transmission information defined by the video transcoder 200, which is also used by the video decoder 300, such as syntax elements with values ​​describing the characteristics and / or processing of video blocks or other decoded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays a decoded image of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

[0038] Although not illustrated in Figure 1, in some instances, the video transcoder 200 and the video decoder 300 may be integrated with the audio encoder and / or audio decoder, and may include appropriate MUX-DEMUX units or other hardware and / or software to handle multiplexed streams that include both audio and video in a public data stream.

[0039] Both the video transcoder 200 and the video decoder 300 can be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), individual logic, software, hardware, firmware, or any combination thereof. When these technologies are partially implemented in software, the device can store instructions for the software in a suitable non-transitory computer-readable medium and use one or more processors to execute the instructions in hardware to perform the technology of this invention. Each of the video transcoder 200 and the video decoder 300 can be included in one or more encoders or decoders, wherein either can be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the video transcoder 200 and / or the video decoder 300 can include integrated circuits, microprocessors, and / or wireless communication devices, such as cellular telephones.

[0040] The video transcoder 200 and video decoder 300 may operate according to video decoding standards such as ITU-T H.265, also known as High Efficiency Video Decoding (HEVC) or its extensions, such as MultiView and / or Extensible Video Decoding Extensions. Alternatively, the video transcoder 200 and video decoder 300 may operate according to other proprietary or industry standards such as ITU-T H.266, also known as Universal Video Decoding (VVC). In other instances, the video transcoder 200 and video decoder 300 may operate according to proprietary video transcoders / formats such as AOMedia Video 1 (AV1), extensions to AV1, and / or subsequent versions of AV1 (e.g., AV2). In other instances, the video transcoder 200 and video decoder 300 may operate according to other proprietary formats or industry standards. However, the technology in this application is not limited to any particular decoding standard or format. Generally speaking, the video transcoder 200 and the video decoder 300 can be configured to perform the technology of this case in combination with any video decoding technology that uses multi-auxiliary-line video decoding technology.

[0041] Generally, video transcoder 200 and video decoder 300 can perform block-based decoding of images. The term "block" typically refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in encoding and / or decoding processes). For example, a block may include a two-dimensional matrix of luma and / or chroma data samples. Generally, video transcoder 200 and video decoder 300 can decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, instead of decoding the sampled red, green, and blue (RGB) data of the image, video transcoder 200 and video decoder 300 can decode the luma and chroma components, where the chroma components may include both red hue and blue hue chroma components. In some instances, video transcoder 200 converts received RGB format data to YUV representation before encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and postprocessing units (not shown) can perform these transformations.

[0042] This case may generally refer to the decoding of an image (e.g., encoding and decoding) to include procedures for encoding or decoding data in an image. Similarly, this case may refer to the decoding of blocks of an image to include procedures for encoding or decoding data in blocks, such as prediction and / or residual decoding. Encoded video bitstreams typically include a series of values ​​representing decoding decisions (e.g., decoding modes) and syntax elements that divide the image into blocks. Therefore, references to decoding an image or block should generally be understood as decoding the values ​​of the syntax elements used to form the image or block.

[0043] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (e.g., video transcoder 200) partitions a decoding tree unit (CTU) into CUs according to a quadtree structure. That is, the video decoder partitions the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes can be called "leaf nodes," and the CU of such leaf nodes can include one or more PUs and / or one or more TUs. The video decoder can further partition the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of the TU. In HEVC, the PU represents inter-frame prediction data, while the TU represents residual data. CUs predicted within a frame include intra-frame prediction information, such as intra-frame pattern indication.

[0044] As another example, the video transcoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, the video decoder (e.g., video transcoder 200) segments the image into multiple decoding tree units (CTUs). The video transcoder 200 can segment the CTUs according to tree structures such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple segmentation types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure consists of two levels: the first level segments according to quadtree segmentation, and the second level segments according to binary tree segmentation. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the decoding units (CUs).

[0045] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of triplet tree (TT) partitioning. Triplet tree or ternary tree partitioning is a partition in which a block is divided into three sub-blocks. In some instances, triplet or ternary tree partitioning divides a block into three sub-blocks without partitioning the original block via a center. The partitioning types in MTT (such as QT, BT, and TT) can be symmetric or asymmetric.

[0046] When operating according to the AV1 transcoder, the video transcoder 200 and video decoder 300 can be configured to decode video data in blocks. In AV1, the largest decoded block that can be processed is called a superblock. In AV1, a superblock can be a 128x128 luminance sample or a 64x64 luminance sample. However, in subsequent video decoding formats (e.g., AV2), superblocks can be defined by different (e.g., larger) luminance sample sizes. In some instances, the superblock is the top level of a block quadtree. The video transcoder 200 can further divide the superblock into smaller decoded blocks. The video transcoder 200 can divide the superblock and other decoded blocks into smaller blocks using square or non-square partitioning. Non-square blocks can include N / 2xN, NxN / 2, N / 4xN, and NxN / 4 blocks. The video transcoder 200 and video decoder 300 can perform separate prediction and transform procedures for each decoded block.

[0047] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video transcoder 200 and the video decoder 300 can encode and decode the decoded blocks within a tile separately without using video data from other tiles. However, the video transcoder 200 and the video decoder 300 can perform filtering across tile boundaries. The tile size can be uniform or non-uniform. Tile-based decoding enables parallel processing and / or multi-threaded processing depending on the encoder and decoder implementation.

[0048] In some instances, the video transcoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other instances, the video transcoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for the two chrominance components (or two QTBT / MTT structures for the corresponding chrominance components).

[0049] The video transcoder 200 and the video decoder 300 can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning or other partitioning structures.

[0050] In some instances, a CTU includes a decoded tree block (CTB) for luminance sampling, two corresponding CTBs for chrominance sampling of an image with three sampling arrays, or a CTB for sampling of a monochrome image or an image that is decoded using three separate color planes and a syntax structure for decoding the samples. For a given value of N, a CTB can be an NxN sampling block such that dividing the components into CTBs is a partition. Components are arrays or single samples from one of the three arrays (luminance and two chrominance) that make up an image in a 4:2:0, 4:2:2, or 4:4:4 color format, or arrays or single samples from an array or array that makes up a monochrome image. In some instances, a decoded block is an MxN sampling block for some values ​​of M and N such that dividing the CTB into decoded blocks is a partition.

[0051] Blocks (e.g., CTUs or CUs) can be encapsulated in various ways within an image. As an example, a brick can refer to a rectangular area of ​​a row of CTUs within a specific tile in an image. A tile can be a rectangular area of ​​a CTU within a specific tile column or a specific tile row in an image. A tile column is a rectangular area of ​​a CTU whose height is equal to the height of the image, and whose width is specified by a syntax element (e.g., in the image parameter set). A tile row is a rectangular area of ​​a CTU whose height is specified by a syntax element (e.g., in the image parameter set), and whose width is equal to the width of the image.

[0052] In some instances, a tile can be divided into multiple bricks, each brick potentially comprising one or more CTU rows within the tile. A tile not divided into multiple bricks can also be referred to as a brick. However, bricks that are a proper subset of a tile may not be referred to as tiles. Bricks in an image can also be arranged in slices. A slice can be an integer number of bricks in an image that can be exclusively contained within a single Network Abstraction Layer (NAL) unit. In some instances, a slice comprises a continuous sequence of multiple complete tiles or a single complete brick from a single tile.

[0053] This design allows for the interchangeable use of "NxN" and "N multiplied by N" to represent the sampling dimensions of a block (e.g., a CU or other video block) in both the vertical and horizontal dimensions; for example, 16x16 samples or 16 multiplied by 16 samples. Generally, a 16x16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an NxN CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily have the same number of samples in the horizontal direction as it does in the vertical direction. For example, a CU can include NxM samples, where M is not necessarily equal to N.

[0054] Video transcoder 200 encodes video data of CUs (Complex Units) representing prediction and / or residual information, as well as other information. Prediction information indicates how the CUs will be predicted to form prediction blocks of the CUs. Residual information typically represents the sample-by-sample difference between the samples of the CUs before encoding and the prediction blocks.

[0055] To predict the CU, the video transcoder 200 typically forms a predicted block of the CU via inter-frame prediction or intra-frame prediction. Inter-frame prediction typically refers to predicting the CU from data of a previously decoded image, while intra-frame prediction typically refers to predicting the CU from data of a previously decoded image within the same image. To perform inter-frame prediction, the video transcoder 200 can use one or more motion vectors to generate the predicted block. The video transcoder 200 can typically perform motion search to identify a reference block that closely matches the CU, for example, in terms of the difference between the CU and the reference block. The video transcoder 200 can use sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to compute a difference metric to determine whether the reference block closely matches the current CU. In some instances, the video transcoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0056] Some instances of VVC also offer affine motion compensation modes, which can be viewed as inter-frame prediction modes. In affine motion compensation modes, the video transcoder 200 can determine two or more motion vectors representing non-translational motion, such as zoom in or out, rotation, perspective motion, or other irregular motion types.

[0057] To perform intra-frame prediction, the video transcoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some instances of VVC provide 67 intra-frame prediction modes, including various orientation modes, as well as planar and DC modes. Generally, the video transcoder 200 selects an intra-frame prediction mode that describes the adjacent samples of the current block (e.g., the block of the CU), and predicts the samples of the current block based on these adjacent samples. Assuming the video transcoder 200 decodes the CTU and CU in raster scan order (from left to right, from top to bottom), such samples can typically be located above, to the upper left, and to the left of the current block within the same image as the current block.

[0058] Video transcoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction modes, video transcoder 200 can encode data indicating which of the various available inter-frame prediction modes is used, as well as the motion information of the corresponding mode. For unidirectional or bidirectional inter-frame prediction, for example, video transcoder 200 can use Advanced Motion Vector Prediction (AMVP) or merging modes to encode motion vectors. Video transcoder 200 can use similar modes to encode motion vectors for affine motion compensation modes.

[0059] AV1 includes two common techniques for encoding and decoding blocks of video data. These two techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when predicting blocks of the current frame of video data using an intra-frame prediction mode, the video transcoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video transcoder 200 encodes the block of the current frame based on the difference between the sampled values ​​in the current block and the predicted values ​​generated from a reference sample in the same frame. The video transcoder 200 determines the predicted values ​​generated from the reference sample based on the intra-frame prediction mode.

[0060] Following predictions such as intra-frame prediction or inter-frame prediction, the video transcoder 200 can compute residual data for the block. Residual data, such as residual blocks, represents the sample-by-sample difference between the block and a predicted block formed using the corresponding prediction mode. The video transcoder 200 can apply one or more transforms to the residual blocks to produce transformed data in the transform domain rather than the sampling domain. For example, the video transcoder 200 can apply Discrete Cosine Transform (DCT), integer transform, wavelet transform, or conceptually similar transforms to the residual video data. Furthermore, the video transcoder 200 can apply secondary transforms after the first transform, such as Mode Correlated Inseparable Secondary Transform (MDNSST), Signal Correlation Transform, Karhunen-Loeve Transform (KLT), etc. The video transcoder 200 generates transform coefficients after applying one or more transforms.

[0061] As mentioned above, after any transformation that produces the transform coefficients, the video transcoder 200 can perform quantization of the transform coefficients. Quantization generally refers to a procedure in which the transform coefficients are quantized to reduce the amount of data used to represent them, thereby providing further compression. By performing the quantization procedure, the video transcoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, the video transcoder 200 can round an n-bit value down to an m-bit value during quantization, where n is greater than m. In some instances, to perform quantization, the video transcoder 200 can perform a bitwise right shift of the value to be quantized.

[0062] After quantization, the video transcoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) transform coefficients before the vector and lower-energy (and therefore higher-frequency) transform coefficients after the vector. In some instances, the video transcoder 200 can use a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other instances, the video transcoder 200 can perform a self-adjusting scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video transcoder 200 can entropy encode the one-dimensional vector, for example, according to context-adjusted binary arithmetic decoding (CABAC). The video transcoder 200 can also entropy encode the values ​​of syntax elements describing relay data associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0063] To perform CABAC, the video transcoder 200 can assign context within a context model to the symbols to be transmitted. For example, the context could involve whether the neighboring values ​​of a symbol are zero. Probability decisions can be based on the context assigned to the symbols.

[0064] The video transcoder 200 can further generate syntax data for the video decoder 300 from, for example, picture headers, block headers, slice headers, or other syntax data (such as sequence parameter sets (SPS), picture parameter sets (PPS), or video parameter sets (VPS)). These syntax data can be block-based, picture-based, or sequence-based. The video decoder 300 can similarly decode such syntax data to determine how to decode the corresponding video data.

[0065] In this manner, the video transcoder 200 can generate a bitstream including encoded video data, such as syntax elements describing the segmentation of an image into blocks (e.g., CUs) and block prediction and / or residual information. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0066] Generally, the video decoder 300 executes a procedure opposite to that executed by the video transcoder 200 to decode encoded video data from a bitstream. For example, the video decoder 300 can use CABAC to decode the values ​​of syntax elements of the bitstream in a manner substantially similar to (but opposite to) the CABAC encoding procedure of the video transcoder 200. Syntax elements can define segmentation information for segmenting an image into CTUs and segmenting each CTU according to a corresponding segmentation structure (e.g., a QTBT structure) to define the CUs of the CTUs. Syntax elements can further define prediction and residual information for blocks (e.g., CUs) of the video data.

[0067] Residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inverse transform the quantized transform coefficients of a block to reproduce the residual block of that block. The video decoder 300 uses signaled prediction modes (intra-frame or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block of the block. The video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along block boundaries. The term "signaling" may generally refer to certain information, such as syntax elements. The term "signaling" can generally refer to the communication of the values ​​of syntax elements and / or other data used for decoding encoded video data. That is, the video transcoder 200 can signal the values ​​of syntax elements in the bitstream. Typically, signaling refers to generating a value in the bitstream. As mentioned above, source device 102 can transmit bitstreams to destination device 116 substantially instantly or non-instantly, for example, when storing syntax elements to storage device 112 for later retrieval by destination device 116.

[0068] According to the technology of this case, as will be described in more detail below, the video transcoder 200 and the video decoder 300 can be configured to perform techniques related to multi-auxiliary line processing, including techniques for determining the number of auxiliary lines and techniques for determining a candidate list of multi-auxiliary lines. In one example, the video transcoder 200 and the video decoder 300 can be configured to determine the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction mode, and to use multi-auxiliary line processing to decode video data blocks based on the number of auxiliary lines.

[0069] Multiple auxiliary lines (MRL)

[0070] In in-frame prediction, the prediction of the current decoding unit (CU) is generated from samples in the auxiliary lines. In some instances, the video transcoder 200 and the video decoder 300 can be configured to perform in-frame prediction from one of multiple auxiliary lines, such as MRL processing. Figure 2 is a conceptual diagram illustrating an example of multi-auxiliary-line processing. The preset auxiliary lines are auxiliary lines 402 (e.g., line 0) above and to the left of the current decoding unit 400, serving as reference samples. As shown in Figure 2, the decoding unit 400 is 4x4 samples in size. Each frame of the decoding unit 400 represents a sample (e.g., a luminance sample or a chrominance sample). Each frame of the auxiliary line 402 represents a reference sample. In this case, the reference sample can be a sample that has already been reconstructed by the video transcoder 200 or a sample that has already been decoded by the video decoder 300.

[0071] In addition to auxiliary line 402 (line 0), auxiliary lines used for MRL processing may also include one or more of auxiliary lines 404 (e.g., line 1), auxiliary lines 406 (e.g., line 2), or auxiliary lines 408 (e.g., line 3), as shown in Figure 2. That is, the auxiliary lines used for MRL processing in intra-frame prediction may be more than one sample away from the upper and left boundaries of the decoding unit 400. An example of MRL processing is described in the following document: Po-Han Lin, Chang-Hao Yau, Sheng-Po Wang, Chun-Lung Lin and Ching-Chieh Lin "CE3: Number of extended reference line for intra prediction", JVET-K0277, July 2018.

[0072] In one instance of Enhanced Compression Model (ECM) software (see https: / / vcgit.hhi.fraunhofer.de / ecm / ECM / - / tree / ECM-2.0), the video transcoder 200 and video decoder 300 can be configured to build and use a candidate list for MRL processing, which includes auxiliary lines 404 and 406 (e.g., {line 1, line 2}). In one instance, the video transcoder 200 and video decoder 300 can be enabled to use MRL processing for both decoder-side Intra-Frame Pattern Derivation (DIMD) and Template-Based Intra-Frame Pattern Derivation (TIMD) decoding modes. DIMD and TIMD decoding modes are intra-frame prediction decoding modes, where the video decoder 300 implicitly determines the intra-frame prediction direction without decoding syntax elements that directly indicate the intra-frame prediction direction.

[0073] In one implementation, when operating according to DIMD and TIMD decoding modes, the video transcoder 200 and video decoder 300 are configured to use the same MRL candidate list as used for general intra-frame prediction (e.g., intra-frame prediction modes, where the intra-frame prediction direction is determined by a method other than DIMD or TIMD). That is, DIMD and TIMD decoding modes share the same MRL candidate list as general intra-frame prediction. This implementation, using the same MRL candidate list for multiple intra-frame prediction modes, lacks flexibility and may result in suboptimal intra-frame prediction decoding using MRL processing, since the video content used for DIMD and TIMD modes may differ.

[0074] Furthermore, in one implementation, the use of non-DCT2 primary or secondary transforms is the same for all MRL candidates. For example, when decoding video data using general intra-frame prediction and DIMD decoding, the video decoder (e.g., video transcoder 200 and / or video decoder 300) can be configured to use non-DCT2 primary and secondary transforms for all MRL candidates. When decoding using TIMD mode, the video decoder is configured not to use any non-DCT2 primary or secondary transforms. However, these transforms are uniformly applied to all MRL candidates (for TIMD or DIMD or for general mode).

[0075] Example

[0076] This case describes several techniques that can solve the above-mentioned problems. The techniques described in this case can be used alone or in any combination.

[0077] When decoding video data using MRL processing, the video transcoder 200 and video decoder 300 are generally configured to use a number of sampled auxiliary lines (e.g., two or more). In one example, the video transcoder 200 can be configured to decode and signal syntax elements that explicitly indicate the number of auxiliary lines to be used. The video transcoder 200 can decode syntax elements at the block, slice, picture, or sequence level (e.g., in slice headers, picture headers, PPS, or SPS). Similarly, the video decoder 300 can be configured to receive and decode syntax elements to determine the number of auxiliary lines. In this context, the "number" of auxiliary lines can refer to the total number of auxiliary lines (e.g., 3 auxiliary lines) or which specific auxiliary lines are used. As an example, lines 0, 1, and 3 in Figure 2. In this way, the auxiliary lines used for MRL processing can be flexibly customized for blocks, slices, pictures, or sequences, potentially improving the decoding efficiency of in-frame prediction using MRL processing.

[0078] Therefore, in one example of this case, the video transcoder 200 and the video decoder 300 can be configured to decode video data blocks using in-frame prediction and multiple auxiliary lines processing. In one example, the video transcoder 200 and the video decoder 300 can be configured to decode syntax elements indicating the number of auxiliary lines to be used for multiple auxiliary lines processing. The video transcoder 200 and the video decoder 300 can then use that number of auxiliary lines to decode the video data block.

[0079] In another example of this case, the video transcoder 200 and the video decoder 300 can be configured to implicitly derive the number of MRL lines for MRL processing by analyzing how the auxiliary lines differ from each other. For example, the number of MRL auxiliary lines can be based on the difference in sample values ​​between two different auxiliary lines. Other methods for distinguishing auxiliary lines and inferring how many MRL auxiliary lines can be used should be considered within the scope of this case. In this way, the decoding efficiency of MRL processing can be improved by selecting the auxiliary lines best suited for in-frame predictive decoding.

[0080] Therefore, in another example of this case, the video transcoder 200 and the video decoder 300 can be configured to use multi-auxiliary-line processing to decode video data blocks. In this example, the video decoder 300 can be configured to implicitly determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the differences between two or more auxiliary lines.

[0081] In some instances, it may be undesirable to use auxiliary lines outside the CTU because the video transcoder 200 and video decoder 300 can be configured to store such auxiliary lines in an on-line buffer, requiring additional memory. To address this issue, in another instance of this case, when decoding video data using MRL processing, the video transcoder 200 and video decoder 300 are configured not to use auxiliary lines outside the CTU. This reduces implementation costs because additional memory is not required to store reference sampling lines outside the CTU boundary for MRL processing.

[0082] However, in other instances, the video transcoder 200 and the video decoder can be configured to use blocks close to the CTU boundary (e.g., blocks with a specific number of auxiliary lines less than the maximum distance can be used). For example, the auxiliary lines can extend 12 samples above the block. Therefore, if the block is only 4 samples from the CTU boundary, these 4 lines can be used, but not all 12 lines.

[0083] In one instance, the aforementioned restriction on blocks outside the CTU boundary can be implemented as an encoder (bitstream) constraint, where the video transcoder 200 is configured not to select an MRL index that violates an auxiliary line used outside the CTU (e.g., the MRL index cannot be signaled). An MRL index is a syntax element that indicates a specific MRL auxiliary line from the MRL candidate list.

[0084] In another instance, the syntax elements indicating auxiliary lines (e.g., MRL indexes) can be constrained, for example, making it impossible to signal an index that violates the aforementioned constraints. In one instance, since MRL auxiliary lines cannot be outside the current CTU, the maximum number of available MRL lines is counted, and the maximum number of available lines and truncation binarization used to count them also change the syntax elements used to signal MRL indexes accordingly.

[0085] Typically, the maximum number of available MRL auxiliary lines may depend on the block location relative to the CTU boundary containing the block, as mentioned above, and / or may depend on the prediction mode, as described below.

[0086] One example where the number of available MRL auxiliary lines may differ based on the prediction mode involves in-frame prediction decoding modes, where the in-frame prediction direction is output at the video decoder 300 (e.g., DIMD or TIMD modes). For example, the video decoder 300 may determine a different number of available auxiliary lines for different modes (e.g., DIMD, TIMD, etc.). For such modes, a smaller number of MRL auxiliary lines may be used compared to other prediction modes (e.g., general prediction modes where the in-frame prediction direction is signaled).

[0087] The video transcoder 200 and video decoder 300 can first determine the intra-frame prediction mode being used. For example, the intra-frame prediction mode can be one of general intra-frame prediction, DIMD, or TIMD. As an example, for DIMD and TIMD modes, the video transcoder 200 and video decoder 300 can implicitly decide to use a different number of MRL auxiliary lines than for general intra-frame prediction. For example, compared to the number of MRL auxiliary lines used for general intra-frame prediction, the video transcoder 200 and video decoder 300 can decide to use fewer MRL auxiliary lines for DIMD and TIMD modes. For example, for DIMD and TIMD modes, the video transcoder 200 and video decoder 300 can be configured to use two MRL auxiliary lines from the MRL candidate list out of a total of N available MRL auxiliary lines. By using fewer MRL auxiliary lines for DIMD and TIMD modes, the management burden of MRL indexing signal transmission can be reduced without significantly affecting distortion.

[0088] Therefore, in another example of this case, the video transcoder 200 and the video decoder 300 can be configured to decode video data blocks using multi-auxiliary-line processing. The video transcoder 200 and the video decoder 300 can determine the intra-frame prediction mode used to decode the video data blocks. In some instances, the intra-frame prediction mode can be a general intra-frame prediction mode, DIMD, TIMD, or one of other intra-frame prediction modes. In one instance, the video transcoder 200 and the video decoder 300 can be configured to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the intra-frame prediction mode. The video transcoder 200 and the video decoder 300 can be configured to determine a multi-auxiliary-line candidate list based on the intra-frame prediction mode, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, and template-based intra-frame mode derivation.

[0089] In one example of truncated binarization, the video transcoder 200 and the video decoder 300 can be configured to use truncated unary decoder MRL index signals with the maximum number of available auxiliary lines. For example, when the entire MRL candidate list is {line 1, line 2, line 3, line 4}, but only {line 1, line 2} is available for the current CU, truncated unary decoder with a maximum number of symbols of 2 is used.

[0090] Generally, the video transcoder 200 and the video decoder 300 can be configured to use the number of auxiliary lines for MRL processing relative to the decoding tree unit boundary constraints, as described above. In another instance, the number of MRL lines used can be self-adjusting; for example, the number of MRL auxiliary lines can depend on the mode (e.g., depending on the in-frame mode).

[0091] In one instance, the MRL candidate list used for TIMD (or DIMD) may differ from the MRL candidate list used for general in-frame prediction. For example, the MRL candidate list for general in-frame prediction might be {line 1, line 3, line 5, line 8}, while the MRL candidate list for TIMD (or DIMD) might be {line 1, line 2, line 6}. In this case, the line number indicates how many samples are taken from the top-left edge of the current decoding auxiliary line. Referring to Figure 2, auxiliary line 0 is adjacent to the left and top edges of decoding unit 400. Auxiliary line 1 is one sample removed from either the left or top edge of decoding unit 400. Similarly, auxiliary line 6 would be six samples removed from either the left or top edge of decoding unit 400.

[0092] In another instance, the MRL candidate list for TIMD (or DIMD) is a subset of the MRL candidate list for general in-frame prediction. In other words, the size of the MRL candidate list for TIMD (or DIMD) may be smaller than the size for general in-frame prediction, and the line indices for TIMD (or DIMD) can be selected from a subset of the line indices for general in-frame prediction. For example, the MRL candidate list for general in-frame prediction is {line 1, line 3, line 5, line 7, line 12}, and the MRL candidate list for TIMD (or DIMD) is {line 1, line 3}.

[0093] In another instance, non-DCT2 primary transforms (e.g., for Multiple Transform Selection (MTS)) or secondary transforms (e.g., for Low-Frequency Non-Separated Transform (LFNST)) can be disabled for a subset of the MRL candidate list. For example, when the MRL candidate list used for general in-frame prediction is {line 1, line 3, line 5, line 8}, MTS or LFNST, or both, can be disabled for {line 5, line 8}. Therefore, the resolution of the MTS or LFNST index will depend on which MRL candidate is used for the current CU.

[0094] Therefore, in another instance of this case, the video transcoder 200 and the video decoder 300 can be configured to disable one or more transformation types for a specific auxiliary line.

[0095] In another instance, the number of candidates in the MRL candidate list may also depend on the CU size, as a larger CU may benefit from having more candidates due to the increased variety.

[0096] Therefore, in another instance of this case, the video transcoder 200 and the video decoder 300 can be configured to determine the candidate list of multiple auxiliary lines based on the decoding unit size.

[0097] The technology in this case can be applied to other modes that use in-frame prediction, such as DIMD, location-dependent in-frame prediction combination (PDPC), combined inter-frame / in-frame prediction (CIIP), and other similar modes.

[0098] Figure 3 is a block diagram illustrating an example video transcoder 200 that can perform the technology of this application. Figure 3 is provided for illustrative purposes and should not be considered as a limitation on the technology extensively illustrated and described in this application. For illustrative purposes, this application describes a video transcoder 200 based on VVC (ITU-T H.266, under development) and HEVC (ITU-TH.265) technologies. However, the technology of this application can be performed by video encoding devices configured for other video decoding standards and video decoding formats (e.g., AV1 and successors to AV1 video decoding formats).

[0099] In the example of Figure 3, the video transcoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy encoding unit 220. Any one or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 can be implemented in one or more processors or in processing circuitry. For example, the units of the video transcoder 200 can be implemented as one or more circuit or logic elements, as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. In addition, the video transcoder 200 may include supplemental or alternative processors or processing circuitry to perform these and other functions.

[0100] Video data memory 230 can store video data to be encoded by components of video transcoder 200. Video transcoder 200 can receive video data stored in video data memory 230 from, for example, video source 104 (FIG. 1). DPB 218 can act as a reference picture memory, storing reference video data for use by video transcoder 200 in predicting subsequent video data. Video data memory 230 and DPB 218 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various instances, video data memory 230 can be on-chip along with other components of video transcoder 200, as shown, or off-chip relative to those components.

[0101] In this context, references to video data memory 230 should not be construed as limited to memory within video transcoder 200 unless explicitly stated otherwise, or memory outside video transcoder 200 unless specifically stated otherwise. Rather, references to video data memory 230 should be understood as memory storing references to video data received by video transcoder 200 for encoding (e.g., video data for the current block to be encoded). Memory 106 of FIG. 1 may also provide temporary storage for outputs from various units of video transcoder 200.

[0102] The various units in Figure 3 are illustrated to explain the operations performed by the video transcoder 200. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit is a circuit that provides a specific function and is pre-defined for the operations it can perform. A programmable circuit is a circuit that can be programmed to perform various tasks and provides flexible functionality in the operations it can perform. For example, a programmable circuit can execute software or firmware that causes it to operate in a manner defined by software or firmware instructions. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some instances, one or more units can be different circuit blocks (fixed-function or programmable), and in some instances, one or more units can be integrated circuits.

[0103] The video transcoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an example where the operation of the video transcoder 200 is performed using software executed by programmable circuitry, memory 106 (FIG. 1) may store instructions (e.g., object code) of the software received and executed by the video transcoder 200, or another memory (not shown) within the video transcoder 200 may store such instructions.

[0104] Video data memory 230 is configured to store received video data. Video transcoder 200 can retrieve images of the video data from video data memory 230 and provide the video data to residual generation unit 204 and mode selection unit 202. The video data in video data memory 230 can be the original video data to be encoded.

[0105] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an in-frame prediction unit 226. The mode selection unit 202 may include additional functional units to perform video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a palette unit, an in-frame copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, and so on.

[0106] The mode selection unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and the rate-distortion values ​​obtained for these combinations. Encoding parameters may include CTU segmentation into CUs, prediction modes for the CUs, transformation types of the residual data for the CUs, quantization parameters of the residual data for the CUs, etc. The mode selection unit 202 can ultimately select a combination of encoding parameters that yields a better rate-distortion value than other test combinations.

[0107] The video transcoder 200 can segment an image retrieved from the video data memory 230 into a series of CTUs and encapsulate one or more CTUs within a slice. The mode selection unit 202 can segment the image's CTUs according to a tree structure (e.g., MTT structure, QTBT structure, superblock structure, or the aforementioned quadtree structure). As mentioned above, the video transcoder 200 can form one or more CUs by segmenting CTUs according to a tree structure. Such CUs can also be collectively referred to as "video blocks" or "blocks".

[0108] Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-frame prediction unit 226) to generate predicted blocks for the current block (e.g., the current CU, or, in HEVC, the overlapping portion of PU and TU). For inter-frame prediction of the current block, motion estimation unit 222 can perform motion search to identify one or more closely matching reference blocks in one or more reference images (e.g., one or more previously decoded images stored in DPB 218). Specifically, motion estimation unit 222 can calculate values ​​representing the similarity between potential reference blocks and the current block, such as based on sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. Motion estimation unit 222 can typically perform these calculations using the sampled differences between the current block and the considered reference blocks. Motion estimation unit 222 can identify the reference block with the lowest value actively derived from these calculations to indicate the reference block that most closely matches the current block.

[0109] Motion estimation unit 222 can generate one or more motion vectors (MVs) that define the position of a reference block in a reference image (relative to the position of the current block in the current image). Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate a prediction block. For example, motion compensation unit 224 can use the motion vectors to retrieve data from the reference block. As another example, if the motion vectors have fractional sampling precision, motion compensation unit 224 can interpolate the values ​​of the prediction block according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data from the two reference blocks identified by the corresponding motion vectors and combine the retrieved data, for example, via sample-wise averaging or weighted averaging.

[0110] When operating according to the AV1 video decoding format, the motion estimation unit 222 and the motion compensation unit 224 can be configured to encode the decoded blocks of the video data (e.g., both the luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, overlap block motion compensation (OBMC), and / or composite inter-frame and intra-frame prediction.

[0111] As another example, for in-frame prediction or in-frame prediction decoding, the in-frame prediction unit 226 can generate a prediction block from samples adjacent to the current block. For example, for directional mode, the in-frame prediction unit 226 can typically mathematically combine the values ​​of adjacent samples and fill these calculated values ​​into the current block along a defined direction to generate a prediction block. As another example, for DC mode, the in-frame prediction unit 226 can calculate the average of samples adjacent to the current block and generate a prediction block to include the resulting average of each sample of the prediction block.

[0112] Intraframe prediction unit 226 can also be configured to perform intraframe prediction using MRL processing. Intraframe prediction unit 226 can be configured to perform any of the MRL processing techniques described above. For example, intraframe prediction unit 226 can be configured to determine the number of auxiliary lines to be used for multi-auxiliary line processing based on the intraframe prediction mode, and use multi-auxiliary line processing to encode video data blocks based on the number of auxiliary lines.

[0113] When operating according to the AV1 video decoding format, the in-frame prediction unit 226 can be configured to encode decoded blocks of video data (e.g., both luma and chroma decoded blocks) using directional in-frame prediction, non-directional in-frame prediction, recursive filter in-frame prediction, luma-to-chroma (CFL) prediction, intra-block copy (IBC), and / or palette modes. The mode selection unit 202 may include additional functional units to perform video prediction according to other prediction modes.

[0114] Mode selection unit 202 provides the prediction block to residual generation unit 204. Residual generation unit 204 receives the original uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The obtained sample-by-sample difference defines the residual block of the current block. In some instances, residual generation unit 204 can also determine the difference between sample values ​​in the residual block to generate the residual block using residual differential pulse decoding modulation (RDPCM). In some instances, one or more subtractor circuits performing binary subtraction can be used to form residual generation unit 204.

[0115] In an instance where the mode selection unit 202 divides the CU into PUs, each PU can be associated with a luminance prediction unit and a corresponding chrominance prediction unit. The video transcoder 200 and the video decoder 300 can support PUs of various sizes. As mentioned earlier, the size of the CU can refer to the size of the luminance decoding block of the CU, while the size of the PU can refer to the size of the luminance prediction unit of the PU. Assuming a particular CU has a size of 2Nx2N, the video transcoder 200 can support PU sizes of 2Nx2N or NxN for intra-frame prediction, and supports symmetrical PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, or similar for inter-frame prediction. The video transcoder 200 and the video decoder 300 can also support asymmetric partitioning of PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0116] In instances where the mode selection unit 202 does not further divide the CU into PUs, each CU can be associated with a luminance decoding block and a corresponding chrominance decoding block. As mentioned above, the size of the CU can refer to the size of the luminance decoding block of the CU. The video transcoder 200 and the video decoder 300 can support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0117] For other video decoding techniques (such as intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding, as examples), mode selection unit 202 generates a prediction block for the current block being encoded via a corresponding unit associated with the decoding technique. In some instances, such as palette mode decoding, mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how the block can be reconstructed based on the selected palette. In such modes, mode selection unit 202 may provide these syntax elements to entropy coding unit 220 for encoding.

[0118] As mentioned above, the residual generation unit 204 receives video data of the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. In order to generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0119] Transform processing unit 206 applies one or more transformations to the residual block to produce a transform coefficient block (referred to herein as the "transform coefficient block"). Transform processing unit 206 can apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply a Discrete Cosine Transform (DCT), a directional transformation, a Karhunen-Loeve Transform (KLT), or a conceptually similar transformation to the residual block. In some instances, transform processing unit 206 can perform multiple transformations on the residual block, such as primary and secondary transformations, such as rotation transformations. In some instances, transform processing unit 206 does not apply any transformations to the residual block.

[0120] When operating according to AV1, the transform processing unit 206 can apply one or more transforms to the residual block to produce a transform coefficient block (referred to herein as the "transform coefficient block"). The transform processing unit 206 can apply various transforms to the residual block to form the transform coefficient block. For example, the transform processing unit 206 can apply a combination of horizontal / vertical transforms, which may include the Discrete Cosine Transform (DCT), the Asymmetric Discrete Sine Transform (ADST), the Inverted ADST (e.g., the reversed ADST), and the Identity Transform (IDTX). When using the Identity Transform, the transform is skipped in either the vertical or horizontal direction. In some instances, transform processing can be skipped.

[0121] Quantization unit 208 can quantize the transform coefficients in a transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video transcoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce information loss, and therefore, the quantized transform coefficients may have lower precision than the original transform coefficients generated by transform processing unit 206.

[0122] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct the residual block from the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the prediction block generated by the mode selection unit 202 (although it may have some degree of distortion). For example, the reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples of the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0123] Filter unit 216 can perform one or more filter operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce blocky artifacts along the edges of the CU. In some instances, the operations of filter unit 216 can be skipped.

[0124] When operating according to AV1, filter unit 216 can perform one or more filter operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. In other instances, filter unit 216 can apply a constrained direction enhancement filter (CDEF), which can be applied after deblocking, and can include an inseparable nonlinear low-pass directional filter applied based on the estimated edge direction. Filter unit 216 can also include a loop recovery filter applied after CDEF, and can include a separable symmetric normalized Wiener filter or a dual-guided filter.

[0125] The video transcoder 200 stores reconstructed blocks in the DPB 218. For example, in an instance where the operation of the filter unit 216 is not performed, the reconstruction unit 214 may store reconstructed blocks in the DPB 218. In an instance where the operation of the filter unit 216 is performed, the filter unit 216 may store filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve a reference image from the DPB 218 for inter-frame prediction of subsequently encoded images, the reference image being formed from reconstructed (and possibly filtered) blocks. Furthermore, the intra-frame prediction unit 226 may use the reconstructed blocks in the DPB 218 of the current image to perform intra-frame prediction of other blocks in the current image.

[0126] Generally, the entropy coding unit 220 can entropy code syntax elements received from other functional components of the video transcoder 200. For example, the entropy coding unit 220 can entropy code quantized transform coefficient blocks from the quantization unit 208. As another example, the entropy coding unit 220 can entropy code predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame pattern information for intra-frame prediction) from the mode selection unit 202. The entropy coding unit 220 can perform one or more entropy coding operations on syntax elements to produce entropy-coded data, where the syntax elements are another instance of video data. For example, the entropy coding unit 220 can perform context-adjustable variable-length coding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adjustable binary arithmetic decoding (SBAC), probability interval partitioning entropy (PIPE) decoding, exponential Golomb coding, or other types of entropy coding operations on the data. In some instances, the entropy coding unit 220 can operate in a bypass mode where syntax elements are not entropy encoded.

[0127] The video transcoder 200 can output a bitstream containing the entropy-encoded syntax elements required to reconstruct a slice or image block. Specifically, the entropy encoding unit 220 can output a bitstream.

[0128] According to AV1, entropy coding unit 220 can be configured as a symbol-to-symbol self-adjusting multi-symbol arithmetic decoder. The syntax elements in AV1 consist of an N-element alphabet, while the context (e.g., a probability model) consists of a set of N probabilities. Entropy coding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy coding unit 220 can perform recursive scaling to update the context using an update factor based on letter size.

[0129] The above operations are described at the block level. This description should be understood as operations applied to the luma decoding block and / or the chroma decoding block. As mentioned earlier, in some instances, the luma decoding block and the chroma decoding block are the luma and chroma components of the CU. In some instances, the luma decoding block and the chroma decoding block are the luma and chroma components of the PU.

[0130] In some instances, it is not necessary to repeat the operations performed relative to the luma decoder for the chroma decoder block. As one example, the operations for identifying the motion vector (MV) and reference image of the luma decoder block do not need to be repeated for identifying the MV and reference image of the chroma block. Instead, the MV of the luma decoder block can be scaled to determine the MV of the chroma block, and the reference image can be the same. As another example, the in-frame prediction procedure can be the same for both the luma and chroma decoders.

[0131] Video transcoder 200 represents an example of a device configured to encode video data, the device including a memory configured to store the video data, and one or more processing units implemented in a circuit and configured to perform techniques related to multi-auxiliary line processing, such techniques including techniques for determining the number of auxiliary lines and techniques for determining a multi-auxiliary line candidate list.

[0132] Figure 4 is a block diagram illustrating an example video decoder 300 that can implement the technology of this application. Figure 4 is provided for illustrative purposes and does not limit the technology extensively illustrated and described in this application. For illustrative purposes, this application describes a video decoder 300 based on VVC (ITU-T H.266, under development) and HEVC (ITU-T H.265) technologies. However, the technology of this application can be implemented by video decoding devices configured for other video decoding standards.

[0133] In the example of Figure 4, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any one or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or processing circuits. For example, the units of the video decoder 300 can be implemented as one or more circuits or logic components, as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include supplementary or alternative processors or processing circuits to perform these and other functions.

[0134] The prediction processing unit 304 includes a motion compensation unit 316 and an in-frame prediction unit 318. The prediction processing unit 304 may include additional units to perform predictions according to other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an in-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other instances, the video decoder 300 may include more, fewer, or different functional components.

[0135] When operating according to AV1, compensation unit 316 can be configured to decode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using translational motion compensation, affine motion compensation, OBMC, and / or composite inter-frame intra-frame prediction, as described above. Intra-frame prediction unit 318 can be configured to decode the decoded blocks of video data (e.g., both luma and chroma decoded blocks) using directional intra-frame prediction, non-directional intra-frame prediction, recursive filter intra-frame prediction, CFL, intra-frame block copy (IBC), and / or palette mode, as described above.

[0136] CPB memory 320 can store video data to be decoded by components of video decoder 300, such as encoded video bitstreams. The video data stored in CPB memory 320 can be obtained, for example, from computer-readable media 110 (FIG. 1). CPB memory 320 may include a CPB storing encoded video data (e.g., syntax elements) from the encoded video bitstream. Furthermore, CPB memory 320 can store video data other than syntax elements of the decoded image, such as temporary data representing the output from various units of video decoder 300. DPB 314 typically stores decoded images that video decoder 300 can output and / or use as reference video data when decoding subsequent data or images of the encoded video bitstream. CPB memory 320 and DPB 314 can be formed from any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. CPB memory 320 and DPB 314 can be provided by the same memory device or separate memory devices. In various instances, CPB memory 320 can be on-chip along with other components of the video decoder 300, or off-chip relative to those components.

[0137] Alternatively or supplementarily, in some instances, the video decoder 300 may retrieve the decoded video data from memory 120 (FIG. 1). That is, memory 120 may store data as discussed above for CPB memory 320. Similarly, when some or all of the functions of the video decoder 300 are implemented in software for execution by the processing circuitry of the video decoder 300, memory 120 may store instructions to be executed by the video decoder 300.

[0138] The various units shown in Figure 4 are illustrated to explain the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to Figure 3, a fixed-function circuit is a circuit that provides a specific function and is pre-defined for the operations that can be performed. A programmable circuit is a circuit that can be programmed to perform various tasks and provides flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by software or firmware instructions. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some instances, one or more units can be different circuit blocks (fixed-function or programmable), and in some instances, one or more units can be integrated circuits.

[0139] The video decoder 300 may include an ALU, an EFU, digital circuitry, analog circuitry, and / or a programmable core formed by programmable circuitry. In an instance where the operation of the video decoder 300 is performed by software executed on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0140] Entropy decoding unit 302 can receive encoded video data from CPB and perform entropy decoding on the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on syntax elements extracted from the bitstream.

[0141] Generally, the video decoder 300 reconstructs the image on a block-by-block basis. The video decoder 300 can perform reconstruction operations on each block individually (where the block currently being reconstructed (i.e., decoded) can be referred to as the "current block").

[0142] Entropy decoding unit 302 can entropy decode the syntax elements of the quantized transform coefficients defining the quantized transform coefficient block, as well as transform information such as quantization parameters (QP) and / or transform mode indications. Inverse quantization unit 306 can use the QP associated with the quantized transform coefficient block to determine the quantization level, and similarly, determine the inverse quantization level to be applied by inverse quantization unit 306. For example, inverse quantization unit 306 can perform a bit-left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thereby form a transform coefficient block including the transform coefficients.

[0143] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse direction transform, or another inverse transform to the transform coefficient block.

[0144] Furthermore, the prediction processing unit 304 generates prediction blocks based on the prediction information syntax elements entropy decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is an inter-frame prediction, the motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements may indicate a reference image in the DPB 314 from which the reference block is retrieved, and a motion vector that identifies the position of the reference block in the reference image relative to the position of the current block in the current image. The motion compensation unit 316 can generally perform the inter-frame prediction procedure in a manner substantially similar to that described with respect to the motion compensation unit 224 (FIG. 3).

[0145] As another example, if the prediction information syntax element indicates that the current block is in-frame prediction, then the in-frame prediction unit 318 can generate a prediction block according to the in-frame prediction pattern indicated by the prediction information syntax element. Again, the in-frame prediction unit 318 can generally perform the in-frame prediction procedure in a manner substantially similar to that described with respect to the in-frame prediction unit 226 (Figure 3). The in-frame prediction unit 318 can retrieve data from the DPB 314 of samples adjacent to the current block.

[0146] The intraframe prediction unit 318 can also be configured to perform intraframe prediction using MRL processing. The intraframe prediction unit 318 can be configured to perform any of the MRL processing techniques described above. For example, the intraframe prediction unit 318 can be configured to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the intraframe prediction mode, and use multi-auxiliary-line processing to decode video data blocks based on the number of auxiliary lines.

[0147] Reconstruction unit 310 can use the prediction block and the residual block to reconstruct the current block. For example, reconstruction unit 310 can add samples of the residual block to the corresponding samples of the prediction block to reconstruct the current block.

[0148] Filter unit 312 can perform one or more filter operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce blocky artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily performed in all instances.

[0149] The video decoder 300 can store reconstructed blocks in the DPB 314. For example, in an instance where the operation of the filter unit 312 is not performed, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an instance where the operation of the filter unit 312 is performed, the filter unit 312 can store the filtered reconstructed blocks in the DPB 314. As mentioned above, the DPB 314 can provide reference information to the prediction processing unit 304, such as samples of the current image for in-frame prediction and samples of previously decoded images for subsequent motion compensation. Furthermore, the video decoder 300 can output decoded images (e.g., decoded video) from the DPB 314 for subsequent display on a display device such as the display device 118 of FIG. 1.

[0150] In this way, video decoder 300 represents an instance of a video decoding device, which includes a memory configured to store video data, and one or more processing units implemented in a circuit and configured to perform techniques related to multi-auxiliary line processing, including techniques for determining the number of auxiliary lines and techniques for determining a multi-auxiliary line candidate list.

[0151] Figure 5 is a flowchart illustrating an example method for encoding the current block according to the technology of this case. The current block may include the current CU. Although described with respect to the video transcoder 200 (Figures 1 and 3), it should be understood that other devices can be configured to perform a method similar to that of Figure 5.

[0152] In this example, the video transcoder 200 initially predicts the current block (350). For example, the video transcoder 200 may form a prediction block for the current block. As part of forming the prediction block, the video transcoder 200 may perform any of the MRL processing techniques described above. The video transcoder 200 may then compute the residual block of the current block (352). To compute the residual block, the video transcoder 200 may compute the difference between the original uncoded block and the prediction block of the current block. The video transcoder 200 may then transform the residual block and quantize the transform coefficients of the residual block (354). Next, the video transcoder 200 may scan the quantized transform coefficients of the residual block (356). During or after scanning, the video transcoder 200 may entropy encode the transform coefficients (358). For example, the video transcoder 200 may encode the transform coefficients using CAVLC or CABAC. The video transcoder 200 can then output blocks of entropy-encoded data (360).

[0153] Figure 6 is a flowchart illustrating an example method for decoding a current block of video data according to the technology of this case. The current block may include the current CU. Although video decoder 300 (Figures 1 and 4) has been described, it should be understood that other devices can be configured to perform methods similar to those in Figure 6.

[0154] The video decoder 300 can receive entropy-coded data of the current block, such as entropy-coded prediction information and entropy-coded data of the transform coefficients of the residual block corresponding to the current block (370). The video decoder 300 can entropy decode the entropy-coded data to determine the prediction information of the current block and reproduce the transform coefficients of the residual block (372). The video decoder 300 can predict the current block (374), for example, using the intra-frame or inter-frame prediction mode indicated by the prediction information of the current block to calculate the prediction block of the current block. For example, when using an intra-frame prediction decoding block, the video decoder 300 can perform any of the above-described MRL processing techniques. The video decoder 300 can then perform a backscan on the reproduced transform coefficients (376) to build a block of quantized transform coefficients. The video decoder 300 can then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to generate a residual block (378). The video decoder 300 can ultimately decode the current block (380) by combining the prediction block and the residual block.

[0155] Figure 7 is a flowchart illustrating another example method of decoding the current block according to the technique of this case. The technique of Figure 7 can be performed by one or more structural units of the video transcoder 200 and / or the video decoder 300, including the in-frame prediction unit 226 and / or the in-frame prediction unit 318.

[0156] In one instance of this case, the video transcoder 200 and / or the video decoder 300 may be configured to determine the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction mode (700), and to use multi-auxiliary line processing to decode (e.g., encode or decode) video data blocks based on the number of auxiliary lines (702).

[0157] In one instance, the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or one of combined intra-frame / intra-frame prediction (CIIP).

[0158] In another example, to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the in-frame prediction mode, the video transcoder 200 and / or the video decoder 300 are also configured to determine the first number of auxiliary lines used for general in-frame prediction and the second number of auxiliary lines used for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the first number of auxiliary lines differs from the second number of auxiliary lines. In one example, the second number of auxiliary lines is less than the first number of auxiliary lines.

[0159] In another instance of this case, the video transcoder 200 and / or the video decoder 300 are further configured to constrain the number of auxiliary lines relative to the decoding tree unit boundary of the decoding tree unit that includes the video data block.

[0160] In another instance of this case, the video transcoder 200 and / or the video decoder 300 are further configured to disable one or more transformation types for a specific auxiliary line among the number of auxiliary lines.

[0161] In another instance of this case, the video transcoder 200 and / or the video decoder 300 are further configured to determine the number of auxiliary lines based on the size of the video data blocks.

[0162] In another instance of this case, the video transcoder 200 and / or the video decoder 300 are also configured to determine the multi-auxiliary line candidate list based on the in-frame prediction mode and the number of auxiliary lines.

[0163] In another instance of this case, the video decoder 300 is also configured to decode a multi-auxiliary line index that indicates a specific auxiliary line in a multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of auxiliary lines, and the video decoder 300 uses the specific auxiliary line for decoding.

[0164] Similarly, in the opposite instance of this case, the video transcoder 200 is also configured to determine the specific auxiliary lines used to encode the video data block, and to encode the multi-auxiliary line index indicating the specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of auxiliary lines.

[0165] Other illustrative aspects of this case are described below.

[0166] State Sample 1A - A method for decoding video data, the method comprising: decoding a syntax element indicating the number of auxiliary lines used for multi-auxiliary line processing; and decoding a block of video data using multi-auxiliary line processing based on the number of auxiliary lines.

[0167] State Sample 2A - A method for decoding video data, the method comprising: implicitly determining the number of auxiliary lines to be used for multi-auxiliary line processing based on the differences between two or more auxiliary lines; and decoding a block of video data using multi-auxiliary line processing based on the number of auxiliary lines.

[0168] State Sample 3A - A method for decoding video data, the method comprising: implicitly determining the number of auxiliary lines to be used for multi-auxiliary line processing based on in-frame prediction patterns; and decoding video data blocks using multi-auxiliary line processing based on the number of auxiliary lines.

[0169] The method of any one of the states 4A-1A-3A also includes: determining a candidate list of multiple auxiliary lines based on the intra-frame prediction mode, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation and template-based intra-frame mode derivation.

[0170] The method of any one of the states 5A-1A-4A also includes: disabling one or more transformation types for a specific auxiliary line.

[0171] The method of any one of state samples 6A-1A-5A also includes: determining a candidate list of multiple auxiliary lines based on the decoding unit size.

[0172] The method of any one of the states 7A-1A-6A also includes: the number of constraint auxiliary lines relative to the boundary of the decoding tree unit.

[0173] The method for any one of states 8A-1A-7A, wherein decoding includes decoding.

[0174] The method of any one of state 9A-state 1A-7A, wherein decoding includes encoding.

[0175] Sample 10A - An apparatus for decoding video data, the apparatus comprising one or more units for performing the method of any one of Samples 1A-9A.

[0176] Devices of state 11A-10A, wherein the one or more units include one or more processors implemented in a circuit.

[0177] The device of any one of states 12A-10A and 11A also includes memory for storing video data.

[0178] The device of any one of the forms 13A-10A-12A also includes a display configured to display decoded video data.

[0179] The device of any one of 14A-10A-13A, wherein the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device or a set-top box.

[0180] The device of any one of the following: 15A-10-14A, wherein the device includes a video decoder.

[0181] The device of any one of the following: 16A-10A-15A, wherein the device includes a video transcoder.

[0182] Sample 17A - A computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to perform any one of Samples 1A-9A.

[0183] State Sample 1B - A method for decoding video data, the method comprising: determining the number of auxiliary lines to be used for multi-auxiliary line processing based on an in-frame prediction mode; and using the multi-auxiliary line processing to decode a block of video data based on the number of auxiliary lines.

[0184] The method of state 2B-state 1B, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or one of combined inter-frame / intra-frame prediction (CIIP).

[0185] The method of state sample 3B-state sample 1B, wherein determining the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction mode includes: determining the number of first auxiliary lines for general in-frame prediction; and determining the number of second auxiliary lines for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the number of first auxiliary lines is different from the number of second auxiliary lines.

[0186] The method of state 4B-state 3B, wherein the number of the second auxiliary lines is less than the number of the first auxiliary lines.

[0187] The method of state 5B-state 1B also includes: constraining the number of auxiliary lines relative to the boundary of the decoding tree unit that includes the video data block.

[0188] The method of state 6B-state 1B also includes: disabling one or more transformation types for a specific auxiliary line among the number of auxiliary lines.

[0189] The method of state 7B-state 1B also includes: further determining the number of auxiliary lines based on the size of the video data block.

[0190] The method for state 8B-state 1B also includes: determining a candidate list of multiple auxiliary lines based on the prediction pattern within the frame and the number of auxiliary lines.

[0191] The method of state 9B-state 8B also includes: decoding a multi-auxiliary line index that indicates a specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of auxiliary lines; and decoding the video data block using the specific auxiliary line.

[0192] The method of state 10B-state 1B also includes: displaying an image including the decoded video data block.

[0193] Type 11B - An apparatus configured to decode video data, the apparatus comprising: a memory configured to store a block of video data; and one or more processors implemented in a circuit and communicating with the memory, the one or more processors being configured to: determine, based on an in-frame prediction mode, the number of auxiliary lines to be used for multi-auxiliary-line processing; and, based on the number of auxiliary lines, decode the block of video data using the multi-auxiliary-line processing.

[0194] The apparatus of state 12B-state 11B, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or one of combined inter-frame / intra-frame prediction (CIIP).

[0195] The apparatus of states 13B-11B, wherein, in order to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the in-frame prediction mode, the one or more processors are also configured to: determine the number of first auxiliary lines for general in-frame prediction; and determine the number of second auxiliary lines for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the number of first auxiliary lines is different from the number of second auxiliary lines.

[0196] The apparatus of state 14B-state 13B, wherein the number of the second auxiliary lines is less than the number of the first auxiliary lines.

[0197] Version 15B - The apparatus according to Version 11B, wherein the one or more processors are also configured to constrain the number of auxiliary lines relative to the boundaries of the decoding tree units comprising video data blocks.

[0198] State 16B - The apparatus according to State 11B, wherein the one or more processors are further configured to disable one or more transformation types for a specific auxiliary line among the number of auxiliary lines.

[0199] Version 17B - The apparatus according to Version 11B, wherein the one or more processors are also configured to: further determine the number of auxiliary lines based on the size of the video data block.

[0200] Version 18B - The apparatus according to Version 11B, wherein the one or more processors are also configured to determine a multi-auxiliary-line candidate list based on the prediction pattern within the frame and the number of auxiliary lines.

[0201] The apparatus of states 19B-18B, wherein the one or more processors are also configured to: decode a multi-auxiliary line index indicating a specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of auxiliary lines; and decode the video data block using the specific auxiliary line.

[0202] The device of type 20B-type 11B, wherein the device is a wireless communication device, the device also includes: a display configured to display an image including decoded video data blocks.

[0203] Format 21B - An apparatus configured to decode video data, the apparatus comprising: a unit for determining the number of auxiliary lines to be used for multi-auxiliary-line processing based on an in-frame prediction mode; and a unit for decoding a block of video data using the multi-auxiliary-line processing based on the number of auxiliary lines.

[0204] Sample 22B - A non-transitory computer-readable storage medium storing instructions, which, when executed, cause one or more processors of a device configured to decode video data to: determine the number of auxiliary lines to be used for multi-auxiliary line processing based on an in-frame prediction mode; and, based on the number of auxiliary lines, use the multi-auxiliary line processing to decode a block of video data.

[0205] Sample 23B - An apparatus configured to encode video data, the apparatus comprising: a memory configured to store blocks of video data; and one or more processors implemented in a circuit and communicating with the memory, the one or more processors being configured to: determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on an in-frame prediction mode; and encode the blocks of video data using the multi-auxiliary-line processing based on the number of auxiliary lines.

[0206] The apparatus of states 24B-23B, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or one of combined inter-frame / intra-frame prediction (CIIP).

[0207] The apparatus of states 25B-23B, wherein, in order to determine the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction mode, the one or more processors are also configured to: determine the number of first auxiliary lines for general in-frame prediction; and determine the number of second auxiliary lines for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the number of first auxiliary lines is different from the number of second auxiliary lines.

[0208] The apparatus of state 26B-state 25B, wherein the number of the second auxiliary lines is less than the number of the first auxiliary lines.

[0209] Version 27B - The apparatus according to Version 23B, wherein the one or more processors are further configured to constrain the number of auxiliary lines relative to the boundaries of the decoding tree units comprising video data blocks.

[0210] State 28B - The apparatus according to State 23B, wherein the one or more processors are further configured to disable one or more transformation types for a specific auxiliary line among the number of auxiliary lines.

[0211] Version 29B - The apparatus according to Version 23B, wherein the one or more processors are further configured to: further determine the number of auxiliary lines based on the size of the video data block.

[0212] The apparatus of state 30B-state 23B, wherein the one or more processors are also configured to determine a multi-auxiliary line candidate list based on the prediction mode within the frame and the number of auxiliary lines.

[0213] The apparatus of states 31B-30B, wherein the one or more processors are also configured to: determine a specific auxiliary line for encoding the video data block; and encode a multi-auxiliary line index indicating a specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of the auxiliary lines.

[0214] The apparatus of states 32B-23B, wherein the apparatus is a wireless communication device, the apparatus also includes: a camera configured to capture an image including the video data block.

[0215] A method for decoding video data, comprising: determining the number of auxiliary lines to be used for multi-auxiliary line processing based on an in-frame prediction mode; and using the multi-auxiliary line processing to decode a block of video data based on the number of auxiliary lines.

[0216] The method of state 2C-state 1C, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or a combination of inter-frame / intra-frame prediction (CIIP).

[0217] The method of any one of the states 3C-1C-2C, wherein determining the number of auxiliary lines to be used for multi-auxiliary line processing based on the in-frame prediction mode includes: determining the number of first auxiliary lines for general in-frame prediction; and determining the number of second auxiliary lines for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the number of first auxiliary lines is different from the number of second auxiliary lines.

[0218] The method of state 4C-state 3C, wherein the number of the second auxiliary lines is less than the number of the first auxiliary lines.

[0219] The method of any one of state 5C-state 1C-4C also includes: constraining the number of auxiliary lines relative to the boundary of the decoding tree unit that includes the video data block.

[0220] The method of any one of the states 6C-1C-5C also includes: disabling one or more transformation types for a specific auxiliary line among the number of auxiliary lines.

[0221] The method of any one of the states 7C-1C-6C also includes: further determining the number of auxiliary lines based on the size of the video data block.

[0222] The method of any one of state sample 8C-state sample 1C-6C also includes: determining a candidate list of multiple auxiliary lines based on the prediction pattern within the frame and the number of auxiliary lines.

[0223] The method of state 9C-state 8C also includes: decoding a multi-auxiliary line index that indicates a specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of the auxiliary lines; and decoding the video data block using the specific auxiliary line.

[0224] The method of any one of the states 10C-1C-9C also includes: displaying an image including the decoded video data block.

[0225] Type 11C - An apparatus configured to decode video data, the apparatus comprising: a memory configured to store a block of video data; and one or more processors implemented in a circuit and communicating with the memory, the one or more processors being configured to: determine, based on an in-frame prediction mode, the number of auxiliary lines to be used for multi-auxiliary-line processing; and, based on the number of auxiliary lines, decode the block of video data using the multi-auxiliary-line processing.

[0226] The apparatus of state 12C-state 11C, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or a combination of inter-frame / intra-frame prediction (CIIP).

[0227] An apparatus of any one of states 13C-11C-12C, wherein, in order to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the in-frame prediction mode, the one or more processors are also configured to: determine a first number of auxiliary lines for general in-frame prediction; and determine a second number of auxiliary lines for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the first number of auxiliary lines is different from the second number of auxiliary lines.

[0228] The apparatus of state 14C-state 13C, wherein the number of the second auxiliary lines is less than the number of the first auxiliary lines.

[0229] Version 15C - The apparatus according to any one of versions 11C-14C, wherein the one or more processors are also configured to constrain the number of auxiliary lines relative to the boundaries of the decoding tree units comprising video data blocks.

[0230] State 16C - The apparatus according to any one of states 11C-15C, wherein the one or more processors are also configured to disable one or more transformation types for a particular auxiliary line among the number of auxiliary lines.

[0231] The apparatus according to any one of the following: 17C-16C, wherein the one or more processors are also configured to further determine the number of auxiliary lines based on the size of the video data block.

[0232] State 18C - The apparatus according to any one of states 11C-16C, wherein the one or more processors are also configured to: determine a multi-auxiliary line candidate list based on the prediction mode within the frame and the number of auxiliary lines.

[0233] The apparatus of state 19C-state 18C, wherein the one or more processors are also configured to: decode a multi-auxiliary line index indicating a specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of auxiliary lines; and decode the video data block using the specific auxiliary line.

[0234] The apparatus according to any one of the following specifications 20C-11C-19C, wherein the apparatus is a wireless communication device, the apparatus also includes: a display configured to display an image including decoded video data blocks.

[0235] Type 21C - An apparatus configured to encode video data, the apparatus comprising: a memory configured to store blocks of video data; and one or more processors implemented in a circuit and communicating with the memory, the one or more processors being configured to: determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on an in-frame prediction mode; and encode the blocks of video data using the multi-auxiliary-line processing based on the number of auxiliary lines.

[0236] The apparatus of state 22C-state 21C, wherein the intra-frame prediction mode includes general intra-frame prediction, decoder-side intra-frame mode derivation, template-based intra-frame mode derivation, and position-dependent intra-frame prediction combination mode, or a combination of inter-frame / intra-frame prediction (CIIP).

[0237] An apparatus of any one of states 23C-21C-22C, wherein, in order to determine the number of auxiliary lines to be used for multi-auxiliary-line processing based on the in-frame prediction mode, the one or more processors are also configured to: determine a first number of auxiliary lines for general in-frame prediction; and determine a second number of auxiliary lines for decoder-side in-frame mode derivation and template-based in-frame mode derivation, wherein the first number of auxiliary lines is different from the second number of auxiliary lines.

[0238] The apparatus of state 24C-state 23C, wherein the number of the second auxiliary lines is less than the number of the first auxiliary lines.

[0239] The apparatus of any one of the states 25C to 21C-24C, wherein the one or more processors are also configured to constrain the number of auxiliary lines relative to the boundaries of the decoding tree units comprising video data blocks.

[0240] The apparatus of any one of the states 26C-21C-25C, wherein the one or more processors are also configured to disable one or more transformation types for a particular auxiliary line among the number of auxiliary lines.

[0241] The apparatus according to any one of the following: 27C-21C-26C, wherein the one or more processors are also configured to further determine the number of auxiliary lines based on the size of the video data block.

[0242] Apparatus 28C - Any one of Apparatus 21C-26C, wherein the one or more processors are also configured to determine a multi-auxiliary-line candidate list based on the prediction mode within the frame and the number of auxiliary lines.

[0243] The apparatus of states 29C-28C, wherein the one or more processors are also configured to: determine a specific auxiliary line to be used for encoding the video data block; and encode a multi-auxiliary line index indicating a specific auxiliary line in the multi-auxiliary line candidate list, wherein the multi-auxiliary line index has a maximum value based on the number of the auxiliary lines.

[0244] The apparatus according to any one of the following: 30C-21C-29C, wherein the apparatus is a wireless communication device, the apparatus also includes: a camera configured to capture an image including the video data block.

[0245] It should be recognized that, depending on the instance, certain actions or events of any technique described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the technique). Furthermore, in some instances, actions or events may be performed concurrently, for example via multithreaded processing, interrupt handling, or multiple processors, rather than sequentially.

[0246] In one or more instances, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to tangible media such as data storage media, or a communication medium, including any medium that facilitates, for example, the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures to implement the technology described herein. A computer program product may include a computer-readable medium.

[0247] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, magnetic disks and optical disks include compact magnetic disks (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where magnetic disks typically reproduce data magnetically, while optical discs reproduce data optically via lasers. Combinations of these should also be included within the scope of computer-readable media.

[0248] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or individual logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some cases, the functions described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined transcoder. Moreover, these techniques can be fully implemented in one or more circuit or logic elements.

[0249] The technology described herein can be implemented in various devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this application to emphasize the functional characteristics of devices configured to perform the disclosed technology, but they do not necessarily need to be implemented by different hardware units. Rather, as mentioned above, various units can be combined in a transcoder hardware unit, or provided by a collection of interactive operating hardware units including one or more processors as described above, along with suitable software and / or firmware.

[0250] Various instances have been described. These and other instances fall within the scope of the attached request items.

[0251] 100: Video Encoding and Decoding System 102: Source device 104: Video Source 106: Memory 108: Output Interface 110: Computer-readable media 112: Storage device 114: Archive Server 116: Destination Equipment 118: Display device 120: Memory 122: Input Interface 200: Video transcoder 202: Mode Selection Unit 204: Residual Generation Unit 206: Transformation Processing Unit 208: Quantization unit 210: Inverse quantization unit 212: Inverse Transformation Processing Unit 214: Reconfiguration Unit 216: Filter Unit 218: Decoding Image Buffer 220: Entropy Coding Unit 222: Motion Estimation Unit 224: Motion Compensation Unit 226: In-frame prediction unit 230: Video Data Memory 300: Video Decoder 302: Entropy Decoding Unit 304: Predictive Processing Unit 306: Inverse quantization unit 308: Inverse Transformation Processing Unit 310: Reconfiguration Unit 312: Filter Unit 314: Decoded Picture Buffer (DPB) 316: Motion Compensation Unit 318: In-frame prediction unit 320: CPB Memory 350: Square 352: Square 354: Square 356: Square 358: Square 360: Square 370: Square 372: Square 374: Square 376: Square 378: Square 380: Square 400: Decoding Unit 402: Auxiliary Line 404: Auxiliary lines 406: Auxiliary lines 408: Auxiliary lines 700: Square 702: Square

[0252] Domestic storage information (please note in order of storage institution, date, and number) none Overseas storage information (please note in the order of storage country, institution, date, and number) none

Claims

1. A method for decoding video data, the method comprising the steps of: determining a first intra-frame prediction mode for decoding a first block; determining a first plurality of candidate auxiliary lines for multi-auxiliary line processing of the first block based on the first intra-frame prediction mode; determining a first auxiliary line among the first plurality of candidate auxiliary lines; decoding video data of the first block using the first auxiliary line; determining a second intra-frame prediction mode for decoding a second block, wherein the second intra-frame prediction mode is different from the first intra-frame prediction mode; determining a second plurality of candidate auxiliary lines for multi-auxiliary line processing of the second block based on the second intra-frame prediction mode, wherein the second plurality of candidate auxiliary lines is different from the first plurality of candidate auxiliary lines; determining a second auxiliary line among the second plurality of candidate auxiliary lines; and decoding video data of the second block using the second auxiliary line.

2. The method of request item 1, wherein the second in-frame prediction mode includes one of the following: decoder-side in-frame mode derivation or template-based in-frame mode derivation.

3. The method of request item 2, wherein the number of the second plurality of candidate auxiliary lines is less than the number of the first plurality of candidate auxiliary lines.

4. The method of claim 1, wherein the first plurality of candidate auxiliary lines includes a first fixed set of candidate auxiliary lines, and the second plurality of candidate auxiliary lines includes a second fixed set of candidate auxiliary lines.

5. The method of request item 1 also includes the following steps: disabling one or more transformation types for at least one of the first plurality of candidate auxiliary lines or the second plurality of candidate auxiliary lines.

6. The method of request item 1 also includes the following steps: further determining the first plurality of candidate auxiliary lines based on a size of the video data of the first block.

7. The method of request item 1 also includes the steps of: determining the first auxiliary line based on a first multi-auxiliary line index associated with the first plurality of candidate auxiliary lines; and determining the second auxiliary line based on a second multi-auxiliary line index associated with the second plurality of candidate auxiliary lines.

8. The method of request item 1 also includes the following steps: Displaying an image including video data of the decoded first block.

9. The method of claim 8 also includes the steps of: determining the first auxiliary line based on a first multi-auxiliary line index associated with the first plurality of candidate auxiliary lines; and determining the second auxiliary line based on a second multi-auxiliary line index associated with the second plurality of candidate auxiliary lines.

10. An apparatus configured to decode video data, the apparatus comprising: A memory configured to store a block of video data; and one or more processors implemented in a circuit and communicating with the memory, the processors being configured to: determine a first in-frame prediction mode for decoding a first block; determine a first plurality of candidate auxiliary lines for multi-auxiliary line processing of the first block based on the first in-frame prediction mode; determine a first auxiliary line among the first plurality of candidate auxiliary lines; decode the video data of the first block using the first auxiliary line; determine a second in-frame prediction mode for decoding a second block, wherein the second in-frame prediction mode is different from the first in-frame prediction mode; determine a second plurality of candidate auxiliary lines for multi-auxiliary line processing of the second block based on the second in-frame prediction mode, wherein the second plurality of candidate auxiliary lines is different from the first plurality of candidate auxiliary lines; determine a second auxiliary line among the second plurality of candidate auxiliary lines; and decode the video data of the second block using the second auxiliary line.

11. The apparatus of claim 10, wherein the second in-frame prediction mode includes one of the following: decoder-side in-frame mode derivation or template-based in-frame mode derivation.

12. The apparatus of claim 11, wherein the number of the second plurality of candidate auxiliary lines is less than the number of the first plurality of candidate auxiliary lines.

13. The apparatus of claim 10, wherein the first plurality of candidate auxiliary lines includes a first fixed set of candidate auxiliary lines, and the second plurality of candidate auxiliary lines includes a second fixed set of candidate auxiliary lines.

14. The apparatus of claim 10, wherein the one or more processors are also configured to: disable one or more transformation types for at least one of the first plurality of candidate auxiliary lines or the second plurality of candidate auxiliary lines.

15. The apparatus of claim 10, wherein the one or more processors are also configured to: further determine the first plurality of candidate auxiliary lines based on a size of the video data of the first block.

16. The apparatus of claim 10, wherein the apparatus is a wireless communication device, the apparatus also comprising: A display is configured to display an image including the video data of the decoded first block.

17. An apparatus configured to decode video data, the apparatus comprising: A unit used to determine the prediction mode within a first frame for decoding a first block; Units for determining a first plurality of candidate auxiliary lines that can be used for multi-auxiliary line processing of the first block based on the first intra-frame prediction mode; units for determining a first auxiliary line from the first plurality of candidate auxiliary lines; units for decoding video data of the first block using the first auxiliary line; units for determining a second intra-frame prediction mode to decode a second block, wherein the second intra-frame prediction mode is different from the first intra-frame prediction mode; units for determining a second plurality of candidate auxiliary lines that can be used for multi-auxiliary line processing of the second block based on the second intra-frame prediction mode, wherein the second plurality of candidate auxiliary lines are different from the first plurality of candidate auxiliary lines; units for determining a second auxiliary line from the second plurality of candidate auxiliary lines; and units for decoding video data of the second block using the second auxiliary line.

18. The apparatus of claim 17, wherein the second in-frame prediction mode includes either decoder-side in-frame mode derivation or template-based in-frame mode derivation.

19. The apparatus of claim 18, wherein the number of the second plurality of candidate auxiliary lines is less than the number of the first plurality of candidate auxiliary lines.

20. A non-transitory computer-readable storage medium storing instructions, which, when executed, cause one or more processors of a device configured to decode video data to perform the following steps: determining a first intra-frame prediction mode for decoding a first block; determining a first plurality of candidate auxiliary lines for multi-auxiliary line processing of the first block based on the first intra-frame prediction mode; determining a first auxiliary line among the first plurality of candidate auxiliary lines; decoding video data of the first block using the first auxiliary line; determining a second intra-frame prediction mode for decoding a second block, wherein the second intra-frame prediction mode is different from the first intra-frame prediction mode; determining a second plurality of candidate auxiliary lines for multi-auxiliary line processing of the second block based on the second intra-frame prediction mode, wherein the second plurality of candidate auxiliary lines is different from the first plurality of candidate auxiliary lines; determining a second auxiliary line among the second plurality of candidate auxiliary lines; and decoding video data of the second block using the second auxiliary line.

21. The non-transitory computer-readable storage medium of request item 20, wherein the second in-frame prediction mode includes either decoder-side in-frame mode derivation or template-based in-frame mode derivation.

22. The non-transitory computer-readable storage medium of request item 21, wherein the number of the second plurality of candidate auxiliary lines is less than the number of the first plurality of candidate auxiliary lines.

23. An apparatus configured to encode video data, the apparatus comprising: A memory configured to store a block of video data; and one or more processors implemented in a circuit and communicating with the memory, the processors being configured to: determine a first in-frame prediction mode for encoding a first block; determine a first plurality of candidate auxiliary lines for multi-auxiliary line processing of the first block based on the first in-frame prediction mode; determine a first auxiliary line among the first plurality of candidate auxiliary lines; encode video data of the first block using the first auxiliary line; determine a second in-frame prediction mode for encoding a second block, wherein the second in-frame prediction mode is different from the first in-frame prediction mode; determine a second plurality of candidate auxiliary lines for multi-auxiliary line processing of the second block based on the second in-frame prediction mode, wherein the second plurality of candidate auxiliary lines is different from the first plurality of candidate auxiliary lines; determine a second auxiliary line among the second plurality of candidate auxiliary lines; and encode video data of the second block using the second auxiliary line.

24. The apparatus of claim 23, wherein the second in-frame prediction mode includes one of the following: decoder-side in-frame mode derivation or template-based in-frame mode derivation.

25. The apparatus of claim 24, wherein the number of the second plurality of candidate auxiliary lines is less than the number of the first plurality of candidate auxiliary lines.

26. The apparatus of claim 23, wherein the first plurality of candidate auxiliary lines includes a first fixed set of candidate auxiliary lines, and the second plurality of candidate auxiliary lines includes a second fixed set of candidate auxiliary lines.

27. The apparatus of claim 23, wherein the one or more processors are also configured to: disable one or more transformation types for at least one of the first plurality of candidate auxiliary lines or the second plurality of candidate auxiliary lines.

28. The apparatus of claim 23, wherein the one or more processors are also configured to: further determine the first plurality of candidate auxiliary lines based on a size of the video data of the first block.

29. The apparatus of claim 23, wherein the one or more processors are also configured to: encode a first multi-auxiliary line index indicating the first auxiliary line; and encode a second multi-auxiliary line index indicating the second auxiliary line.

30. The apparatus of claim 23, wherein the apparatus is a wireless communication device, the apparatus also comprising: A camera configured to capture an image including the first block of video data.

Images