Candidate list construction method, video encoding and decoding method, video encoding and decoding device and video encoding and decoding system

Abstract

A multi-reference-line intra-frame prediction mode candidate list construction method, a video encoding and decoding method, a video encoding and decoding device and a video encoding and decoding system are characterized in that a template region is predicted according to N * M combinations of N extended reference lines and M intra-frame prediction modes, and the intra-frame prediction mode candidate list is obtained according to an ascending order of errors between reconstruction values and prediction values of the template region. And filling the K combinations into a candidate list of the TMRLIP mode. And when the coding end selects the combination in the candidate list for the current block, the TMRLIP mode mark and the TMRLIP mode index are coded, and the decoding end determines the extended reference line and the intra-frame prediction mode selected for the current block by decoding the TMRLIP mode mark and the TMRLIP mode index. According to the embodiment of the invention, the extended reference line and the intra-frame prediction mode are combined for coding and decoding, the coding efficiency can be improved, and the coding performance is improved.

Description

A candidate list construction method, video encoding and decoding method, device and system

[0001] Cross-references

[0002] This application claims priority to PCT international application No. PCT / CN2022 / 103946, filed on July 5, 2022, entitled “A candidate list construction method, video encoding and decoding method, device and system”. The entire contents of the PCT international application are incorporated into this application by this reference. Technical Field

[0003] The embodiments of the present disclosure relate to, but are not limited to, video technology, and more specifically, to a method for constructing a multi-reference row intra-frame prediction mode candidate list, a video encoding and decoding method, device, and system. Background Art

[0004] Digital video compression technology primarily compresses large amounts of digital video data for easier transmission and storage. Currently popular video codec standards, such as H.266 / Versatile Video Coding (VVC), employ a block-based hybrid coding framework. Each video frame is divided into square largest coding units (LCUs) of equal size (e.g., 128x128, 64x64, etc.). Each LCU can be divided into rectangular coding units (CUs) based on a specific rule. Coding units may also be divided into prediction units (PUs) and transform units (TUs). The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-frame prediction and inter-frame prediction, which are used to reduce or remove inherent redundancy in the video. Intra-frame blocks are predicted using neighboring pixels as reference, while inter-frame blocks use information from spatially adjacent blocks and reference information from other frames. In contrast to the prediction signal, the residual information is transformed, quantized, and entropy-encoded into a bitstream on a block-by-block basis. These techniques are described in standards and implemented in various fields related to video compression.

[0005] With the surge in Internet videos and people's increasing demand for video clarity, although existing digital video compression standards can save a lot of video data, there is still a need to pursue better digital video compression technology to reduce the bandwidth and traffic pressure of digital video transmission.

[0006] SUMMARY OF THE INVENTION

[0007] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0008] An embodiment of the present disclosure provides a method for constructing a multi-reference row intra prediction mode candidate list, including:

[0009] Obtain N×M combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block, where N≥1, M≥1, and N×M≥2;

[0010] Predicting the template area of ​​the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template area and the predicted value;

[0011] In order of the errors from small to large, K combinations corresponding to the errors are filled into a candidate list of the template-based multi-reference row intra prediction TMRL_IP mode of the current block, where 1≤K≤N×M.

[0012] An embodiment of the present disclosure further provides a video decoding method, including:

[0013] Decode the multi-reference line intra prediction MRL_IP mode flag of the current block to determine whether the current block uses the MRL_IP mode;

[0014] When it is determined that the current block uses the MRL_IP mode, continue decoding the MRL_IP mode index of the current block and construct a candidate list of the MRL_IP mode of the current block, where the candidate list is filled with combinations of extended reference rows and intra prediction modes of the current block candidates;

[0015] Determining a combination of an extended reference row and an intra prediction mode selected for the current block based on the candidate list and the MRL_IP mode index, and predicting the current frame based on the selected combination;

[0016] The TMRL_IP mode index is used to indicate the position of the selected combination of the extended reference row and the intra prediction mode in the candidate list.

[0017] An embodiment of the present disclosure further provides a video encoding method, including:

[0018] Constructing a candidate list of a multiple reference line intra prediction MRL_IP mode for the current block, wherein the candidate list is filled with combinations of extended reference lines and intra prediction modes of the current block candidates;

[0019] Through rate-distortion optimization, a combination of reference row and intra prediction mode is selected for intra prediction of the current block;

[0020] When the encoding condition of the MRL_IP mode of the current block is satisfied, encoding the MRL_IP mode flag of the current block to indicate that the current block uses the MRL_IP mode, and encoding the MRL_IP mode index of the current block to indicate the position of the selected combination in the candidate list;

[0021] The encoding condition at least includes: the selected combination is in the candidate list.

[0022] An embodiment of the present disclosure further provides a code stream, wherein the code stream includes block-level syntax elements for intra-frame prediction, and the syntax elements include a multiple reference row intra-frame prediction MRL_IP mode flag and an MRL_IP mode index of the current block, wherein the MRL_IP mode flag is used to indicate whether the current block uses the MRL_IP mode, and the MRL_IP mode index is used to indicate the position of the combination of extended reference row and intra-frame prediction mode selected for the current block in a candidate list of the MRL_IP mode.

[0023] An embodiment of the present disclosure also provides a device for constructing a multi-reference row intra-frame prediction mode candidate list, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the method for constructing a multi-reference row intra-frame prediction mode candidate list as described in any embodiment of the present disclosure.

[0024] An embodiment of the present disclosure further provides a video decoding device, comprising a processor and a memory storing a computer program, wherein the processor can implement the video decoding method as described in any embodiment of the present disclosure when executing the computer program.

[0025] An embodiment of the present disclosure further provides a video encoding device, including a processor and a memory storing a computer program, wherein the processor can implement the video encoding method as described in any embodiment of the present disclosure when executing the computer program.

[0026] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes the video encoding device described in any embodiment of the present disclosure and the video decoding device described in any embodiment of the present disclosure.

[0027] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method for constructing a multi-reference row intra-frame prediction mode candidate list as described in any embodiment of the present disclosure, or implements the video decoding method as described in any embodiment of the present disclosure, or implements the video encoding method as described in any embodiment of the present disclosure.

[0028] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0029] Summary of the Figures

[0030] The accompanying drawings are used to provide an understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation to the technical solutions of the present disclosure.

[0031] FIG1A is a schematic diagram of a coding and decoding system according to an embodiment of the present disclosure;

[0032] FIG1B is a framework diagram of an encoding end according to an embodiment of the present disclosure;

[0033] FIG1C is a framework diagram of a decoding end according to an embodiment of the present disclosure;

[0034] FIG2 is a schematic diagram of an intra-frame prediction mode according to an embodiment of the present disclosure;

[0035] FIG3 is a schematic diagram of adjacent intra-frame prediction blocks of a current block according to an embodiment of the present disclosure;

[0036] FIG4 is a schematic diagram of a template of a current block and a template reference area according to an embodiment of the present disclosure;

[0037] FIG5 is a schematic diagram of multiple reference rows around a current block according to an embodiment of the present disclosure;

[0038] FIG6 is a flowchart of a video encoding method according to an embodiment of the present disclosure;

[0039] FIG7 is a flowchart of a method for constructing an MRL_IP mode candidate list according to an embodiment of the present disclosure;

[0040] FIG8A is a schematic diagram of a template area around a current block and an extended reference row according to an embodiment of the present disclosure;

[0041] FIG8B is a schematic diagram of a template area around a current block and an extended reference row according to another embodiment of the present disclosure;

[0042] FIG9 is a flowchart of a video decoding method according to an embodiment of the present disclosure;

[0043] FIG10 is a schematic diagram of an apparatus for constructing an MRL_IP mode candidate list according to an embodiment of the present disclosure.

[0044] Details

[0045] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure.

[0046] In the description of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be interpreted as being more preferred or advantageous than other embodiments. "And / or" in this article is a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "Multiple" refers to two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0047] When describing representative exemplary embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can readily understand that these sequences can vary and still remain within the spirit and scope of the disclosed embodiments.

[0048] The local illumination compensation method and video coding and decoding method proposed in the embodiments of the present disclosure can be applied to various video coding and decoding standards, such as: H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), AVS (Audio Video Coding Standard), and other standards developed by MPEG (Moving Picture Experts Group), AOM (Alliance for Open Media), JVET (Joint Video Experts Team) and extensions of these standards, or any other customized standards.

[0049] FIG1A is a block diagram of a video encoding and decoding system that can be used in an embodiment of the present disclosure. As shown in the figure, the system is divided into an encoding end device 1 and a decoding end device 2. The encoding end device 1 generates a code stream. The decoding end device 2 can decode the code stream. The decoding end device 2 can receive the code stream from the encoding end device 1 via a link 3. The link 3 includes one or more media or devices capable of moving the code stream from the encoding end device 1 to the decoding end device 2. In one example, the link 3 includes one or more communication media that enable the encoding end device 1 to send the code stream directly to the decoding end device 2. The encoding end device 1 modulates the code stream according to a communication standard (e.g., a wireless communication protocol) and sends the modulated code stream to the decoding end device 2. The one or more communication media may include wireless and / or wired communication media and may form part of a packet network. In another example, the code stream can also be output from the output interface 15 to a storage device, and the decoding end device 2 can read the stored data from the storage device via streaming or downloading.

[0050] As shown in the figure, the encoding end device 1 includes a data source 11, a video encoding device 13 and an output interface 15. The data source 11 includes a video capture device (such as a camera), an archive containing previously captured data, an input interface for receiving data from a content provider, a computer graphics system for generating data, or a combination of these sources. The video encoding device 13 encodes the data from the data source 11 and outputs it to the output interface 15. The output interface 15 may include at least one of a regulator, a modem and a transmitter. The decoding end device 2 includes an input interface 21, a video decoding device 23 and a display device 25. The input interface 21 includes at least one of a receiver and a modem. The input interface 21 can receive a code stream via the link 3 or from a storage device. The video decoding device 23 decodes the received code stream. The display device 25 is used to display the decoded data. The display device 25 can be integrated with other devices of the decoding end device 2 or set separately. The display device 25 is optional for the decoding end. In other examples, the decoding end may include other devices or equipment that apply the decoded data.

[0051] Based on the video coding and decoding system shown in FIG. 1A , various video coding and decoding methods can be used to implement video compression and decompression.

[0052] FIG1B is a block diagram of an exemplary video encoding device that can be used in embodiments of the present disclosure. As shown, the video encoding device 1000 includes a prediction unit 1100, a partitioning unit 1101, a residual generation unit 1102 (indicated by the circle with a plus sign after the partitioning unit 1101 in the figure), a transform processing unit 1104, a quantization unit 1106, an inverse quantization unit 1108, an inverse transform processing unit 1110, a reconstruction unit 1112 (indicated by the circle with a plus sign after the inverse transform processing unit 1110 in the figure), a filter unit 1113, a decoded image buffer 1114, and an entropy coding unit 1115. The prediction unit 1100 includes an inter-frame prediction unit 1121 and an intra-frame prediction unit 1126. The decoded image buffer 1114 may also be referred to as a decoded image buffer, a decoded picture buffer, or a decoded picture buffer. The video encoder 20 may also include more, fewer, or different functional components than shown in this example. For example, the transform processing unit 1104 and the inverse transform processing unit 1110 may be eliminated in some cases.

[0053] The division unit 1101 cooperates with the prediction unit 1100 to divide the received video data into slices, coding tree units (CTUs) or other larger units. The video data received by the division unit 1101 can be a video sequence including video frames such as I frames, P frames or B frames.

[0054] The prediction unit 1100 can divide a CTU into coding units (CUs) and perform intra-frame prediction coding or inter-frame prediction coding on the CU. When performing intra-frame prediction and inter-frame prediction on the CU, the CU can be divided into one or more prediction units (PUs).

[0055] The inter-frame prediction unit 1121 can perform inter-frame prediction on the PU and generate prediction data for the PU, which includes the prediction block of the PU, the motion information of the PU, and various syntax elements. The inter-frame prediction unit 1121 can include a motion estimation (ME) unit and a motion compensation (MC) unit. The motion estimation unit can be used to estimate motion to generate a motion vector, and the motion compensation unit can be used to obtain or generate a prediction block based on the motion vector.

[0056] The intra-prediction unit 1126 may perform intra prediction on the PU and generate prediction data for the PU. The prediction data for the PU may include a prediction block and various syntax elements for the PU.

[0057] The residual generation unit 1102 may subtract the prediction blocks of the PUs into which the CU is partitioned from the original blocks of the CU to generate the residual blocks of the CU.

[0058] The transform processing unit 1104 may divide a CU into one or more transform units (TUs). The division of a prediction unit (PU) and a transform unit (TU) may be different. A TU-associated residual block is a sub-block obtained by dividing the residual block of the CU. One or more transforms are applied to the TU-associated residual block to generate a TU-associated coefficient block.

[0059] The quantization unit 1106 may quantize the coefficients in the coefficient block based on the selected quantization parameter. The degree of quantization of the coefficient block may be adjusted by adjusting the quantization parameter (QP: Quantizer Parameter).

[0060] The inverse quantization unit 1108 and the inverse transform unit 1110 may apply inverse quantization and inverse transform to the coefficient block, respectively, to obtain a reconstructed residual block associated with the TU.

[0061] The reconstruction unit 1112 may add the reconstructed residual block and the prediction block generated by the prediction unit 1100 to generate a reconstructed image.

[0062] Filter unit 1113 performs loop filtering on the reconstructed image and stores the filtered reconstructed image in decoded image buffer 1114 as a reference image. Intra-frame prediction unit 1126 extracts reference images of blocks adjacent to the PU from decoded image buffer 1114 to perform intra-frame prediction. Inter-frame prediction unit 1121 can use the reference image of the previous frame cached in decoded image buffer 1114 to perform inter-frame prediction on the PU of the current frame image.

[0063] The entropy coding unit 1115 may perform entropy coding operations on received data (eg, syntax elements, quantized coefficient blocks, motion information, etc.).

[0064] FIG1C is a block diagram of an exemplary video decoding device that can be used in embodiments of the present disclosure. As shown, the video decoding device 101 includes an entropy decoding unit 150, a prediction unit 152, an inverse quantization unit 154, an inverse transform unit 156, a reconstruction unit 158 ​​(represented by the circle with a plus sign after the inverse transform unit 155), a filter unit 159, and a decoded image buffer 160. In other embodiments, the video decoder 30 may include more, fewer, or different functional components. For example, the inverse transform unit 155 may be eliminated in some cases.

[0065] Entropy decoding unit 150 may perform entropy decoding on the received bitstream to extract syntax elements, quantized coefficient blocks, motion information for PUs, etc. Prediction unit 152, inverse quantization unit 154, inverse transform processing unit 156, reconstruction unit 158, and filter unit 159 may each perform corresponding operations based on the syntax elements extracted from the bitstream.

[0066] The inverse quantization unit 154 may perform inverse quantization on the coefficient block associated with the quantized TU.

[0067] Inverse transform processing unit 156 may apply one or more inverse transforms to the inverse quantized coefficient block to generate a reconstructed residual block for the TU.

[0068] Prediction unit 152 includes inter-prediction unit 162 and intra-prediction unit 164. If a PU is encoded using intra prediction, intra prediction unit 164 may determine an intra prediction mode for the PU based on syntax elements decoded from the codestream, perform intra prediction based on the determined intra prediction mode and reconstructed reference information of the PU's neighbors obtained from decoded picture buffer 160, and generate a prediction block for the PU. If a PU is encoded using inter prediction, inter prediction unit 162 may determine one or more reference blocks for the PU based on the motion information of the PU and corresponding syntax elements, and generate a prediction block for the PU based on the reference blocks obtained from decoded picture buffer 160.

[0069] The reconstruction unit 158 ​​may obtain a reconstructed image based on the reconstructed residual block associated with the TU and the prediction block of the PU generated by the prediction unit 152 .

[0070] The filter unit 159 may perform loop filtering on the reconstructed image, and the filtered reconstructed image is stored in the decoded image buffer 160. The decoded image buffer 160 may provide a reference image for subsequent motion compensation, intra-frame prediction, inter-frame prediction, etc., and may also output the filtered reconstructed image as decoded video data for presentation on a display device.

[0071] Based on the above-described video encoding and decoding devices, the following basic encoding and decoding process can be performed. On the encoding side, a frame of an image is divided into blocks. Intra-frame prediction, inter-frame prediction, or other algorithms are performed on the current block to generate a predicted block for the current block. The predicted block is subtracted from the original block of the current block to obtain a residual block. The residual block is transformed and quantized to obtain quantized coefficients, and the quantized coefficients are entropy encoded to generate a bitstream. On the decoding side, intra-frame prediction or inter-frame prediction is performed on the current block to generate a predicted block for the current block. The quantized coefficients obtained from the decoded bitstream are then inversely quantized and inversely transformed to obtain a residual block. The predicted block and residual block are added together to obtain a reconstructed block. The reconstructed block forms a reconstructed image. The reconstructed image is then subjected to image-based or block-based loop filtering to obtain a decoded image. The encoding side also performs similar operations as the decoding side to obtain a decoded image, which is often referred to as a reconstructed image. The decoded image obtained by the encoding side can serve as a reference frame for inter-frame prediction of subsequent frames. The block division information determined by the encoding side, as well as information and parameters for prediction, transform, quantization, entropy coding, and loop filtering modes, can be written into the bitstream if necessary. The decoding end determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information and parameter information as the encoding end by decoding the code stream or analyzing the existing information, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end.

[0072] Although the above is an example of a block-based hybrid coding framework, the embodiments of the present disclosure are not limited thereto. With the development of technology, one or more modules in the framework and one or more steps in the process can be replaced or optimized.

[0073] Herein, the current block may be a block-level coding unit such as a current coding unit (CU) or a current prediction unit (PU) in the current image.

[0074] During intra-frame prediction, the encoder typically uses various angular and non-angular modes to predict the current block, generating a predicted block. Based on the rate-distortion information calculated between the predicted block and the original block, the encoder selects the optimal intra-frame prediction mode for the current block. This intra-frame prediction mode is encoded and transmitted via the bitstream to the decoder. The decoder decodes the selected intra-frame prediction mode and uses it to perform intra-frame prediction for the current block. Throughout the development of digital video codec standards, non-angular modes have remained relatively stable, including the mean mode (DC mode) and the planar mode (Planar mode). Angular modes have continued to increase with the evolution of digital video codec standards. For example, the H series of international digital video coding standards, H.264 / AVC, includes only eight traditional angular prediction modes and one traditional non-angular prediction mode. H.265 / HEVC expands this to 33 traditional angular prediction modes and two traditional non-angular prediction modes. In H.266 / VVC, the traditional intra-frame prediction modes are Planar mode, DC mode, and 65 angular modes, as shown in Figure 2. The DC mode is suitable for large flat areas, and its prediction value is obtained by calculating the average value of the reference pixels on the left and / or above. The Planar mode is suitable for pixel gradients, that is, for areas where pixel values ​​change slowly.

[0075] If the intra prediction mode of the current block is directly encoded, 7 bits are required to encode each of the 67 modes, resulting in a very large amount of data. Based on statistical characteristics, pixels closer to the current block are more likely to use the same intra prediction mode as the current block. This characteristic is why most probable mode (MPM) technology is adopted in HEVC, VVC, and the Enhanced Compression Model (ECM). ECM is based on the VTM-10.0 reference software and integrates various new tools to further explore codec performance.

[0076] MPM first constructs an MPM list and populates it with the six intra prediction modes most likely to be selected by the current block. If the intra prediction mode selected for the current block is in the MPM list, only its index number (only 3 bits) needs to be encoded. If the intra prediction mode selected for the current block is not in the MPM list but is one of the 61 non-MPM modes, the intra prediction mode is encoded using truncated binary code (TBC) during the entropy coding stage.

[0077] In VVC, regardless of whether multiple reference lines (MRL) and intra sub-block partitioning (ISP) are applied, the MPM list has 6 prediction modes. The MPM in ECM is divided into MPM and Secondary MPM, which use lists of length 6 and length 16 respectively. Of the 6 modes in the MPM list, the Planar mode is always filled in the first position in the MPM. The remaining 5 positions are filled in the following three steps in sequence until all 5 positions are filled. The extra modes will automatically enter the Secondary MPM.

[0078] In the first step, the intra prediction modes used by the prediction blocks at the five adjacent positions around the current block are sequentially filled; as shown in Figure 3, the five positions include the upper left (AL), upper (A), upper right (AR), left (L) and lower left (BL) positions of the current block.

[0079] In the second step, the pattern derived from the gradient histogram is used based on the reconstructed pixels around the current block;

[0080] In the third step, select an angle mode that is close to the angle mode selected in the first step.

[0081] The Secondary MPM list may consist of some main angle modes other than the intra prediction modes in the MPM.

[0082] Since the encoding and decoding order of the MPM flag (mpm_flag) is after the MRL mode, the encoding and decoding of the MPM in ECM depends on the MRL flag. When the current block does not use the MRL mode, the MPM flag needs to be decoded to determine whether the current block uses the MPM. When the current block uses the MRL mode, there is no need to decode the MPM flag, and the current block uses the MPM by default.

[0083] The following table is an example of MPM-related syntax elements:

[0084]

[0085] Template-based intra mode derivation (TIMD) is an intra prediction mode for luma frames. TIMD is generated from candidate intra prediction modes in the MPM list and a template region (template). In ECM, as shown in Figure 4, the left and upper neighboring regions of a current block (e.g., the current CU) 11 constitute the template region 12 of the current block. The left neighboring region is called the left template region (left template), and the upper neighboring region is called the upper template region or upper template region (upper template).

[0086] As shown in the figure, a template reference area 13 is provided outside (left and top of) template area 12. Exemplary sizes and positions of each area are shown. In one example, the width L1 of the left template and the height L2 of the top template are both 4. Template reference area 13 can be located in a row above or a column to the left of the template area.

[0087] TIMD assumes that the distribution characteristics of the current block and the template area of ​​the current block are consistent. The reconstructed value of the template reference area is used as the reconstructed value of the reference row. The template area is predicted by traversing all intra-frame prediction modes in the MPM and Secondary MPM to obtain a prediction result. The error between the reconstructed value on the template area and the prediction result of each mode is then calculated and expressed as the sum of absolute transformed differences (SATD). The optimal intra-frame prediction mode with the smallest SATD is selected and used as the TIMD mode for the current block. The decoding end can derive the TIMD mode using the same derivation method. If the sequence allows the use of TIMD, each current block requires a flag bit to indicate whether TIMD is used. If the intra-frame prediction mode selected for the current block is TIMD mode, the current block is predicted using TIMD mode, and the decoding process of the remaining syntax elements related to intra-frame prediction, such as ISP and MPM, can be skipped, thereby significantly reducing the coding bits of the mode.

[0088] After obtaining the SATD between the reconstructed value of the template area and the prediction result of each mode (the predicted value of the template area), the final TIMD mode to be used can be determined according to the following method:

[0089] Assume that mode1 and mode2 are two angle modes for intra-frame prediction in MPM, mode1 is the angle mode with the smallest SATD, and its SATD is cost1; mode2 is the angle mode with the second smallest SATD, and its SATD is cost2:

[0090] When cost1×2≤cost2, mode1 is used as the TIMD mode of the current block;

[0091] When cost1×2>cost2, the prediction mode obtained by weighting the prediction results of mode1 and mode2 is used as the TMID mode of the current block, also known as the TIMD fusion mode.

[0092] The weighting method and weights are shown in the following formula:

[0093] Pred=Pred mode1 ×w1+Pred mode2 ×w2

[0094]

[0095] Among them, Pred is the prediction result of the current block using TIMD fusion mode, Pred mode1 Use the prediction result of mode1 for the current block, Pred mode2 The prediction result of mode2 is used for the current block, and w1 and w2 are the weights calculated according to cost1 and cost2.

[0096] In HEVC, intra prediction uses the row and column immediately above and to the left of the current block as references. If the reconstructed values ​​of these rows and columns differ significantly from the original pixel values, the prediction quality of the current block will be significantly affected. To address this issue, VVC adopts the MRL intra prediction technique. In addition to using the nearest row of reference pixels (reference line 0), VVC can also use reference lines with indexes 1 (reference line 1) and 2 (reference line 2) as extended reference lines for intra prediction. To reduce coding complexity, MRL is only used in non-planar modes within the MPM. When predicting each angular mode, the encoder tries all three reference lines. Through rate-distortion optimization, it selects the reference line with the lowest rate-distortion cost (RD cost) for the current block. The index of this selected reference line is encoded and sent to the decoder. The decoder decodes the reference line index and then uses it to determine the selected reference line for the current block for prediction.

[0097] The example shown in FIG5 shows four reference lines for the current block, including reference line 0 221 adjacent to the current block, i.e., a reference line with an index of 0; reference line 1 222 one line away from the current block, i.e., a reference line with an index of 1; reference line 2 223 two lines away from the current block, i.e., a reference line with an index of 2; and reference line 3 224 three lines away from the current block, i.e., a reference line with an index of 3. The current block can have more reference lines, i.e., reference lines with an index of 4 or more. Prediction can use only the reconstructed values ​​of a portion of the reference lines. In this document, the reference line indices are numbered as shown in FIG5 .

[0098] In this article, although the reference row is called a "row", this is for convenience of expression. A reference row actually includes one row and one column. Generally, the reconstructed value of the reference row used in prediction also includes the reconstructed value of one row and one column, which is the same as the usual description method in the industry.

[0099] In one example, the template area of ​​the current block is set on a reference row with an index of 0, then the reference row with an index of 0 is referred to as the reference row where the template area is located, and the reference rows with indices of 1 to 3 are referred to as reference rows outside the template area. If the template area of ​​the current block is set on reference rows with indices of 0 and 1, then the reference rows with indices of 0 and 1 are the reference rows where the template area is located, and the reference rows with indices of 2 and 3 are reference rows outside the template area. In this article, the reference rows of the current block include the reference row with an index of 0 and the reference rows with an index greater than 0, and the reference rows with an index greater than 0 are referred to as extended reference rows. In the case where the reference row where the template area is located is the reference row with an index of 0, all extended reference rows are reference rows outside the template area. In the case where there are multiple reference rows where the template area is located, the reference rows where the template area is located include extended reference rows.

[0100] In ECM, decoding methods related to multiple reference lines are shown in Tables 1 and 2. Table 1 is used when the current block does not use TIMD, and Table 2 is used when the current block uses TIMD.

[0101] Table 1

[0102]

[0103] The MRL index can be encoded using a unary truncated code based on a context model, and after encoding, multiple binary identifiers based on the context model are obtained. The binary identifiers can also be called binary identifier bits, binary symbols, binary bits, etc.

[0104] In ECM, the MRL mode can use more reference lines. In order to encode the reference line selected for the current block, the indexes of multiple candidate reference lines are filled into a list. In this paper, this list is called the multiple reference line index list, abbreviated as the MRL index list, but it can also be called the multiple reference line list, the candidate reference line list, the reference line index list, etc. When the current block does not use TIMD, the length of the MRL index list is 6, that is, there are 6 positions in total, which can be filled with the indexes of 6 reference lines. The indexes of these 6 reference lines and their order are fixed, which are 0, 1, 3, 5, 7, 12 respectively, and can be expressed by the following formula: MULTI_REF_LINE_IDX[6] = {0, 1, 3, 5, 7, 12}. In this multi-reference row index list, the index filled in the first position is 0, which is the index of the reference row closest to the current block. The indexes filled in the second to sixth positions are 1, 3, 5, 7, and 12, respectively, which are the indexes of the five extended reference rows arranged in order from near to far according to the distance to the current block.

[0105] In the case where the reference row selected for the current block is in the MRL index list, a multi-reference row index (multiRefIdx) is used to represent the position of the reference row selected for the current block in the MRL index list, and the selected reference row can be represented by encoding the multi-reference row index. Take the MRL index list {0,1,3,5,7,12} as an example, where the indexes of the 1st to 6th positions are 0 to 5 respectively. Assuming that the reference row with index 0 is selected for the current block, and the reference row with index 0 is at the 1st position in the MRL index list, the multi-reference row index is 0. Assuming that the reference row with index 7 is selected for the current block, and the reference row with index 7 is at the 5th position in the MRL index list, the multi-reference row index is 4, and so on for other cases.

[0106] The multi-reference row index can be encoded using a unary truncated code based on a context model or the like. The smaller the value of the multi-reference row index, the shorter the code length and the faster the decoding.

[0107] The MRL mode can also be used simultaneously with the TIMD mode. When using TIMD, the MRL index list length is 3, which can be filled with the indexes of 3 reference lines and the order between the indexes is fixed, expressed as MULTI_REF_LINE_IDX[3] = {0, 1, 3}. The corresponding multi-reference line encoding and decoding method is shown in Table 2:

[0108]

[0109] For ease of understanding, the above text divides MRL into two tables for examples when using TIMD and when not using TIMD, but these two tables can also be combined into one table.

[0110] In different standards, technologies like MPM, which use blocks surrounding the current block to derive a list of most probable modes, may have different names. For example, in AV2 (AVM), it's called Adaptive Intra Mode Coding (AIMC), and in AVS3, when used for screen content coding, it's called Frequency-based Intra Mode Coding (FIMC). In non-screen content coding, technologies like MPM are often used. In contrast, technologies like MRL, which use multiple reference lines for intra prediction, are called Multiple Reference Line Selection for Intra Prediction (MRLS) in AV2 (AVM). However, these are merely different names; when this embodiment uses terms like MPM and MRL, these essentially identical technologies in other standards should also be covered.

[0111] In ECM, when the selected mode is not TIMD, the use of MRL technology is bundled with the non-planar mode in MPM. Up to 5 extended reference lines and 5 intra prediction modes in MPM are combined, for a total of 25 combinations. For these 25 combinations, up to 3 equally probable binary flags (MPM syntax elements) and 6 binary flags of context models (1 is an MPM syntax element and 5 are MRL syntax elements) are required, for a total of 9 binary flags, which is more expensive to encode and decode.

[0112] If the current block is less than 13 rows above the upper boundary of the CTU, the reference row with index 12 cannot be used. At this time, there are 20 combinations of the 5 intra-frame prediction modes in MPM and the 4 available extended reference rows in MRL. If the current block is less than 8 rows above the upper boundary of the CTU, the reference row with index 7 cannot be used. At this time, there are 15 combinations of the 5 intra-frame prediction modes in MPM and the 3 available extended reference rows in MRL. If the current coding block is less than 6 rows above the upper boundary of the CTU, the reference row with index 5 cannot be used. At this time, there are 10 combinations of the 5 intra-frame prediction modes in MPM and the 4 available extended reference rows in MRL.

[0113] This means that although the codec flags support 25 combinations of extended reference lines in MRL and intra-frame prediction modes in MPM, some combinations cannot be used when the number of reference lines above the current block is insufficient. Additional flags are still needed to support these combinations. The 9 binary flags used in MPM and MRL are not fully utilized, and the coding performance needs to be improved.

[0114] Although the extended reference row and intra-frame prediction mode need to be combined when performing intra-frame prediction on the current block, during encoding and decoding, the above methods all construct the MRL index list and MPM list respectively, and determine the multi-reference row index and MPM index of the current block respectively according to the reference row and intra-frame prediction mode selected for the current block, and encode and decode the multi-reference row index and MPM index respectively, so that the decoding end can determine the reference row and intra-frame prediction mode selected for the current block.

[0115] Different from the above method, the embodiment of the present disclosure provides a multiple reference line intra prediction mode, abbreviated as MRL_IP (Multiple reference line & intra_intra prediction) mode, which is a method of encoding and decoding a combination of extended reference lines and intra prediction modes.

[0116] The video encoding method according to an embodiment of the present disclosure is shown in FIG6 , including:

[0117] Step 110 , constructing a candidate list of multiple reference line intra prediction MRL_IP modes for the current block, wherein the candidate list is filled with combinations of candidate extended reference lines and intra prediction modes for the current block;

[0118] Step 120 , selecting a reference row and intra prediction mode combination for intra prediction for the current block through rate-distortion optimization;

[0119] The reference lines include a reference line with an index of 0 and an extended reference line. The combination of a reference line and an intra-frame prediction mode selected for the current block may be a combination of a reference line with an index of 0 and an intra-frame prediction mode, or a combination of an extended reference line and an intra-frame prediction mode.

[0120] Step 130: When the encoding condition of the MRL_IP mode of the current block is satisfied, encode the MRL_IP mode flag of the current block to indicate that the current block uses the MRL_IP mode, and encode the MRL_IP mode index of the current block to indicate the position of the selected combination in the candidate list;

[0121] The encoding condition at least includes: the selected combination is in the candidate list.

[0122] In this paper, the candidate list is populated with combinations of extended reference rows and intra prediction modes for the current block candidate. This means that the combinations in the candidate list must participate in the rate-distortion optimization of the current block, that is, the mode selection process that selects the current block prediction mode based on the rate-distortion cost. This ensures that all combinations in the candidate list have a chance of being selected.

[0123] In this article, N, M, and K, which represent the number of parameters, are all positive integers and need no additional explanation.

[0124] The candidate list of the MRL_IP mode constructed by the embodiment of the present disclosure is filled with the combination of the extended reference row and the intra-frame prediction mode of the current block candidate, and is no longer a single list of candidate extended reference rows or a list of candidate intra-frame prediction modes. When the combination of the reference row and the intra-frame prediction mode selected for the current block is in the candidate list (in this case, the extended reference row is selected for the current block), that is, the selected combination is a combination in the candidate list and other coding conditions are met, the MRL_IP mode flag of the current block is encoded to indicate that the current block uses the MRL_IP mode, and the MRL_IP mode index of the current block is encoded to indicate the position of the selected combination in the candidate list; the decoding end can determine the extended reference row and intra-frame prediction mode selected for the current block based on the MRL_IP mode flag and the MRL_IP mode index. The combined coding and decoding method of this embodiment can reduce the coding cost and improve the coding performance.

[0125] In an exemplary embodiment of the present disclosure, the candidate list for the MRL_IP mode is a template-based multiple reference line intra prediction mode candidate list constructed according to the method of the embodiment of the present disclosure. The "template-based multiple reference line intra prediction mode" is referred to herein as the TMRL_IP mode, abbreviated as the TMRL mode. The TMRL_IP mode is an MRL_IP mode that predicts and sorts combinations based on a template. The MRL_IP mode, however, is a mode that encodes and decodes combinations of extended reference lines and intra prediction modes for the current block candidate into a candidate list and is not limited to the TMRL_IP mode.

[0126] The TMRL_IP mode is a prediction algorithm that uses N×M combinations of N extended reference lines and M intra prediction modes for the current block to predict the template region of the current block. The error between the reconstructed value of the template region and the predicted value is calculated. The corresponding K combinations are added to the candidate list of the TMRL_IP mode for the current block in ascending order of error, where 1≤K≤N×M, where K is a set value. In one example, the template region is predicted based on 25 combinations of extended reference lines and intra prediction modes in the ECM. These 25 combinations are sorted in ascending order of error, and the 12 combinations with the smallest error, i.e., the most likely selection, are added to the candidate list of the TMRL_IP mode. This allows the TMRL_IP mode index to be encoded using fewer coding bits. Even if some of the extended reference lines {1, 3, 5, 7, 12} (e.g., extended reference lines with indices 7 and 12) are outside the CTU boundary, the coding bits of the TMRL_IP mode index can still be fully and efficiently utilized by sorting the 15 combinations and adding the 12 most likely selections to the candidate list. Sorting in ascending order of error allows reference rows and prediction modes that are more likely to be selected for prediction to be retained in the candidate list. The more likely combinations are ranked higher in the list, reducing the coding cost.

[0127] The creation of the candidate list for TMRL_IP mode will be described in more detail below.

[0128] In another exemplary embodiment of the present disclosure, the candidate list for constructing the multi-reference row intra-frame prediction MRL_IP mode of the current block is not sorted based on the template, but is created in other ways. For example, based on the statistical results of a large amount of sample data, the combination of the reference row and the intra-frame prediction mode with the highest probability of being selected is filled into the candidate list of the MRL_IP mode. The candidate list of the MRL_IP mode can only be filled with the combination of the extended reference row and the intra-frame prediction mode, but it is not necessary to do so. The candidate list can also be filled with the combination of the reference row with index 0 and other intra-frame prediction modes. If the selected combination is the combination of the reference row with index 0 in the candidate list and the intra-frame prediction mode, the selected combination can also be indicated by encoding the MRL_IP mode index.

[0129] In an exemplary embodiment of the present disclosure, the encoding condition further includes: the current block does not use the template-based intra prediction mode to derive the TIMD; the method further includes:

[0130] When the current block uses TIMD, the encoding of the MRL_IP mode flag and MRL_IP mode index of the current block is skipped;

[0131] When the current block does not use TIMD but the selected combination is not in the candidate list, the MRL_IP mode flag of the current block is encoded to indicate that the current block does not use the MRL_IP mode, and the encoding of the MRL_IP mode index of the current block is skipped.

[0132] This embodiment is based on the situation that the TIMD mode is encoded and decoded before the MRL_IP mode. If the current block uses the TIMD mode, there is no need to use the MRL_IP mode, so the encoding of the MRL_IP mode flag and the MRL_IP mode index is skipped. If the current block does not use the TIMD mode, there are two situations: the combination selected for the current block is in the candidate list of the MRL_IP mode, or it is not in the candidate list of the MRL_IP mode, if the selected combination is not in the candidate list. The MRL_IP mode flag needs to be encoded to indicate that the current block does not use the MRL_IP mode, and the encoding of the MRL_IP mode index is skipped. If the selected combination is in the candidate list, both the MRL_IP mode flag and the MRL_IP mode index need to be encoded.

[0133] When the current block does not use TIMD, the MRL_IP mode flag and MRL_IP mode index provided in this embodiment can replace the original multi-reference row index multiRefIdx. However, when the current block uses TIMD, the multi-reference row index can still be used to represent the selected reference row and encode the multi-reference row index.

[0134] In an exemplary embodiment of the present disclosure, constructing a candidate list for the MRL_IP mode of the current block includes: constructing the candidate list only when all conditions for allowing the current block to use the MRL_IP mode are met, and the conditions for allowing the use of the MRL_IP mode include any one or more of the following:

[0135] Condition 1: the current block is a block in a luma frame; that is, the MRL_IP mode is only used for luma frames (i.e., luma images), but the present disclosure is not limited thereto;

[0136] Condition 2: The current block is not located at the upper boundary of the coding tree unit (CTU). If it is located at the upper boundary of the CTU, no reference row can be used above the current block. Therefore, this embodiment uses not being located at the upper boundary of the CTU as a condition for allowing the use of the MRL_IP mode.

[0137] Condition 3: The current block allows the use of multiple reference lines (MRL), that is, the MRL_IP mode is allowed only when the multiple reference line mode is allowed.

[0138] Condition 4: The size of the current block is not larger than the maximum size of the current block for which the MRL_IP mode can be used; this maximum size can be preset. Larger blocks are generally flatter and less likely to have angular details. For such large blocks, the use of the MRL_IP mode can be limited.

[0139] Step 5: The aspect ratio of the current block meets the requirement of the MRL_IP mode for the aspect ratio of the current block. For example, the MRL_IP mode is allowed only when the aspect ratio of the current block is not greater than a preset value.

[0140] In an exemplary embodiment of the present disclosure, the MRL_IP pattern index is encoded using the Columbus-Rice coding method. Columbus coding can more reasonably classify candidate combinations into categories with different codeword lengths for encoding and decoding, thereby improving coding efficiency.

[0141] Taking K=12 and the divisor as 4 as an example, assuming the number to be encoded is n, n∈[0,12], calculate the prefix prefix and suffix suffix according to the following formula:

[0142] Suffix: suffix = n-prefix × 4.

[0143] The prefix uses a truncated unary code based on the context model to encode and decode each binary identification bit. The prefix value and output bits are shown in the following table.

[0144] Prefix output bits 00110211

[0145] The suffix uses a truncated binary code with equal probability to encode and decode each binary identifier. The suffix value and output bits are shown in the following table:

[0146] suffix output bits 000101210311

[0147] When the K value is different, the calculation method of the prefix and suffix can be adjusted according to the size of the K value, such as setting a different divisor. Although in the above example, the prefix uses a truncated unary code based on the context model and the suffix uses a truncated binary code with equal probability, in other examples, the prefix can also use a truncated unary code with equal probability and the suffix can use a truncated binary code based on the context model, and so on. This disclosure does not impose specific limitations on this.

[0148] In an exemplary embodiment of the present disclosure, when the encoding conditions of the MRL_IP mode of the current block are met, the method also includes: skipping the encoding of syntax elements of any one or more of the following modes: MPM mode, intra-frame sub-block partitioning ISP mode, multi-transform selection MTS mode, low-frequency non-divisible transform LFNST mode, TIMD mode.

[0149] For example, when the MRL_IP mode is placed before the TIMD mode for encoding and decoding, the encoding conditions of the MRL_IP mode no longer include that the current block does not use TIMD. When the current block uses the MRL_IP mode, the current block is not allowed to use TIMD, and the encoding of the TIMD mode syntax elements can be skipped.

[0150] For example, when the current block uses the MRL_IP mode, the MRL_IP mode flag and the MRL_IP mode index can simultaneously indicate the reference row and intra prediction mode selected by the current block. In this case, there is no need to encode or decode the MPM-related syntax elements.

[0151] For example, in a specific transform mode, the MRL_IP mode may be restricted from being used simultaneously with the multiple transform selection (MTS) mode and / or the low frequency non-separable transform (LFNST) mode.

[0152] An embodiment of the present disclosure provides a method for constructing a TMRL_IP mode candidate list, as shown in FIG7 , including:

[0153] Step 210 , obtaining N×M combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block, where N≥1, M≥1, and N×M≥2;

[0154] Step 220 , predicting the template region of the current block according to the N×M combinations, and calculating the error between the reconstructed value of the template region and the predicted value;

[0155] The error of this step can be expressed by the sum of absolute difference (SAD) or the sum of absolute transformed difference (SATD), but is not limited to this. It can also be expressed by the sum of squared difference (SSD), mean absolute difference (MAD), mean squared error (MSE), etc.

[0156] Step 230 , filling K combinations corresponding to the errors into a candidate list of the template-based multi-reference row intra prediction TMRL_IP mode for the current block in ascending order of the errors, 1≤K≤N×M.

[0157] When constructing a candidate list of TMRL_IP modes for the current block, this embodiment obtains N×M combinations based on N extended reference lines and M intra-frame prediction modes, predicts the template area of ​​the current block based on the N×M combinations, calculates the error between the reconstructed value and the predicted value of the template area, and fills the corresponding K combinations into the candidate list in ascending order of error. The candidate list created by this embodiment can realize the combined encoding of extended reference lines and intra-frame prediction modes, thereby improving encoding efficiency. Moreover, by using different combinations to predict the template area and sorting the errors, it is possible to select K combinations with a greater likelihood of being selected from the N×M combinations based on the similarity in distribution characteristics between the current block and the current block template area, and put the combinations with a greater likelihood of being selected at the front of the candidate list, so that the TMRL_IP mode index of the selected combination during encoding is smaller, thereby reducing the actual encoding cost.

[0158] In an exemplary embodiment of the present disclosure, the template region of the current block is set on a reference line closest to the current block; alternatively, the template region of the current block is set on multiple reference lines closest to the current block, and the N extended reference lines are extended reference lines located outside the template region. In FIG8A , the template region of the current block is set on reference line 30 with an index of 0. In the candidate list for constructing the TMRL_IP mode, N available extended reference lines are selected from predefined extended reference lines with indices {1, 3, 5, 7, 12}. If there are more than 13 reference lines between the top of the current block and the CTU boundary, five extended reference lines with indices {1, 3, 5, 7, 12} are selected for combination. If there are six or seven reference lines between the top of the current block and the CTU boundary, three extended reference lines with indices {1, 3, 5} are selected for combination, and so on.

[0159] Figure 8A shows five extended reference rows involved in the combination, including reference row 31 with index 1, reference row 33 with index 3, reference row 35 with index 5, reference row 37 with index 7, and reference row 39 with index 12. Unlike Figure 8A , in the example shown in Figure 8B , the template region 40 of the current block is set on two reference rows with indexes 0 and 1, and the five extended reference rows involved in the combination are reference row 42 with index 2, reference row 43 with index 3, reference row 45 with index 5, reference row 47 with index 7, and reference row 49 with index 12. In other words, this example selects N available extended reference rows from the predefined extended reference rows with indexes {2, 3, 5, 7, 12}. There are many other options for selecting template regions and extended reference rows, such as setting the template region of the current block on three reference rows with indexes 0, 1, and 2, or on four reference rows with indexes 0 through 3, and so on. When the template area is wider, the prediction is relatively more accurate.

[0160] In an exemplary embodiment of the present disclosure, the N extended reference rows of the current block are predefined N max Among the extended reference lines, the extended reference lines are located outside the template area of ​​the current block and do not exceed the CTU boundary; where N max The maximum number of extended reference lines that can be used in TMRL_IP mode. This embodiment limits the N extended reference lines used for combination to those outside the template area of ​​the current block and not exceeding the CTU boundary. However, if the hardware supports it, extended reference lines that exceed the CTU boundary can also be selected for combination.

[0161] In an exemplary embodiment of the present disclosure, N max =5, the predefined five extended reference rows are reference rows with indexes {1, 3, 5, 7, 12} or {2, 3, 5, 7, 12}. In another exemplary embodiment of the present disclosure, the predefined N max The extended reference rows are the N closest to the current block, indexed starting from 1. max extended reference rows; or, the N nearest reference rows to the current block whose indexes start at 1 and are odd numbers. max extended reference rows; or, the N closest to the current block with an even index starting from 2. max Selecting odd or even reference rows can simplify the calculation.

[0162] In an exemplary embodiment of the present disclosure, the M intra prediction modes are only allowed to be selected from the angular mode, or only allowed to be selected from the angular mode and the DC mode, or allowed to be selected from the angular mode, the DC mode and the Planar mode.

[0163] In an exemplary embodiment of the present disclosure, the M intra prediction modes are selected through the following process:

[0164] Step 1: Determine the intra-frame prediction modes used by the prediction blocks at multiple adjacent positions around the current block, select the allowed intra-frame prediction modes in sequence, and remove the duplicate modes;

[0165] When the number of intra-frame prediction modes selected in the first step is equal to M, the process ends; when the number of intra-frame prediction modes selected in the first step is less than M and includes an angle mode, the process proceeds to the second step;

[0166] In the second step, starting from the first angle mode selected, the angle mode expansion operation is performed in sequence to obtain the extended angle mode, and the extended angle mode that is different from all the selected angle modes is selected until the total number of the selected intra-frame prediction modes is equal to M.

[0167] In an example of this embodiment, in the first step, for the current block located at the upper boundary of the CTU and the current block not located at the upper boundary of the CTU, the prediction blocks at multiple adjacent positions around the current block include some or all of the following prediction blocks: the prediction block to the left of the current block, the prediction block above it, the prediction block to the upper left, the prediction block to the lower left, and the prediction block to the upper right; as shown in Figure 3, the prediction block to the left of the current block, the prediction block above it, the prediction block to the upper left, the prediction block to the lower left, and the prediction block to the upper right are respectively marked with L, A, AL, BL, and AR in the figure, where M≥5.

[0168] This example allows the current block at the upper boundary of the CTU to select M intra-frame prediction modes from the intra-frame prediction modes used by the prediction blocks in adjacent positions, such as above, to the upper left, and to the upper right, thereby removing the access restrictions for the prediction blocks in these positions. Therefore, when constructing a candidate list for the current block at the upper boundary of the CTU, a more diverse range of intra-frame prediction modes can be used for the combination, increasing the diversity of the combinations in the constructed candidate list and improving encoding performance.

[0169] In an example of this embodiment, the multiple adjacent positions around the current block include 5 adjacent positions around the current block, and the 5 adjacent positions are upper left, upper, upper right, left and lower left in sequence; as shown in FIG3 .

[0170] In an example of this embodiment, when the number of intra-frame prediction modes selected based on the intra-frame prediction modes used by prediction blocks at multiple neighboring positions around the current block is less than M, the first step further includes: continuing to select the intra-frame prediction modes participating in the combination in at least one of the following ways:

[0171] Derivate two intra-frame prediction modes based on the reconstructed values ​​around the current block using the gradient histogram of the DIMD derived by the intra-frame mode at the decoding end, and select an intra-frame prediction mode that is different from the selected intra-frame prediction mode from the two derived intra-frame prediction modes;

[0172] If the selected intra prediction mode does not include the DC mode, the DC mode is selected.

[0173] This example can select the intra-frame prediction mode participating in the combination from the intra-frame prediction mode and / or DC mode derived from DIMD. This method can increase the diversity of the intra-frame prediction modes participating in the combination, and then increase the diversity of the combinations in the constructed candidate list to improve the encoding performance.

[0174] In an example of this embodiment, in the second step, for all the angle modes selected in the first step, starting from the first angle mode selected, the following processing is performed in sequence according to the set order: an angle mode expansion operation is performed to obtain an extended angle mode, and if the extended angle mode is different from the selected angle mode, the extended angle mode is selected; until the total number of selected intra-frame prediction modes is equal to M, the process ends; wherein the number of all angle modes selected in the first step is greater than or equal to 1 and less than or equal to 5, that is, when the intra-frame prediction modes selected in the first step are all angle modes, the expansion operation can also be performed on all angle modes one by one without reducing the number, which can increase the possibility of extended angle modes and improve encoding performance.

[0175] In an example of this embodiment, the process of selecting the M intra-frame prediction modes also includes: if no angle mode is selected in the first step, or the total number of intra-frame prediction modes selected after the second step is still less than M, performing a third step: determining the intra-frame prediction modes allowed to be selected from the predefined intra-frame prediction mode set, and sequentially selecting intra-frame prediction modes that are different from the selected intra-frame prediction modes from the determined intra-frame prediction modes allowed to be selected, until the total number of selected intra-frame prediction modes is equal to M.

[0176] In one example of this embodiment, in the process of selecting M intra-frame prediction modes, only the angle mode is allowed to be selected. In another example of this embodiment, in the process of selecting M intra-frame prediction modes, only the angle mode and the DC mode are allowed to be selected; in another example of this embodiment, in the process of selecting M intra-frame prediction modes, the angle mode, the DC mode and the Planar mode are allowed to be selected. The Planar mode has limited effect in combination with the extended reference row and may not be used in the combination. The situation with the DC mode is similar. However, if the increased computational complexity can be accepted, the Planar mode and the DC mode may also be added to the candidate list for combination.

[0177] In an example of this embodiment, the extended operation of the angle mode includes any one or more of the following operations: add 1 and subtract 1 operations on the angle mode; add 2 and subtract 2 operations on the angle mode; add 3 and subtract 3 operations on the angle mode; and add 4 and subtract 4 operations on the angle mode.

[0178] In an exemplary embodiment of the present disclosure, the M intra-frame prediction modes use some or all of the intra-frame prediction modes in the most probable mode MPM except the Planar mode; or, use some or all of the intra-frame prediction modes in the MPM and the second MPM except the Planar mode; or, use some or all of the intra-frame prediction modes in the most probable mode MPM except the Planar mode and the DC mode; or, use some or all of the intra-frame prediction modes in the MPM and the second MPM except the Planar mode and the DC mode; or, use some or all of the intra-frame prediction modes in the most probable mode MPM except the Planar mode, the DC mode and the decoder-side intra mode derivation (DIMD) mode; or, the M intra-frame prediction modes use some or all of the intra-frame prediction modes in the MPM and the second MPM except the Planar mode, the DC mode and the DIMD mode.

[0179] In this embodiment, all intra-frame prediction modes except the Planar mode in the MPM are used as predefined intra-frame prediction modes, or all intra-frame prediction modes except the Planar mode in the MPM and the second MPM are used as predefined intra-frame prediction modes. When the reference lines of the current block include all predefined extended reference lines, all predefined intra-frame prediction modes are used. When the reference lines of the current block include only some predefined extended reference lines, some predefined intra-frame prediction modes are used.

[0180] In an exemplary embodiment of the present disclosure, M intra prediction modes are selected in the following manner:

[0181] Select M' intra prediction modes;

[0182] Predicting the templates of the current block according to the reference rows of the current block outside the template and the M' intra prediction modes, and calculating errors between reconstructed values ​​of the templates and predicted values, to obtain M' errors;

[0183] Select the M intra-frame prediction modes with the smallest corresponding errors from the M' intra-frame prediction modes as the M intra-frame prediction modes participating in the combination, M <M’。

[0184] The template of the current block in this embodiment is used to select M intra prediction modes from M' intra prediction modes, and the template region of the aforementioned current block is used to select K combinations from N×M combinations. The two can be different, but they can also occupy the same region.

[0185] When selecting M' intra prediction modes in this embodiment, various methods for selecting M intra prediction modes in the above embodiments of the present disclosure can be used. For example, directly select from the lists of MPM and the second MPM, or select through the first step using the method of the aforementioned embodiment, or select through the first step and the second step, or select through the first step, the second step, and the third step, and so on.

[0186] In an exemplary embodiment of the present disclosure, N≤N max , 2≤N max ≤12; 2≤M≤18; K≤K max , 6≤K max ≤36; where N max is the maximum number of extended reference lines allowed to be used in the TMRL_IP mode, and K max is the maximum number of candidate combinations allowed to be used in the TMRL_IP mode. Although the value ranges of the relevant parameters of N, M, and K are given here, this is only exemplary, and these value ranges should not be regarded as limitations to the present disclosure. Other embodiments of the present disclosure can also take values outside these value ranges of this embodiment.

[0187] In an example of this embodiment, N = 5, M = 10, K = 20; in another example, N = 5, M = 6, K = 12; in another example, N = 5, M = 8, K = 16; in another example, N = 5, M = 12, K = 24.

[0188] In an exemplary embodiment of the present disclosure, there are at least two sets of values for N, M, and K. The first set of values is N1, M1, K1, and the second set of values is N2, M2, K2, where N1≤N2, M1≤M2, K1≤K2, and N1×M1 < N2×M2; the first set of values is used when constructing the candidate list for the current block of the first size, and the second set of values is used when constructing the candidate list for the current block of the second size, and the first size is smaller than the second size. Here, the first size and the second size can respectively represent multiple sizes. For example, the first size can include 4×4, 4×8, �×8, etc., and the second size can include 16×8, 16×16, 8×16, etc.

[0189] This embodiment uses different N, M, and K values ​​for current blocks of different sizes. When the current block size is small, smaller values ​​are used to construct the candidate list for the TMRL_IP mode. When the current block size is large, larger values ​​are used to construct the candidate list for the TMRL_IP mode. This achieves a better balance between computational complexity and performance.

[0190] In an exemplary embodiment of the present disclosure, predicting the template region of the current block based on the N×M combinations includes: when the current block is located at the left edge of the image, predicting the template region above the current block based on the N×M combinations, and not predicting the template region to the left of the current block. This embodiment simplifies operations and reduces computation time without impacting performance.

[0191] In an exemplary embodiment of the present disclosure, predicting the template area of ​​the current block according to the N×M combinations includes: predicting each of the N×M combinations in the following manner:

[0192] Calculating an initial prediction value of the template region according to a reconstruction value of the extended reference line in the combination and an intra-frame prediction mode in the combination, wherein the reconstruction value of the extended reference line is an original reconstruction value of the extended reference line or a filtered reconstruction value;

[0193] The initial prediction value of the template area is subjected to 4-tap filtering or 6-tap filtering, and the filtering result is used as the prediction value of the template area obtained according to the combined prediction.

[0194] When predicting the template area of ​​the current block, this embodiment may not filter the reconstructed value of the extended reference row, but use the original reconstructed value for calculation. It may also use a filter with shorter taps (such as a 4-tap filter) to reduce the calculation complexity and speed up the calculation speed.

[0195] In an exemplary embodiment of the present disclosure, predicting the template region of the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template region and the predicted value obtained by prediction, includes:

[0196] According to K combinations, the entire template area of ​​the current block is predicted respectively, and the corresponding K errors are formed into an error set, and the maximum error in the error set is recorded as D max ;

[0197] For each of the remaining combinations, first predict the single-side template area of ​​the current block according to the combination, and calculate the error D1 between the reconstructed value and the predicted value of the single-side template area of ​​the current block, such as D1≥D max, complete the prediction of the combination, such as D1 <D max , and then predict the template area on the other side of the current block based on the combination, and calculate the error D2 between the reconstructed value and the predicted value of the entire template area of ​​the current block, such as D2 <D max , add D2 to the error set, and D max Delete the maximum error D in the error set from the error set and update max , such as D2≥D max , then the prediction of the combination is completed;

[0198] After completing the prediction of the N×M combinations, the K combinations corresponding to the K errors in the error set are taken as the K combinations with the smallest corresponding errors.

[0199] In one example, the errors in the error set can be arranged in ascending order. When D2 is added to the error set, D2 is inserted at a position such that the errors in the error set are still arranged in ascending order. However, in other examples, the K errors in the error set can be sorted after the prediction of the N×M combinations is completed.

[0200] In this embodiment, it is not necessary to perform prediction and error calculation on the entire template area for all combinations, and the combinations can be sorted, which can reduce the computational complexity and speed up the computational speed.

[0201] In an exemplary embodiment of the present disclosure, the step of filling the candidate list of the TMRL_IP mode of the current block with the K combinations corresponding to the errors in ascending order of the errors includes: starting from the first position of the candidate list, filling the candidate list with the K combinations corresponding to the errors in ascending order of the errors. In this embodiment, the candidate list of the TMRL_IP mode is only filled with combinations of extended reference rows and intra-frame prediction modes. The combination of a reference row indexed as 0 and an intra-frame prediction mode is indicated by other traditional modes, such as MPM.

[0202] In another exemplary embodiment of the present disclosure, starting from the i-th position of the candidate list, in the order of the errors from small to large, the K combinations corresponding to the errors are filled into the candidate list; the i-th position of the candidate list is filled with a combination of a reference row with an index of 0 and one or more intra-frame prediction modes, i≥2. The candidate list of the TMRL_IP mode in this embodiment is not only filled with a combination of an extended reference row and an intra-frame prediction mode, but also with a combination of a reference row with an index of 0 and an intra-frame prediction mode. In this case, when the combination selected for the current block is a combination of a reference row with an index of 0 and an intra-frame prediction mode, it can also be represented by the TMRL_IP mode index. At this time, the TMRL_IP mode flag can still be used.

[0203] An embodiment of the present disclosure provides a video decoding method as shown in FIG9 , including:

[0204] Step 310 , decoding the multi-reference line intra prediction MRL_IP mode flag of the current block to determine whether the current block uses the MRL_IP mode;

[0205] Step 320: If it is determined that the current block uses the MRL_IP mode, continue decoding the MRL_IP mode index of the current block and construct a candidate list of the MRL_IP mode of the current block, wherein the candidate list is filled with combinations of extended reference rows and intra prediction modes of the current block candidates;

[0206] Step 330 , determining a combination of extended reference rows and intra prediction modes selected for the current block based on the candidate list and the MRL_IP mode index, and predicting the current frame based on the selected combination;

[0207] The TMRL_IP mode index is used to indicate the position of the selected combination of the extended reference row and the intra prediction mode in the candidate list.

[0208] This embodiment, after determining that the current block uses the MRL_IP mode by decoding the MRL_IP mode flag, populates the combination of the extended reference row and the intra-frame prediction mode into the MRL_IP mode candidate list. The decoded MRL_IP mode index and the candidate list are used to determine the selected combination for the current block and perform prediction. In other words, the MRL_IP mode index can simultaneously indicate the selected extended reference row and intra-frame prediction mode for the current block, eliminating the need for two indexes. This reduces coding overhead.

[0209] In an exemplary embodiment of the present disclosure, the candidate list of the MRL_IP mode is a candidate list of the TMRL_IP mode constructed according to the method of the embodiment of the present disclosure. However, the MRL_IP mode constructed in other embodiments of the present disclosure is not limited to the TMRL_IP mode. For example, based on the statistical results of a large amount of sample data, the combination of the reference row and the intra-frame prediction mode with the highest probability of being selected can be filled into the candidate list of the MRL_IP mode. The candidate list of the MRL_IP mode can be filled with only the combination of the extended reference row and the intra-frame prediction mode, but can also be filled with the combination of the reference row with index 0 and other intra-frame prediction modes.

[0210] In this embodiment, when constructing the candidate list of the TMRL_IP mode, K can be a set value, but K in this embodiment is determined according to the TMRL_IP mode index. When the TMRL_IP mode index indicates that the combination of the selected extended reference row and the intra-frame prediction mode is located at the Lth position in the candidate list, K=L, and L is a positive integer. For example, the TMRL_IP mode index is 5, assuming that the index is numbered from 0, indicating that the selected combination is located at the 6th position in the candidate list, then K=6. Unlike the decoding end, the encoding end needs to determine the TMRL_IP mode index based on the position of the selected combination in the candidate list, and needs to use the set K value. After the decoding end parses the obtained TMRL_IP mode index, the constructed candidate list only needs to include the combination of the position indicated by the TMRL_IP mode index. By reducing the number of combinations to be filled in the candidate list of the TMRL_IP mode of the current block, the operation of constructing the candidate list can be simplified, saving computing resources.

[0211] In an exemplary embodiment of the present disclosure, before decoding the MRL_IP mode flag of the current block, the method further includes: if all conditions for allowing the current block to use the MRL_IP mode are met, decoding the MRL_IP mode flag of the current block, wherein the conditions for allowing the use of the MRL_IP mode include any one or more of the following:

[0212] The current block is a block in the luma frame;

[0213] The current block allows the use of MRL;

[0214] The current block is not located at the upper boundary of the coding tree unit CTU;

[0215] The current block does not use template-based intra prediction mode to derive TIMD.

[0216] In this embodiment, when one of the above conditions is met, the MRL_IP mode is not allowed to be used, and the decoding of the MRL_IP mode flag and the MRL_IP mode index can be skipped. However, this is not necessarily the case in other embodiments. For example, when the MRL_IP mode is encoded and decoded before the TIMD, the use of the TIMD in the current block cannot be used as a condition for not allowing the use of the MRL_IP mode. For another example, if the hardware in the future can support obtaining reference rows outside the CTU boundary, the current block being located at the upper boundary of the CTU will no longer be a condition for not allowing the use of the MRL_IP mode, and so on.

[0217] In an exemplary embodiment of the present disclosure, the method further includes: decoding a multiple reference row index of the current block, where it is determined through decoding that the current block allows the use of MRL, the current block is not located at the upper boundary of a CTU, and the current block uses TIMD, where the multiple reference row index is used to indicate the position of the reference row selected for the current block in the multiple reference row index list. In this embodiment, when the current block uses TIMD, the MRL_IP mode is not allowed, but the use of MRL is still allowed. Thus, the reference row selected for the current block can still be determined by decoding the multiple reference row index of the current block, and then combined with the TIMD mode selected for the current block, prediction of the current block can be performed.

[0218] In an exemplary embodiment of the present disclosure, when it is determined that the current block uses the MRL_IP mode according to the MRL_IP mode flag, the method further includes: skipping decoding of syntax elements of any one or more of the following modes: MPM mode, sub-partition mode, multi-transform selection (MTS) mode, low-frequency non-divisible transform (LFNST) mode, and TIMD mode. Correspondingly, on the encoder side, if, during encoding, one or more of the above modes is skipped when the current block uses the MRL_IP mode flag, then on the decoder side, when it is determined through decoding that the current block uses the MRL_IP mode flag, decoding of these modes is also skipped.

[0219] An embodiment of the present disclosure further provides a video decoding method, specifically relating to an intra-frame prediction decoding process, and also describes the intra-frame prediction encoding process at the encoding end.

[0220] In this embodiment, the encoding end constructs a candidate list of the TMRL_IP mode, and when a combination in the candidate list is selected for the current block through mode selection, the syntax elements of the TMRL_IP mode are encoded and decoded. By combining the extended reference line and the intra-frame prediction mode (traditional intra-frame prediction mode, including DC mode, Planar mode and multiple angle modes) for encoding and decoding, the encoding efficiency can be improved.

[0221] This embodiment constructs a template based on N predefined extended reference lines and M intra-frame prediction modes at reference line 0, i.e., reference line 0. (x, -1) and (-1, y) are the coordinates relative to the upper-left corner (0, 0) of the current block. The template region of one row and one column of the current block in this embodiment can be seen in template region 30 shown in FIG8A . This figure also adds five predefined extended reference lines, indexed {1, 3, 5, 7, 12}.

[0222] When constructing the candidate list of the TMRL_IP mode, calculate the SAD between the predicted value and the reconstructed value of N×M combinations on the template, sort the corresponding combinations in ascending order of SAD, and fill the K combinations with smaller SAD among the NxM combinations into the candidate list of the TMRL_IP mode in ascending order of SAD, N×M ≥ 2.

[0223] Constructing a candidate list for the TMRL_IP mode is an operation that both the encoder and decoder must perform. When the encoding conditions are met, such as the combination selected for the current block being in the candidate list, the encoder encodes a TMRL_IP mode flag to indicate the use of the TMRL_IP mode. The encoder also determines the TMRL_IP mode index based on the position of the selected combination in the candidate list. For example, if the selected combination is in the first position, the TMRL_IP mode index is 0; if it is in the second position, the TMRL_IP mode index is 1, and so on. The TMRL_IP mode index can be encoded using, but is not limited to, Golomb-Rice coding.

[0224] The following description is made using an example where N=, M=6 and K=12. The indices of the predefined five extended reference rows are {1, 3, 5, 7, 12}, and the six intra prediction modes are selected step by step.

[0225] The video decoding method of this embodiment includes:

[0226] Step 1: Decode the syntax elements related to the TMRL_IP mode;

[0227] The decoder parses syntax elements related to intra-frame prediction modes, including those related to TIMD, MRL, and other modes. The TMRL_IP mode proposed in this embodiment can be considered an evolution of the MRL mode, and the syntax elements of the TMRL_IP mode can also be considered part of the syntax elements of the MRL mode. Of course, the two modes can also be considered as two different modes.

[0228] When the current block uses TIMD mode, the decoding method of the MRL mode syntax elements remains unchanged. When the current block does not use TIMD mode, the syntax elements of the TMRL_IP mode need to be decoded. The syntax related to the current block decoding is shown in the following table:

[0229]

[0230]

[0231] The "cu_tmrl_flag" in the table is the TMRL_IP mode flag. When it is equal to 1, it means that the current block uses the TMRL_IP mode, which also defines that the intra-frame prediction type of the current luminance sample is a template-based multi-reference row frame prediction mode; when "cu_tmrl_flag" is equal to 0, it means that the current block does not use the TMRL_IP mode, which also defines that the intra-frame prediction mode type of the current luminance sample is not a template-based multi-reference row frame prediction mode.

[0232] The "tmrl_idx" in the table is the TMRL_IP mode index, which represents the position of the extended reference row and intra prediction mode combination selected for the current block within the TMRL_IP mode candidate list. Alternatively, it defines the index of the selected combination within the candidate list sorted by the TMRL_IP mode (indicating the position of the combination). "tmrl_idx" can be encoded and decoded using the Columbine method, which is not detailed here.

[0233] As can be seen from the table above, before decoding cu_tmrl_flag, the following conditions are first determined: the current block allows the use of MRL (i.e., sps_mrl_enabled_flag is 1), the current block is not located at the upper boundary of the CTU (i.e., (y0%CtbSizeY)>0), and the current block does not use TIMD. If these conditions are met, cu_tmrl_flag is decoded. If the other two conditions are met and the current block uses TIMD, the multi-reference row index intra_luma_ref_idx of the current block is decoded.

[0234] The ISP mode flag (intra_subpartitions_mode_flag) in the table is decoded after the TMRL_IP mode related syntax elements. When the current block does not use the TMRL_IP mode (!cu_tmrl_flag is established), the intra_subpartitions_mode_flag is decoded.

[0235] Step 2: construct a candidate list of TMRL_IP modes, and determine the extended reference row and intra prediction mode selected for the current block according to the TMRL_IP mode index and the candidate list;

[0236] After the parsing stage, before predicting the current block, if the current block uses the TMRL_IP mode, it is necessary to construct a candidate list of the TMRL_IP mode, and determine the extended reference row and intra prediction mode selected for the current block based on the TMRL_IP mode index and the candidate list.

[0237] To construct a candidate list for the TMRL_IP mode, it is necessary to first determine the candidate extended reference rows and the candidate intra prediction modes.

[0238] ■Determine candidate extension reference lines

[0239] Candidate extended reference lines are selected from predefined extended reference lines. The position of the current block in the image determines which of the predefined extended reference lines can be used. In principle, the upper reference line that can be used by the current block should not exceed the upper CTU boundary. In one example, all extended reference lines with indexes {1, 3, 5, 7, 12} that do not exceed the CTU boundary are added to the candidate extended reference lines. To achieve better encoding and decoding performance or reduce complexity, more or fewer extended reference lines may be used.

[0240] ■Determine candidate intra prediction modes

[0241] In this embodiment, the TMRL_IP mode is not bound to the MPM (it may be bound in other embodiments), but rather a candidate list of intra prediction modes is constructed, and the intra prediction mode used for the combination is selected from this candidate list. The candidate list is derived as follows:

[0242] First, among the 67 traditional prediction modes, the Planar mode and the DC mode are removed, or only the Planar mode is removed while the DC mode is retained. The removed modes are not added to the candidate list, that is, they are not used as intra-frame prediction modes participating in the combination in the TMRL_IP mode.

[0243] In this embodiment, the candidate prediction mode list to be constructed is 6 in length. When constructing the candidate prediction mode list, non-overlapping intra-frame prediction modes are first selected in sequence based on the intra-frame prediction modes used by the prediction blocks at the five neighboring positions around the current block to populate the candidate prediction mode list. In one example, the prediction blocks at the neighboring positions around the current block (referred to as blocks for short) are accessed in the following order to obtain their intra-frame prediction modes and select them:

[0244] Access the block on the left to determine whether the block is an intra-frame coding block. If so and the intra-frame prediction mode of the block is not planar mode, select the intra-frame prediction mode of the block and fill it into the candidate list.

[0245] Access the block above to determine whether the block is an intra-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the existing intra-frame prediction mode in the candidate list (that is, it is different from the existing intra-frame prediction mode), then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0246] Access the block on the left to determine whether the block is an inter-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0247] Access the block above to determine whether the block is an inter-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0248] Access the block on the upper left side to determine whether the block is an intra-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0249] Access the block in the lower left corner to determine whether the block is an intra-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0250] Access the block in the upper right corner to determine whether the block is an intra-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0251] Access the upper left block to determine whether the block is an inter-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0252] Access the lower left block to determine whether the block is an inter-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0253] Access the upper right block to determine whether the block is an inter-frame coding block. If so, and the intra-frame prediction mode of the block is not planar mode and does not overlap with the intra-frame prediction mode already in the candidate list, then select the intra-frame prediction mode of the block and fill it into the candidate list.

[0254] The current block of the embodiment of the present disclosure (including other embodiments) can be a block in an inter-frame coded frame or a block in an intra-frame coded frame. When the current block is a block in an inter-frame coded frame and is parsed to be an intra-frame coded block (a block predicted using an intra-frame prediction mode), the predicted block at a neighboring position of the intra-frame coded block may be an inter-frame coded block. In this case, the intra-frame prediction mode of the inter-frame coded block can be transferred from the reference block of the inter-frame coded block.

[0255] For the current block at the upper boundary of the CTU, this embodiment also uses the above sequence to access the prediction blocks at the five positions adjacent to the current block, obtain their intra-frame prediction modes and select them, and does not skip accessing the prediction blocks at the upper left, upper, and upper right.

[0256] After selecting non-overlapping intra prediction modes from the intra prediction modes used by the prediction blocks at the five neighboring positions of the current block to populate the candidate prediction mode list, the angle mode expansion operation can be performed directly, or one or both of the following selection operations can be performed first:

[0257] The first method is to derive two intra-frame prediction modes based on the reconstructed values ​​around the current block and the gradient histogram of DIMD. The mode that does not overlap with the existing modes in the list is selected from the two derived intra-frame prediction modes and filled into the candidate list.

[0258] The second method is to select the DC mode and add it to the candidate list if the DC mode is not included in the intra prediction modes selected in the candidate list.

[0259] Then, the angle pattern expansion operation is performed on the patterns that have been filled into the list. Specifically, all the selected angle patterns can be added or subtracted in turn, and the non-repeated expanded angle patterns are selected and filled into the candidate prediction pattern list in turn. If the number of patterns filled in the candidate list reaches 6, the filling is stopped.

[0260] The specific operation process of adding one and subtracting one is shown in the following table:

[0261]

[0262] The above "angle mode - 1" refers to the angle mode obtained by subtracting 1 from the index of the filled angle mode. For example, if the filled angle mode is mode 3, angle mode - 1 is angle mode 2. The above "angle mode + 1" refers to the angle mode obtained by adding 1 to the index of the filled angle mode. For example, if the filled angle mode is mode 3, angle mode + 1 is angle mode 4.

[0263] If angle pattern -1 is smaller than angle pattern 2, for example, angle pattern -1 results in angle pattern 1 (angle patterns are indexed from 2, and angle pattern 1 does not exist), then the angle pattern in the direction opposite to -1 is selected. Assuming there are 65 angle patterns, the angle pattern in the opposite direction is angle pattern 66. If the angle pattern after angle pattern +1 is larger than angle pattern 66, then the angle pattern in the direction opposite to +1 is selected. For example, if the filled angle pattern is angle pattern 66, then angle pattern +1 does not exist, and the angle pattern in the direction opposite to +1 is selected, which is angle pattern 2.

[0264] When expanding the angle mode described above, the angle mode is added by 1 or subtracted by 1. In other embodiments, the addition and subtraction operations can be expanded from addition and subtraction to addition and subtraction of X or X. For example, if X = 3, the angle mode can be added by 1 or subtraction of 1, added by 2 or subtraction of 2, and added by 3 or subtraction of 3 until the candidate prediction mode list is filled. As shown in the following table, delta = 1, 2, ..., X-1.

[0265]

[0266]

[0267] In another embodiment of the present disclosure, the extended operation of the angle mode includes addition 1 and subtraction 1 operations, addition 2 and subtraction 2 operations, addition 3 and subtraction 3 operations, and addition 4 and subtraction 4 operations. Assuming that there are 4 angle modes selected in the first step, an exemplary setting order can be: adding 1 and subtraction 1 operations to the first selected angle mode (referred to as the first angle mode), adding 1 and subtraction 1 operations to the second selected angle mode (referred to as the second angle mode), adding 1 and subtraction 1 operations to the third selected angle mode (referred to as the third angle mode), adding 1 and subtraction 1 operations to the fourth selected angle mode (referred to as the fourth angle mode), adding 2 and subtraction 2 operations to the first angle mode, adding 2 and subtraction 2 operations to the second angle mode, adding 2 and subtraction 2 operations to the third angle mode, adding 2 and subtraction 2 operations to the fourth angle mode, and then continuing to perform addition 3 and subtraction 3 operations on the first angle mode to the fourth angle mode, and then performing addition 4 and subtraction 4 operations on the first angle mode to the fourth angle mode. During the angle expansion process of this embodiment, after completing the expansion operation of each angle mode, the two expanded angle modes obtained can be compared with the selected angle mode, and the angle modes that are different from the selected angle modes can be selected and the total number of selected intra-frame prediction modes can be updated. When the total number is equal to M, the construction of the candidate list of intra-frame prediction modes is completed, that is, the process of selecting M intra-frame prediction modes is completed.

[0268] If the candidate prediction mode list is not full after the angle mode expansion operation is performed on the modes that have been filled in the list, the candidate prediction mode list is filled with non-repeated modes in the predefined mode set until the candidate prediction mode list is full. The mode set includes some angle modes selected according to statistical laws, as follows:

[0269] mpm_default[]={DC_IDX,VER_IDX,HOR_IDX,VER_IDX-4,VER_IDX+4,14,22,42,58,10,26,38,62,6,30,34,66,2,48,52,16};

[0270] Among them, DC_IDX indicates DC mode, VER_IDX indicates vertical mode, HOR_IDX indicates horizontal mode, and the remaining numbers indicate the angle mode corresponding to the number.

[0271] In this embodiment, the length of the candidate prediction mode list is 6. For performance reasons, more angle modes can be tried and the length can be set to a value greater than 6, such as 8, 10, 12, 14, etc. For reduced complexity, fewer modes can be tried and the length can be set to a value less than 6.

[0272] When determining candidate intra-frame prediction modes, this embodiment excludes the Planar mode and the DC mode, or only the Planar mode. However, if complexity is not considered, these two modes may not be excluded, that is, the Planar mode, the DC mode, and all angle modes may all be used as candidate intra-frame prediction modes and participate in the combination with the extended reference row.

[0273] Compared to directly using the first N intra-frame prediction modes in the MPM list and the second MPM list, the candidate list of intra-frame prediction modes constructed through the above steps in this embodiment eliminates the restriction on access to the upper prediction block at the upper boundary of the CTU, thereby increasing the diversity of the candidate list constructed for the current block when the current block is at the upper boundary of the CTU. In addition, by adding the DIMD and DC modes, the diversity of intra-frame prediction modes in the candidate list can be further increased.

[0274] ■Build a candidate list for TMRL_IP mode

[0275] After determining the candidate extended reference lines and intra-frame prediction modes, all combinations in the extended reference line list and candidate prediction mode list can be tried one by one. These combinations are predicted separately on the template area (Template as shown below) where reference line 0 is located, see Figure 8A. The error between the reconstructed value of the template area and the predicted value obtained by each combination is calculated. The K combinations with the smallest errors are filled into the candidate list of the TMRL_IP mode in ascending order of error.

[0276] During the prediction process, this embodiment only limits the use of the TMRL_IP mode when the current block occupies the first row of the CTU. When the current block is located at the left edge of the image, the TMRL_IP mode can still be used. In this case, since the reference line 0 on the left is already outside the image boundary, the left template is not used during prediction, that is, only the upper template area is predicted.

[0277] The prediction process on the template area can be completely consistent with other normal intra-frame angle prediction processes, that is, the reconstructed value of the reference row is first filtered and then used as the initial prediction value of the template area. After the template area is predicted based on the reconstructed value after filtering the reference row and the intra-frame prediction mode in the combination, the initial prediction result is subjected to 4 or 6 tap filtering and then used as the predicted value. Considering the complexity of the operation, the filtering step of the reconstructed value of the reference row can be omitted, or a filter with shorter taps can be used. In this embodiment, when predicting the prediction value of the template area, the reconstructed value of the reference row pixel is not filtered, and the initial prediction result is a 4-tap interpolation filter with 1 / 32 accuracy at a non-integer angle. The 4-tap filter is as follows:

[0278]

[0279] The appropriate filter is selected based on the positional deviation between the downstream and row pixels at the current angle. The deviation iFact is derived as follows: predModeIntra is obtained by mapping the traditional angle to the wide angle, and intraPredAngle is obtained by looking up the predModeIntra table, as follows:

[0280]

[0281] Determine iFact by intraPredAngle and the index refIdx of the selected reference row,

[0282] iFact=((x+1+refIdx)*intraPredAngle)&31

[0283] Finally, the template region is predicted based on the angle, reference line, and filter in the current combination. The SAD between the predicted value of the template region and the reconstructed value of the template region is calculated, and the SAD is sorted in ascending order. The K combinations with the smallest SAD are selected and added to the candidate list of the TMRL_IP mode.

[0284] During the sorting process, the quick algorithm can be used. In the above process, 5 reference lines and 6 prediction modes need to be tried, with a total of 30 combinations. However, only K combinations with the smallest SAD need to be selected. In this embodiment, after predicting the first K combinations and obtaining the corresponding SAD, starting from the (K + 1)-th combination, only K combinations with the smallest errors (which can also be called costs) need to be maintained and updated. Starting from the (K + 1)-th combination, only the upper template area is predicted and the corresponding SAD is calculated. When the SAD calculated based on the upper template is greater than the largest one among the K combinations with the smallest errors, the prediction and error calculation for the left template area can be skipped. For details, refer to the foregoing embodiment.

[0285] In this embodiment, the value of K is the same as that at the encoding end, i.e., K = 12. However, in another embodiment of the present disclosure, K is determined according to the TMRL_IP mode index. When the TMRL_IP mode index indicates that the combination of the selected extended reference line and the intra prediction mode is in the L-th position in the candidate list (i.e., the L-th combination in the candidate list is selected), let K = L. If L' represents the value of the TMRL_IP mode index and 0 ≤ L' < K, then L = L' + 1. When L is less than 12 (the value of K set at the encoding end), it is not necessary to sort out 12 combinations with the smallest costs from the NxM combinations, but only L combinations with the smallest costs need to be sorted out. The decoder determines the value of K according to the TMRL_IP mode index, which can reduce the sorting complexity of the decoder.

[0286] Step 3: Determine the combination of the extended reference line and the intra prediction mode selected for the current block according to the constructed candidate list of the TMRL_IP mode and the decoded TMRL_IP mode index, and perform intra prediction on the current block according to the selection.

[0287] When using the TMRL_IP mode, the reference line index refIdx and the variable predModeIntra define the mode used for intra prediction, both of which are determined according to the TMRL_IP mode index "tmrl_idx" and the candidate list of the TMRL_IP mode.

[0288] On the ECM-4.0 reference software, using the method described in this embodiment, with the settings of N = 5 (the 5 extended reference lines are 1, 3, 5, 7, 12), M = 6 (6 predefined prediction modes, excluding the Planar and DC modes), and K = 12 (only selecting the first 12 sorted combinations with small SAD among all combinations), the following results are measured under the AI configuration:

[0289]

[0290] With N=5 (5 extended reference rows are 1, 3, 5, 7, 12), M=8 (8 predefined prediction modes) and K=16, the results are as follows under AI configuration:

[0291]

[0292] With N=5 (5 extended reference lines are 1, 3, 5, 7, 12), M=12 (12 predefined prediction modes), and K=24, the results measured under AI configuration are as follows:

[0293]

[0294] With N=5 (5 extended reference lines are 1, 3, 5, 7, and 12), M=8 (8 predefined prediction modes, excluding Planar and including DC modes), and K=16, the results are as follows under the AI ​​configuration:

[0295]

[0296] The meanings of the parameters in the table are as follows:

[0297] EncT: Encoding Time, 10X% means that after the reference row sorting technology is integrated, the encoding time is 10X% compared with before integration, which means that the encoding time increases by X%.

[0298] DecT: Decoding Time, 10X% means that after the reference row sorting technology is integrated, the decoding time is 10X% compared with before integration, which means that the decoding time increases by X%.

[0299] ClassA1 and ClassA2 are test video sequences with a resolution of 3840x2160, ClassB is a test sequence with a resolution of 1920x1080, ClassC is 832x480, ClassD is 416x240, and ClassE is 1280x720; ClassF is a screen content sequence with several different resolutions.

[0300] Y, U, V are the three color components, and the columns where Y, U, V are located represent the BD-rate of the test results on Y, U, V. rate) indicator, the smaller the value, the better the encoding performance.

[0301] All intra indicates the test configuration of all intra-frame configurations.

[0302] With N=5 (the five extended reference rows are 1, 3, 5, 7, and 12, respectively), M=10 (10 predefined prediction modes), and K=20, and the first M prediction modes in the MPM list and the second MPM list are directly used in the candidate list construction, the experimental results under the configurations of all intra and random access are shown in Table A below:

[0303]

[0304]

[0305] Under the same settings of N=5, M=10, and K=20, but using the method of this embodiment to construct the candidate list (including allowing the current block at the upper boundary of the CTU to access the adjacent prediction blocks at the upper, upper right, and upper left positions to select the intra prediction mode, selecting the DIMD-derived mode and the DC mode before angular expansion, and performing the expansion operation on all selected angular modes), the test results under the AI ​​configuration are shown in Table B below:

[0306]

[0307]

[0308] A comparison shows that in the "Overall" column, directly combining the first M prediction modes from the MPM list and the second MPM list can achieve 0.08%, 0.01%, and 0.04% BD-rate savings for Y / U / V in the "all intra" configuration, and 0.03%, 0.08%, and 0.13% BD-rate savings for Y / U / V in the "random access" configuration, compared to the reference software. Constructing a candidate list using the method in this embodiment, however, can achieve 0.11%, 0.10%, and 0.07% BD-rate savings for Y / U / V in the "all intra" configuration, and 0.05%, -0.05%, and 0.06% BD-rate savings for Y / U / V in the "random access" configuration, compared to the reference software. This improves the overall performance.

[0309] It can be seen that using the TMRL_IP mode of this embodiment for intra-frame prediction coding and decoding can achieve a significant improvement in coding performance.

[0310] In this embodiment, both the extended reference line and the prediction mode use a template area of ​​one row and one column, sorted and filtered in ascending SAD order. For the extended reference line, if all extended reference lines (including reference line 1) are sorted, only the template area of ​​one row and one column can be used. However, for prediction mode screening, to select a more appropriate prediction mode, using more reference rows, as in the TIMD mode, can yield more accurate results. In other embodiments, the method by which the TMRL_IP mode determines the candidate prediction mode list can also be varied.

[0311] For example, when a candidate prediction mode list of length 6 needs to be constructed, a list of length greater than 6 can be first constructed using the same construction and filling method as in this embodiment. Then, the 4 rows and 4 columns closest to the current block are used as a template. The template is predicted using the 5th reference row and the intra-frame prediction mode in the candidate prediction mode list. The error (SAD or SATD) between the predicted value and the reconstructed value of the template is calculated. The error is sorted in ascending order, and the 6 intra-frame prediction modes with the smallest error are selected as the intra-frame prediction modes in the TMRL_IP mode candidate list of length 6 to be constructed. Furthermore, the 6-length candidate list of the TMRL_IP mode is merely an example, and the value can be adjusted as needed.

[0312] The angle modes in this embodiment take 65 as an example, but in other embodiments, the angles can be expanded to 129 or more to obtain better performance. When expanded to more angles, the number of filters for intra-frame prediction should also be increased accordingly. For example, when there are 129 angles, a filter with 1 / 64 precision is used.

[0313] An embodiment of the present disclosure further provides a bitstream, wherein the bitstream includes block-level syntax elements for intra-frame prediction, wherein the syntax elements include a multiple reference line intra-frame prediction MRL_IP mode flag and an MRL_IP mode index for the current block, wherein the MRL_IP mode flag is used to indicate whether the current block uses the MRL_IP mode, and the MRL_IP mode index is used to indicate the position of the combination of extended reference line and intra-frame prediction mode selected for the current block in the candidate list of the MRL_IP mode. The bitstream of this embodiment can be generated by the video encoding method of any embodiment of the present disclosure.

[0314] An embodiment of the present disclosure also provides a device for constructing a multi-reference row intra-frame prediction mode candidate list, as shown in Figure 10, comprising a processor 71 and a memory 73 storing a computer program, wherein when the processor 71 executes the computer program, it can implement the method for constructing a multi-reference row intra-frame prediction mode candidate list as described in any embodiment of the present disclosure.

[0315] An embodiment of the present disclosure further provides a video decoding device, see FIG10 , comprising a processor and a memory storing a computer program, wherein the processor can implement the video decoding method as described in any embodiment of the present disclosure when executing the computer program.

[0316] An embodiment of the present disclosure further provides a video encoding device, see FIG10 , comprising a processor and a memory storing a computer program, wherein the processor can implement the video encoding method as described in any embodiment of the present disclosure when executing the computer program.

[0317] The processors of the above embodiments of the present disclosure may be general-purpose processors, including central processing units (CPUs), network processors (NPs), microprocessors, etc., or other conventional processors; the processors may also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), discrete logic or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other equivalent integrated or discrete logic circuits, or a combination of the above devices. That is, the processors of the above embodiments may be any processing device or device combination that implements the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. If the embodiments of the present disclosure are partially implemented in software, the instructions for the software may be stored in a suitable non-volatile computer-readable storage medium, and one or more processors may be used to execute the instructions in hardware to implement the methods of the embodiments of the present disclosure. The term "processor" used herein may refer to the above structure or any other structure suitable for implementing the technology described herein.

[0318] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes the video encoding device described in any embodiment of the present disclosure and the video decoding device described in any embodiment of the present disclosure.

[0319] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method for constructing a multi-reference row intra-frame prediction mode candidate list as described in any embodiment of the present disclosure, or implements the video decoding method as described in any embodiment of the present disclosure, or implements the video encoding method as described in any embodiment of the present disclosure.

[0320] In one or more exemplary embodiments above, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, such as according to a communication protocol. In this way, a computer-readable medium may generally correspond to a non-transitory tangible computer-readable storage medium or a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in this disclosure. A computer program product may include a computer-readable medium.

[0321] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may also be referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient (transient) media, but rather refer to non-transient tangible storage media. As used herein, disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, or Blu-ray disc, among others, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0322] In some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0323] The technical solutions of the embodiments of the present disclosure can be implemented in a wide variety of devices or equipment, including wireless mobile phones, integrated circuits (ICs), or a group of ICs (e.g., chipsets). Various components, modules, or units are described in the embodiments of the present disclosure to emphasize the functional aspects of the devices configured to perform the described techniques, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

Claims

1. A method for constructing a multi-reference row intra prediction mode candidate list, comprising: Obtain N×M combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block, where N≥1, M≥1, and N×M≥2; Predicting the template area of ​​the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template area and the predicted value; In order of the errors from small to large, K combinations corresponding to the errors are filled into a candidate list of the template-based multi-reference row intra prediction TMRL_IP mode of the current block, where 1≤K≤N×M.

2. The construction method according to claim 1, wherein: The template area of ​​the current block is set on a reference line closest to the current block; or The template region of the current block is set on a plurality of reference lines closest to the current block, and the N extended reference lines are extended reference lines located outside the template region.

3. The construction method according to claim 1, wherein: The error is expressed as absolute error and SAD, or as error transform absolute value and SATD.

4. The construction method according to claim 1, wherein: The N extended reference lines of the current block are predefined N max Among the extended reference lines, the extended reference lines are located outside the template area of ​​the current block and do not exceed the CTU boundary; Among them, N max The maximum number of extended reference lines that can be used in TMRL_IP mode.

5. The construction method according to claim 4, wherein: The predefined N max Extended reference behavior: 5 extended reference rows with indices {1,3,5,7,12}; or 5 extended reference rows with indices {2,3,5,7,12}; or The N closest to the current block, indexed starting from 1 max extended reference lines; or The index starts from 1 and the N nearest to the current block are odd numbers. max extended reference lines; or The N closest to the current block whose index starts from 2 and is an even number max extended reference lines.

6. The construction method according to claim 1, wherein: The M intra prediction modes are selected by the following process: Step 1: Determine the intra-frame prediction modes used by the prediction blocks at multiple adjacent positions around the current block, select the allowed intra-frame prediction modes in sequence, and remove the duplicate modes; When the number of intra-frame prediction modes selected in the first step is equal to M, the process ends; when the number of intra-frame prediction modes selected in the first step is less than M and includes an angle mode, the process proceeds to the second step; In the second step, starting from the first angle mode selected, the angle mode expansion operation is performed in sequence to obtain the extended angle mode, and the extended angle mode that is different from all the selected angle modes is selected until the total number of the selected intra-frame prediction modes is equal to M.

7. The construction method according to claim 6, wherein: In the first step, for the current block located at the upper boundary of the CTU and the current block not located at the upper boundary of the CTU, the prediction blocks at multiple adjacent positions around the current block include part or all of the following prediction blocks: the prediction block to the left of the current block, the prediction block above the current block, the prediction block to the upper left, the prediction block to the lower left, and the prediction block to the upper right; M≥5.

8. The construction method according to claim 6 or 7, wherein: When the number of intra-frame prediction modes selected based on the intra-frame prediction modes used by prediction blocks at multiple neighboring positions around the current block is less than M, the first step further includes: continuing to select the intra-frame prediction modes to participate in the combination in at least one of the following ways: Derivate two intra-frame prediction modes based on the reconstructed values ​​around the current block using the gradient histogram of the DIMD derived by the intra-frame mode at the decoding end, and select an intra-frame prediction mode that is different from the selected intra-frame prediction mode from the two derived intra-frame prediction modes; If the selected intra prediction mode does not include the DC mode, the DC mode is selected.

9. The construction method according to claim 6, wherein: In the second step, for all the angle modes selected in the first step, starting from the first angle mode selected, the following processing is performed in sequence according to the set order: an angle mode expansion operation is performed to obtain an extended angle mode. If the extended angle mode is different from the selected angle mode, the extended angle mode is selected; until the total number of selected intra-frame prediction modes is equal to M, the process ends; wherein, the number of all angle modes selected in the first step is greater than or equal to 1 and less than or equal to 5.

10. The construction method according to claim 6, wherein: The process of selecting the M intra-frame prediction modes also includes: when no angle mode is selected in the first step, or the total number of intra-frame prediction modes selected after the second step is still less than M, performing a third step: determining the intra-frame prediction modes allowed to be selected from the predefined intra-frame prediction mode set, and sequentially selecting intra-frame prediction modes that are different from the selected intra-frame prediction modes from the determined intra-frame prediction modes allowed to be selected, until the total number of selected intra-frame prediction modes is equal to M.

11. The construction method according to claim 6 or 10, wherein: The intra prediction modes allowed to be selected include only the angular mode, or only the angular mode and the DC mode, or include the angular mode, the DC mode, and the Planar mode.

12. The construction method according to claim 6, wherein: The extended operation of the angle mode includes any one or more of the following operations: Add 1 and subtract 1 operations for angle mode; Add 2 and subtract 2 operations for angle mode; Add 3 and subtract 3 operations for angle mode; Add 4 and subtract 4 operations for angle mode.

13. The construction method according to claim 1, wherein: The M intra prediction modes use some or all of the intra prediction modes in the most probable mode MPM except the Planar mode; or The M intra prediction modes use some or all of the intra prediction modes in the MPM and the second MPM except the Planar mode; or The M intra prediction modes use some or all of the intra prediction modes in the most probable mode MPM except the Planar mode and the DC mode; or The M intra prediction modes use some or all of the intra prediction modes in the MPM and the second MPM except the Planar mode and the DC mode; or The M intra prediction modes use some or all of the intra prediction modes in the most probable mode (MPM) except the Planar mode, the DC mode, and the decoding-end intra mode-derived DIMD mode; or The M intra prediction modes use some or all of the intra prediction modes except Planar, DC, and DIMD modes in the MPM and the second MPM.

14. The construction method according to claim 1, wherein: The M intra prediction modes are selected as follows: Select M' intra prediction modes; Predicting the templates of the current block according to the reference rows of the current block outside the template and the M' intra prediction modes, and calculating errors between reconstructed values ​​of the templates and predicted values, to obtain M' errors; Select the M intra-frame prediction modes with the smallest corresponding errors from the M' intra-frame prediction modes as the M intra-frame prediction modes participating in the combination, M <M’。 15. The construction method according to claim 1, wherein: N≤N max , 2≤N max ≤12; 2≤M≤18; K≤K max , 6≤K max ≤36; where N max K is the maximum number of extended reference lines allowed in TMRL_IP mode. max The maximum number of candidate combinations allowed in TMRL_IP mode.

16. The construction method according to claim 1, wherein: N = 5, M = 10, K = 20; or N=5, M=6, K=12; or N=5, M=8, K=16; or N=5, M=12, K=24.

17. The construction method according to claim 1, wherein: There are at least two groups of values ​​for N, M and K. The first group is N1, M1, K1, and the second group is N2, M2, K2, where N1≤N2, M1≤M2, K1≤K2, and N1×M1 <N2×M2; The first set of values ​​is used when constructing the candidate list for a current block of a first size, and the second set of values ​​is used when constructing the candidate list for a current block of a second size, the first size being smaller than the second size.

18. The construction method according to claim 1, wherein: The predicting the template area of ​​the current block according to the N×M combinations includes: when the current block is located at the left boundary of the image, predicting the template area above the current block according to the N×M combinations, and not predicting the template area on the left side of the current block.

19. The construction method according to claim 1, wherein: The predicting of the template area of ​​the current block according to the N×M combinations includes: predicting each of the N×M combinations in the following manner: Calculating an initial prediction value of the template region according to a reconstruction value of the extended reference line in the combination and an intra-frame prediction mode in the combination, wherein the reconstruction value of the extended reference line is an original reconstruction value of the extended reference line or a filtered reconstruction value; The initial prediction value of the template area is subjected to 4-tap filtering or 6-tap filtering, and the filtering result is used as the prediction value of the template area obtained according to the combined prediction.

20. The construction method according to claim 1, wherein: The predicting the template area of ​​the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template area and the predicted value obtained by prediction, includes: According to K combinations, the entire template area of ​​the current block is predicted respectively, and the corresponding K errors are formed into an error set, and the maximum error in the error set is recorded as D max ; For each of the remaining combinations, first predict the single-side template area of ​​the current block according to the combination, and calculate the error D1 between the reconstructed value and the predicted value of the single-side template area of ​​the current block, such as D1≥D max , complete the prediction of the combination, such as D1 <D max , and then predict the template area on the other side of the current block based on the combination, and calculate the error D2 between the reconstructed value and the predicted value of the entire template area of ​​the current block, such as D2 <D max , add D2 to the error set, and D max Delete the maximum error D in the error set from the error set and update max , such as D2≥D max , then the prediction of the combination is completed; After completing the prediction of the N×M combinations, the K combinations corresponding to the K errors in the error set are taken as the K combinations with the smallest corresponding errors.

21. The construction method according to claim 1, wherein: The step of filling the K combinations corresponding to the errors into a candidate list of the TMRL_IP mode of the current block in ascending order of the errors includes: Starting from the first position of the candidate list, in order of the errors from smallest to largest, fill the candidate list with K combinations corresponding to the errors; or Starting from the i-th position of the candidate list, the K combinations corresponding to the errors are filled into the candidate list in order of the errors from small to large; the i-th position of the candidate list is filled with a combination of a reference row with an index of 0 and one or more intra-frame prediction modes, i≥2.

22. A video decoding method, comprising: Decode the multi-reference line intra prediction MRL_IP mode flag of the current block to determine whether the current block uses the MRL_IP mode; When it is determined that the current block uses the MRL_IP mode, continue decoding the MRL_IP mode index of the current block and construct a candidate list of the MRL_IP mode of the current block, where the candidate list is filled with combinations of extended reference rows and intra prediction modes of the current block candidates; Determining a combination of an extended reference row and an intra prediction mode selected for the current block based on the candidate list and the MRL_IP mode index, and predicting the current frame based on the selected combination; The MRL_IP mode index is used to indicate the position of the selected combination of the extended reference row and the intra prediction mode in the candidate list.

23. The video decoding method according to claim 18, comprising: The candidate list of the MRL_IP mode is a candidate list of the template-based multi-reference row intra-frame prediction TMRL_IP mode constructed according to the method described in any one of claims 1 to 21; wherein K is a set value, or K is determined according to the TMRL_IP mode index, and when the TMRL_IP mode index indicates that the combination of the selected extended reference row and intra-frame prediction mode is located at the Lth position in the candidate list, K=L, and L is a positive integer.

24. The video decoding method according to claim 22 or 23, comprising: Before decoding the MRL_IP mode flag of the current block, the method further includes: if all conditions for allowing the current block to use the MRL_IP mode are met, decoding the MRL_IP mode flag of the current block, wherein the conditions for allowing the use of the MRL_IP mode include any one or more of the following: The current block is a block in the luma frame; The current block allows the use of MRL; The current block is not located at the upper boundary of the coding tree unit CTU; The current block does not use template-based intra prediction mode to derive TIMD.

25. The video decoding method according to claim 22, comprising: The method further comprises: When decoding determines that the current block allows the use of MRL, the current block is not located at the upper boundary of the CTU, and the current block uses TIMD, the multiple reference row index of the current block is decoded, and the multiple reference row index is used to indicate the position of the reference row selected for the current block in the multiple reference row index list.

26. The video decoding method according to claim 22, comprising: When it is determined according to the MRL_IP mode flag that the current block uses the MRL_IP mode, the method also includes: skipping the decoding of syntax elements of any one or more of the following modes: MPM mode, intra-frame sub-block partitioning ISP mode, multi-transform selection MTS mode, low-frequency non-divisible transform LFNST mode, and TIMD mode.

27. A video encoding method, comprising: Constructing a candidate list of a multiple reference line intra prediction MRL_IP mode for the current block, wherein the candidate list is filled with combinations of extended reference lines and intra prediction modes of the current block candidates; Through rate-distortion optimization, a combination of reference row and intra prediction mode is selected for intra prediction of the current block; When the encoding condition of the MRL_IP mode of the current block is satisfied, encoding the MRL_IP mode flag of the current block to indicate that the current block uses the MRL_IP mode, and encoding the MRL_IP mode index of the current block to indicate the position of the selected combination in the candidate list; The encoding condition at least includes: the selected combination is in the candidate list.

28. The video encoding method according to claim 27, wherein: The candidate list of the MRL_IP mode is a candidate list of the template-based multi-reference line intra prediction TMRL_IP mode constructed according to the method according to any one of claims 1 to 21; wherein K is a set value.

29. The video encoding method according to claim 27, wherein: The encoding condition further includes: the current block does not use the template-based intra prediction mode to derive the TIMD; The method further comprises: When the current block uses TIMD, the encoding of the MRL_IP mode flag and MRL_IP mode index of the current block is skipped; When the current block does not use TIMD but the selected combination is not in the candidate list, the MRL_IP mode flag of the current block is encoded to indicate that the current block does not use the MRL_IP mode, and the encoding of the MRL_IP mode index of the current block is skipped.

30. The video encoding method of claim 27, wherein: The constructing of the candidate list of the MRL_IP mode of the current block includes: constructing the candidate list only when all conditions for allowing the current block to use the MRL_IP mode are met, and the conditions for allowing the use of the MRL_IP mode include any one or more of the following: The current block is a block in the luma frame; The current block is not located at the upper boundary of the coding tree unit CTU; The current block allows the use of multiple reference lines MRL; The size of the current block is not greater than the maximum size of the current block that can use the MRL_IP mode; The aspect ratio of the current block meets the requirement of the MRL_IP mode for the aspect ratio of the current block.

31. The video encoding method of claim 27, wherein: The MRL_IP pattern index is encoded using the Columbus-Rice coding method.

32. The video encoding method according to claim 27, comprising: When the encoding conditions of the MRL_IP mode of the current block are met, the method also includes: skipping the encoding of syntax elements of any one or more of the following modes: MPM mode, intra-frame sub-block partitioning ISP mode, multi-transform selection MTS mode, low-frequency non-divisible transform LFNST mode, and TIMD mode.

33. A code stream, wherein The code stream includes block-level syntax elements for intra-frame prediction, and the syntax elements include a multiple reference row intra-frame prediction MRL_IP mode flag and an MRL_IP mode index of the current block, wherein the MRL_IP mode flag is used to indicate whether the current block uses the MRL_IP mode, and the MRL_IP mode index is used to indicate the position of the combination of extended reference row and intra-frame prediction mode selected for the current block in the candidate list of the MRL_IP mode.

34. The code stream of claim 33, wherein: The code stream is generated by the video encoding method according to any one of claims 27 to 32.

35. A device for constructing a candidate list of intra prediction modes for multiple reference lines, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the method for constructing a multi-reference row intra prediction mode candidate list as described in any one of claims 1 to 21.

36. A video decoding device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the video decoding method according to any one of claims 22 to 26.

37. A video encoding device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the video encoding method according to any one of claims 27 to 32.

38. A video encoding and decoding system, wherein: It includes the video encoding device as claimed in claim 37 and the video decoding device as claimed in claim 36.

39. A non-transitory computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it can implement the method for constructing a multi-reference row intra-frame prediction mode candidate list as described in any one of claims 1 to 21, or implement the video decoding method as described in any one of claims 22 to 26, or implement the video encoding method as described in any one of claims 27 to 32.

Images