Intra-frame prediction fusion method, video encoding and decoding method, device and system

Abstract

The invention relates to an intra-frame prediction fusion method, a video encoding and decoding method, a video encoding and decoding device and a video encoding and decoding system. Under the condition that an intra-frame prediction mode selected by a current block comprises an angle mode, whether a limiting condition that the current block uses an intra-frame prediction fusion IPF is established or not is determined; when at least one of the limiting conditions is established, the use of IPF is limited when intra prediction is performed on the current block. By limiting the use of the IPF, it can be avoided that too many modes are fused during prediction, and the coding performance is improved. The embodiment of the invention also provides a video encoding and decoding method using the intra-frame prediction fusion method, a corresponding device and a system.

Description

Intra-frame prediction fusion method, video encoding and decoding method, device and system

[0001] Cross-references

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on July 8, 2022, with application number 202210806435.X and titled “A candidate list construction method, video encoding and decoding method, device and system”. The entire contents of the application are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present disclosure relate to, but are not limited to, video technology, and more specifically, to an intra-frame prediction fusion method, a video encoding and decoding method, a device, and a system. Background Art

[0004] Digital video compression technology primarily compresses large amounts of digital video data for easier transmission and storage. Currently popular video codec standards, such as H.266 / Versatile Video Coding (VVC), employ a block-based hybrid coding framework. Each video frame is divided into square largest coding units (LCUs) of equal size (e.g., 128x128, 64x64, etc.). Each LCU can be divided into rectangular coding units (CUs) based on a specific rule. Coding units may also be divided into prediction units (PUs) and transform units (TUs). The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-frame prediction and inter-frame prediction, which are used to reduce or remove inherent redundancy in the video. Intra-frame blocks are predicted using neighboring pixels as reference, while inter-frame blocks use information from spatially adjacent blocks and reference information from other frames. In contrast to the prediction signal, the residual information is transformed, quantized, and entropy-encoded into a bitstream on a block-by-block basis. These techniques are described in standards and implemented in various fields related to video compression.

[0005] With the surge in Internet videos and people's increasing demand for video clarity, although existing digital video compression standards can save a lot of video data, there is still a need to pursue better digital video compression technology to reduce the bandwidth and traffic pressure of digital video transmission.

[0006] SUMMARY OF THE INVENTION

[0007] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0008] An embodiment of the present disclosure provides an intra-frame prediction fusion method, including:

[0009] In the case where the intra prediction mode selected by the current block includes an angle mode, determining whether a restriction condition for using the intra prediction fusion IPF for the current block is met;

[0010] When at least one of the restriction conditions is met, the IPF is restricted from being used when performing intra-frame prediction on the current block.

[0011] An embodiment of the present disclosure further provides a video decoding method, including:

[0012] Decode the bitstream and determine the reference row and intra prediction mode selected for the current block;

[0013] Predicting the current block according to the intra-frame prediction fusion method described in any embodiment of the present disclosure to obtain a predicted value of the current block;

[0014] A reconstructed value of the current block is determined according to the predicted value of the current block.

[0015] An embodiment of the present disclosure further provides a video encoding method, including:

[0016] Determine the reference row and intra prediction mode selected by the current block through mode selection;

[0017] Predicting the current block according to the intra-frame prediction fusion method described in any embodiment of the present disclosure to obtain a predicted value of the current block;

[0018] The residual of the current block is determined based on the original value and the predicted value of the current block.

[0019] An embodiment of the present disclosure further provides a video encoding method, including:

[0020] When decoding determines that the current block uses the template-based multi-reference line intra prediction TMRL mode, continue decoding the TMRL mode index and TMRL fusion flag of the current block;

[0021] Constructing a candidate list of the TMRL mode of the current block, and determining the extended reference row and intra prediction mode selected for the current block according to the candidate list and the TMRL mode index;

[0022] In the case where the TMRL fusion flag indicates that intra-frame prediction fusion IPF is used, a weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block;

[0023] The first prediction result is obtained by predicting the current block based on the extended reference row and the intra-frame prediction mode, and the second prediction result is obtained by predicting the current block based on another reference row and the intra-frame prediction mode, and the intra-frame prediction mode is an angle mode.

[0024] An embodiment of the present disclosure further provides a video encoding method, including:

[0025] Constructing a candidate list of template-based multiple reference line intra prediction (TMRL) modes for the current block, wherein the candidate list is filled with combinations of extended reference lines and intra prediction modes of the current block candidates; selecting a reference line and intra prediction mode combination for the current block through rate-distortion optimization;

[0026] When the encoding condition of the TMRL mode of the current block is satisfied, encoding the TMRL mode flag of the current block to indicate that the current block uses the TMRL mode, and encoding the TMRL mode index of the current block to indicate the position of the selected combination in the candidate list;

[0027] Encode the TMRL fusion flag of the current block to indicate whether the current block uses intra-frame prediction fusion IPF or not;

[0028] The encoding condition at least includes: the selected combination is in the candidate list.

[0029] An embodiment of the present disclosure further provides a method for constructing a multi-reference row intra prediction mode candidate list, including:

[0030] Obtain N×M original combinations of extended reference lines and angle modes according to the N extended reference lines and M intra prediction modes of the current block;

[0031] Obtaining a corresponding fusion combination according to a combination including a predetermined angle pattern in the N×M original combinations;

[0032] Predicting the template region of the current block according to the N×M original combinations and the obtained fusion combination, and calculating the error between the reconstructed value of the template region and the predicted value;

[0033] In order of the errors from small to large, K combinations corresponding to the errors are filled into a candidate list of the template-based multi-reference row intra prediction TMRL mode of the current block, where K, N, and M are set positive integers, and 1≤K≤N×M.

[0034] An embodiment of the present disclosure further provides a method for constructing a multi-reference row intra prediction mode candidate list, including:

[0035] Obtain N×M original combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block;

[0036] Predicting the template area of ​​the current block respectively according to the N×M original combinations, and calculating the error between the reconstructed value of the template area and the predicted value;

[0037] For each of the K original combinations with the smallest error, including a combination of predetermined angle modes, determine whether fusion is required. If fusion is required, fill the fused combination corresponding to the original combination into the candidate list of the template-based multi-reference row intra-frame prediction TMRL mode of the current block. If fusion is not required, fill the original combination into the candidate list, where K, N, and M are set positive integers, and 1≤K≤N×M.

[0038] An embodiment of the present disclosure further provides a method for constructing a multi-reference row intra prediction mode candidate list, including:

[0039] Obtain N×M original combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block;

[0040] Performing a fusion process on the N×M original combinations, the fusion process comprising: for each original combination including a predetermined angle pattern, when a set condition is met, replacing the original combination with a corresponding fusion combination;

[0041] Predicting the template area of ​​the current block according to the N×M combinations after the fusion process, and calculating the error between the reconstructed value of the template area and the predicted value;

[0042] Filling K combinations corresponding to the errors into a candidate list of a template-based multiple reference row intra prediction (TMRL) mode for the current block in an ascending order of the errors;

[0043] Among them, K, N, M are set positive integers, 1≤K≤N×M;

[0044] In which, when predicting the current block based on the fusion combination corresponding to the original combination, the weighted sum of the first prediction result and the second prediction result is used as the prediction value of the current block; wherein, the first prediction result is the result of predicting the current block based on the extended reference row and angle pattern in the original combination, and the second prediction result is the result of predicting the current block based on the second reference row and the angle pattern in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

[0045] An embodiment of the present disclosure further provides a code stream, wherein the code stream is generated by the video encoding method described in any embodiment of the present disclosure.

[0046] An embodiment of the present disclosure further provides an intra-frame prediction fusion device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the intra-frame prediction fusion method as described in any embodiment of the present disclosure.

[0047] An embodiment of the present disclosure also provides a device for constructing a multi-reference row intra-frame prediction mode candidate list, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the method for constructing a multi-reference row intra-frame prediction mode candidate list described in any embodiment of the present invention.

[0048] An embodiment of the present disclosure further provides a video decoding device, comprising a processor and a memory storing a computer program, wherein the processor can implement the video decoding method described in any embodiment of the present disclosure when executing the computer program.

[0049] An embodiment of the present disclosure further provides a video encoding device, including a processor and a memory storing a computer program, wherein the processor can implement the video encoding method described in any embodiment of the present disclosure when executing the computer program.

[0050] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes the video encoding device described in any embodiment of the present disclosure and the video decoding device described in any embodiment of the present disclosure.

[0051] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the intra-frame prediction fusion method described in any embodiment of the present disclosure, the method for constructing a multi-reference row intra-frame prediction mode candidate list described in any embodiment of the present disclosure, or implements the video decoding method described in any embodiment of the present disclosure, or implements the video encoding method described in any embodiment of the present disclosure.

[0052] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0053] Summary of the Figures

[0054] The accompanying drawings are used to provide an understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation to the technical solutions of the present disclosure.

[0055] FIG1A is a schematic diagram of a coding and decoding system according to an embodiment of the present disclosure;

[0056] FIG1B is a framework diagram of an encoding end according to an embodiment of the present disclosure;

[0057] FIG1C is a framework diagram of a decoding end according to an embodiment of the present disclosure;

[0058] FIG2 is a schematic diagram of an intra-frame prediction mode according to an embodiment of the present disclosure;

[0059] FIG3 is a schematic diagram of adjacent intra-frame prediction blocks of a current block according to an embodiment of the present disclosure;

[0060] FIG4 is a schematic diagram of a template of a current block and a template reference area according to an embodiment of the present disclosure;

[0061] FIG5 is a schematic diagram of multiple reference rows around a current block according to an embodiment of the present disclosure;

[0062] FIG6 is a flowchart of a video encoding method according to an embodiment of the present disclosure;

[0063] FIG. 7 is a flowchart of a method for constructing a TMRL pattern candidate list according to an embodiment of the present disclosure.

[0064] FIG8A is a schematic diagram of a template area around a current block and an extended reference row according to an embodiment of the present disclosure;

[0065] FIG8B is a schematic diagram of a template area around a current block and an extended reference row according to another embodiment of the present disclosure;

[0066] FIG9 is a flowchart of a video decoding method according to an embodiment of the present disclosure.

[0067] FIG10 is a flowchart of an intra-frame prediction fusion method according to an embodiment of the present disclosure;

[0068] FIG11 is a flowchart of a video encoding method according to another embodiment of the present disclosure;

[0069] FIG12 is a flowchart of a video decoding method according to another embodiment of the present disclosure;

[0070] FIG13 is a flowchart of a video decoding method according to another embodiment of the present disclosure;

[0071] FIG14 , FIG15 and FIG16 are respectively a method and a flowchart of constructing a TMRL candidate list in three embodiments of the present disclosure;

[0072] FIG17 is a schematic diagram of a device for constructing a TMRL mode candidate list according to an embodiment of the present disclosure;

[0073] FIG18 is a flowchart of a video encoding method according to another embodiment of the present disclosure.

[0074] Details

[0075] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure.

[0076] In the description of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be interpreted as being more preferred or advantageous than other embodiments. "And / or" in this article is a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "Multiple" refers to two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0077] When describing representative exemplary embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can readily understand that these sequences can vary and still remain within the spirit and scope of the disclosed embodiments.

[0078] The local illumination compensation method and video coding and decoding method proposed in the embodiments of the present disclosure can be applied to various video coding and decoding standards, such as: H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), AVS (Audio Video Coding Standard), and other standards developed by MPEG (Moving Picture Experts Group), AOM (Alliance for Open Media), JVET (Joint Video Experts Team) and extensions of these standards, or any other customized standards.

[0079] FIG1A is a block diagram of a video encoding and decoding system that can be used in an embodiment of the present disclosure. As shown in the figure, the system is divided into an encoding end device 1 and a decoding end device 2. The encoding end device 1 generates a code stream. The decoding end device 2 can decode the code stream. The decoding end device 2 can receive the code stream from the encoding end device 1 via a link 3. The link 3 includes one or more media or devices capable of moving the code stream from the encoding end device 1 to the decoding end device 2. In one example, the link 3 includes one or more communication media that enable the encoding end device 1 to send the code stream directly to the decoding end device 2. The encoding end device 1 modulates the code stream according to a communication standard (e.g., a wireless communication protocol) and sends the modulated code stream to the decoding end device 2. The one or more communication media may include wireless and / or wired communication media and may form part of a packet network. In another example, the code stream can also be output from the output interface 15 to a storage device, and the decoding end device 2 can read the stored data from the storage device via streaming or downloading.

[0080] As shown in the figure, the encoding end device 1 includes a data source 11, a video encoding device 13 and an output interface 15. The data source 11 includes a video capture device (such as a camera), an archive containing previously captured data, an input interface for receiving data from a content provider, a computer graphics system for generating data, or a combination of these sources. The video encoding device 13 encodes the data from the data source 11 and outputs it to the output interface 15. The output interface 15 may include at least one of a regulator, a modem and a transmitter. The decoding end device 2 includes an input interface 21, a video decoding device 23 and a display device 25. The input interface 21 includes at least one of a receiver and a modem. The input interface 21 can receive a code stream via the link 3 or from a storage device. The video decoding device 23 decodes the received code stream. The display device 25 is used to display the decoded data. The display device 25 can be integrated with other devices of the decoding end device 2 or set separately. The display device 25 is optional for the decoding end. In other examples, the decoding end may include other devices or equipment that apply the decoded data.

[0081] Based on the video coding and decoding system shown in FIG. 1A , various video coding and decoding methods can be used to implement video compression and decompression.

[0082] FIG1B is a block diagram of an exemplary video encoding device that can be used in embodiments of the present disclosure. As shown, the video encoding device 1000 includes a prediction unit 1100, a partitioning unit 1101, a residual generation unit 1102 (indicated by the circle with a plus sign after the partitioning unit 1101 in the figure), a transform processing unit 1104, a quantization unit 1106, an inverse quantization unit 1108, an inverse transform processing unit 1110, a reconstruction unit 1112 (indicated by the circle with a plus sign after the inverse transform processing unit 1110 in the figure), a filter unit 1113, a decoded image buffer 1114, and an entropy coding unit 1115. The prediction unit 1100 includes an inter-frame prediction unit 1121 and an intra-frame prediction unit 1126. The decoded image buffer 1114 may also be referred to as a decoded image buffer, a decoded picture buffer, or a decoded picture buffer. The video encoder 20 may also include more, fewer, or different functional components than shown in this example. For example, the transform processing unit 1104 and the inverse transform processing unit 1110 may be eliminated in some cases.

[0083] The division unit 1101 cooperates with the prediction unit 1100 to divide the received video data into slices, coding tree units (CTUs) or other larger units. The video data received by the division unit 1101 can be a video sequence including video frames such as I frames, P frames or B frames.

[0084] The prediction unit 1100 can divide a CTU into coding units (CUs) and perform intra-frame prediction coding or inter-frame prediction coding on the CU. When performing intra-frame prediction and inter-frame prediction on the CU, the CU can be divided into one or more prediction units (PUs).

[0085] The inter-frame prediction unit 1121 can perform inter-frame prediction on the PU and generate prediction data for the PU, which includes the prediction block of the PU, the motion information of the PU, and various syntax elements. The inter-frame prediction unit 1121 can include a motion estimation (ME) unit and a motion compensation (MC) unit. The motion estimation unit can be used to estimate motion to generate a motion vector, and the motion compensation unit can be used to obtain or generate a prediction block based on the motion vector.

[0086] The intra-prediction unit 1126 may perform intra prediction on the PU and generate prediction data for the PU. The prediction data for the PU may include a prediction block and various syntax elements for the PU.

[0087] The residual generation unit 1102 may subtract the prediction blocks of the PUs into which the CU is partitioned from the original blocks of the CU to generate the residual blocks of the CU.

[0088] The transform processing unit 1104 may divide a CU into one or more transform units (TUs). The division of a prediction unit (PU) and a transform unit (TU) may be different. A TU-associated residual block is a sub-block obtained by dividing the residual block of the CU. One or more transforms are applied to the TU-associated residual block to generate a TU-associated coefficient block.

[0089] The quantization unit 1106 may quantize the coefficients in the coefficient block based on the selected quantization parameter. The degree of quantization of the coefficient block may be adjusted by adjusting the quantization parameter (QP: Quantizer Parameter).

[0090] The inverse quantization unit 1108 and the inverse transform unit 1110 may apply inverse quantization and inverse transform to the coefficient block, respectively, to obtain a reconstructed residual block associated with the TU.

[0091] The reconstruction unit 1112 may add the reconstructed residual block and the prediction block generated by the prediction unit 1100 to generate a reconstructed image.

[0092] Filter unit 1113 performs loop filtering on the reconstructed image and stores the filtered reconstructed image in decoded image buffer 1114 as a reference image. Intra-frame prediction unit 1126 extracts reference images of blocks adjacent to the PU from decoded image buffer 1114 to perform intra-frame prediction. Inter-frame prediction unit 1121 can use the reference image of the previous frame cached in decoded image buffer 1114 to perform inter-frame prediction on the PU of the current frame image.

[0093] The entropy coding unit 1115 may perform entropy coding operations on received data (eg, syntax elements, quantized coefficient blocks, motion information, etc.).

[0094] FIG1C is a block diagram of an exemplary video decoding device that can be used in embodiments of the present disclosure. As shown, the video decoding device 101 includes an entropy decoding unit 150, a prediction unit 152, an inverse quantization unit 154, an inverse transform unit 156, a reconstruction unit 158 ​​(represented by the circle with a plus sign after the inverse transform unit 155), a filter unit 159, and a decoded image buffer 160. In other embodiments, the video decoder 30 may include more, fewer, or different functional components. For example, the inverse transform unit 155 may be eliminated in some cases.

[0095] Entropy decoding unit 150 may perform entropy decoding on the received bitstream to extract syntax elements, quantized coefficient blocks, motion information for PUs, etc. Prediction unit 152, inverse quantization unit 154, inverse transform processing unit 156, reconstruction unit 158, and filter unit 159 may each perform corresponding operations based on the syntax elements extracted from the bitstream.

[0096] The inverse quantization unit 154 may perform inverse quantization on the coefficient block associated with the quantized TU.

[0097] Inverse transform processing unit 156 may apply one or more inverse transforms to the inverse quantized coefficient block to generate a reconstructed residual block for the TU.

[0098] Prediction unit 152 includes inter-prediction unit 162 and intra-prediction unit 164. If a PU is encoded using intra prediction, intra prediction unit 164 may determine an intra prediction mode for the PU based on syntax elements decoded from the codestream, perform intra prediction based on the determined intra prediction mode and reconstructed reference information of the PU's neighbors obtained from decoded picture buffer 160, and generate a prediction block for the PU. If a PU is encoded using inter prediction, inter prediction unit 162 may determine one or more reference blocks for the PU based on the motion information of the PU and corresponding syntax elements, and generate a prediction block for the PU based on the reference blocks obtained from decoded picture buffer 160.

[0099] The reconstruction unit 158 ​​may obtain a reconstructed image based on the reconstructed residual block associated with the TU and the prediction block of the PU generated by the prediction unit 152 .

[0100] The filter unit 159 may perform loop filtering on the reconstructed image, and the filtered reconstructed image is stored in the decoded image buffer 160. The decoded image buffer 160 may provide a reference image for subsequent motion compensation, intra-frame prediction, inter-frame prediction, etc., and may also output the filtered reconstructed image as decoded video data for presentation on a display device.

[0101] Based on the above-described video encoding and decoding devices, the following basic encoding and decoding process can be performed. On the encoding side, a frame of an image is divided into blocks. Intra-frame prediction, inter-frame prediction, or other algorithms are performed on the current block to generate a predicted block for the current block. The predicted block is subtracted from the original block of the current block to obtain a residual block. The residual block is transformed and quantized to obtain quantized coefficients, and the quantized coefficients are entropy encoded to generate a bitstream. On the decoding side, intra-frame prediction or inter-frame prediction is performed on the current block to generate a predicted block for the current block. The quantized coefficients obtained from the decoded bitstream are then inversely quantized and inversely transformed to obtain a residual block. The predicted block and residual block are added together to obtain a reconstructed block. The reconstructed block forms a reconstructed image. The reconstructed image is then subjected to image-based or block-based loop filtering to obtain a decoded image. The encoding side also performs similar operations as the decoding side to obtain a decoded image, which is often referred to as a reconstructed image. The decoded image obtained by the encoding side can serve as a reference frame for inter-frame prediction of subsequent frames. The block division information determined by the encoding side, as well as information and parameters for prediction, transform, quantization, entropy coding, and loop filtering modes, can be written into the bitstream if necessary. The decoding end determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information and parameter information as the encoding end by decoding the code stream or analyzing the existing information, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end.

[0102] Although the above is an example of a block-based hybrid coding framework, the embodiments of the present disclosure are not limited thereto. With the development of technology, one or more modules in the framework and one or more steps in the process can be replaced or optimized.

[0103] Herein, the current block may be a block-level coding unit such as a current coding unit (CU) or a current prediction unit (PU) in a current image.

[0104] When performing intra-frame prediction, the encoder typically uses various angular and non-angular modes to predict the current block to obtain a predicted block. Based on the rate-distortion information calculated between the predicted block and the original block, the encoder selects the optimal intra-frame prediction mode for the current block. This intra-frame prediction mode is encoded and transmitted via the bitstream to the decoder. The decoder decodes the selected intra-frame prediction mode and performs intra-frame prediction on the current block using this intra-frame prediction mode. In this article, the reference row and intra-frame prediction mode selected for the current block are also referred to as the reference row and intra-frame prediction mode selected for the current block.

[0105] In VVC and ECM, various traditional intra-frame prediction modes use reconstructed information around the current block to predict the current block. These include: planar mode (mode index 0), mean mode (DC mode, mode index 1), and 65 angular prediction modes (mode indices 2 to 66). Figure 2 shows the angular directions of the angular modes with mode indices 2 to 66. Herein, the angular prediction modes are referred to as angular modes.

[0106] Since rectangular prediction blocks are introduced in VVC, for rectangular blocks, the angle directions of some angle modes with indexes of 2 to 66 can be replaced with wider angle directions. As shown in the figure, the angle modes with indexes of -14 to -1, 67 to 80 are angle modes obtained by wide-angle replacement. The selection of these angle modes does not need to be represented by a flag bit, but is obtained through the correspondence between the shape of the current block and the prediction mode index (2 to 66) selected by the current block.

[0107] In the VVC standard text, wide angle is replaced as follows:

[0108] The variable whRatio is Abs(Log2(blockWidth)-Log2(blockHeight)).

[0109] For non-square prediction blocks, the angular prediction mode predModeIntra (2 to 66) is replaced with a wide angle according to whether the following conditions are met. predModeIntra is represented by an index:

[0110] If all three of the following conditions are met, predModeIntra will be equal to (predModeIntra + 65).

[0111] The current block width is greater than its height

[0112] Angle mode index greater than or equal to 2

[0113] Angle mode index less than (whRatio>1)? (8+2*whRatio):8

[0114] Otherwise, if all three of the following conditions are met, predModeIntra will be equal to (predModeIntra - 67).

[0115] The current block is taller than it is wide

[0116] Angle mode index is less than or equal to 66

[0117] Angle mode index greater than (whRatio>1)? (60-2*whRatio): 60

[0118] After completing the matching of the wide angle mode, each angle prediction mode predModeIntra (-14 to 80) will have an angle value intraPredAngle.

[0119] Table 1. Correspondence between angle prediction mode and angle value

[0120] predModeIntra-14-13-12-11-10-9-8-7-6-5-4-3-2-1234intraPredAngle5123412561711281028673645751453935322926predModeIntra567891011121 31415161718192021intraPredAngle232018161412108643210-1-2-3predModeIntra2223242526272829303132333435363738intraPredAngle-4-6-8-10 -12-14-16-18-20-23-26-29-32-29-26-23-20predModeIntra3940414243444546474849505152535455intraPredAngle-18-16-14-12-10-8-6-4-3-2-10 12346predModeIntra5657585960616263646566676869707172intraPredAngle810121416182023262932353945515764predModeIntra7374757677787980 intraPredAngle7386102128171256341512

[0121] The angle value of the angle mode will be used for subsequent angle prediction.

[0122] In this article, each angle mode predModeIntra corresponds to an angle, and the angle of each angle prediction mode is the angle of the line segment corresponding to the angle prediction mode in Figure 1 in the rectangular coordinate system. For example, the angle mode with index number 34, the angle value intraPredAngle is -32, and the angle is -45° or 45° or 135°, which is related to the 0° direction defined by the rectangular coordinate system.

[0123] Unless otherwise specified, the intra-frame prediction mode mentioned herein refers to a traditional intra-frame prediction mode including a Planar mode, a DC mode, and an angular mode.

[0124] If the intra prediction mode of the current block is directly encoded, 7 bits are required to encode each of the 67 modes, resulting in a very large amount of data. Based on statistical characteristics, pixels closer to the current block are more likely to use the same intra prediction mode as the current block. This characteristic is why most probable mode (MPM) technology is adopted in HEVC, VVC, and the Enhanced Compression Model (ECM). ECM is based on the VTM-10.0 reference software and integrates various new tools to further explore codec performance.

[0125] MPM first constructs an MPM list and populates it with the six intra prediction modes most likely to be selected by the current block. If the intra prediction mode selected by the current block is in the MPM list, only its index number (only 3 bits) needs to be encoded. If the intra prediction mode selected by the current block is not in the MPM list but is one of the 61 non-MPM modes, the intra prediction mode is encoded using Truncated Binary Code (TBC) during the entropy coding stage.

[0126] In VVC, regardless of whether multiple reference lines (MRL) and intra sub-block partitioning (ISP) are applied, the MPM list has 6 prediction modes. The MPM in ECM is divided into MPM and Secondary MPM, which use lists of length 6 and length 16 respectively. Of the 6 modes in the MPM list, the Planar mode is always filled in the first position in the MPM. The remaining 5 positions are filled in the following three steps in sequence until all 5 positions are filled. The extra modes will automatically enter the Secondary MPM.

[0127] In the first step, the intra prediction modes used by the prediction blocks at the five adjacent positions around the current block are sequentially filled; as shown in Figure 3, the five positions include the upper left (AL), upper (A), upper right (AR), left (L) and lower left (BL) positions of the current block.

[0128] In the second step, the pattern derived from the gradient histogram is used based on the reconstructed pixels around the current block;

[0129] In the third step, select an angle mode that is close to the angle mode selected in the first step.

[0130] The Secondary MPM list may consist of some main angle modes other than the intra prediction modes in the MPM.

[0131] Since the encoding and decoding order of the MPM flag (mpm_flag) is after the MRL mode, the encoding and decoding of the MPM in ECM depends on the MRL flag. When the current block does not use the MRL mode, the MPM flag needs to be decoded to determine whether the current block uses the MPM. When the current block uses the MRL mode, there is no need to decode the MPM flag, and the current block uses the MPM by default.

[0132] Template-based intra mode derivation (TIMD) and decoder-side intra mode derivation (DIMD) are two intra prediction technologies that are not included in the VVC standard but are adopted in the ECM reference software. These two technologies can derive the intra prediction mode of the current block based on the reconstructed pixel values ​​around the current block at the decoder, thereby eliminating the need to encode the intra prediction mode index and thus saving bits.

[0133] TIMD is an intra-frame prediction mode for luma frames. The TIMD mode is generated by selecting candidate intra-frame prediction modes from the MPM list and using a template region (referred to as the template). In ECM, as shown in Figure 4, the left and upper adjacent regions of a current block (e.g., the current CU) 11 constitute the template region 12 of the current block. The left adjacent region is referred to as the left template region (referred to as the left template), and the upper adjacent region is referred to as the upper template region or upper template region (referred to as the upper template).

[0134] As shown in the figure, a template reference area 13 is provided outside (left and top of) template area 12. Exemplary sizes and positions of each area are shown. In one example, the width L1 of the left template and the height L2 of the top template are both 4. Template reference area 13 can be located in a row above or a column to the left of the template area.

[0135] TIMD assumes that the distribution characteristics of the current block and the template area of ​​the current block are consistent. The reconstructed value of the template reference area is used as the reconstructed value of the reference row. The template area is predicted through all intra-frame prediction modes in the MPM and Secondary MPM to obtain a prediction result. The error between the reconstructed value on the template area and the prediction result of each mode is then calculated and expressed as the sum of absolute transformed differences (SATD). The optimal intra-frame prediction mode with the smallest SATD is selected and used as the TIMD mode for the current block. The decoding end can derive the TIMD mode using the same derivation method. If the sequence allows the use of TIMD, each current block requires a flag bit to indicate whether TIMD is used. If the intra-frame prediction mode selected for the current block is TIMD mode, the current block is predicted using TIMD mode, and the decoding process of the remaining syntax elements related to intra-frame prediction, such as ISP and MPM, can be skipped, thereby significantly reducing the coding bits of the mode.

[0136] After obtaining the SATD between the reconstructed value of the template area and the prediction result of each mode (the predicted value of the template area), the final TIMD mode to be used can be determined according to the following method:

[0137] Assume that mode1 and mode2 are two angle modes for intra-frame prediction in MPM, mode1 is the angle mode with the smallest SATD, and its SATD is cost1; mode2 is the angle mode with the second smallest SATD, and its SATD is cost2:

[0138] When cost1×2≤cost2, mode1 is used as the TIMD mode of the current block;

[0139] When cost1×2>cost2, the prediction mode obtained by weighting the prediction results of mode1 and mode2 is used as the TMID mode of the current block, also known as the TIMD fusion mode.

[0140] The weighting method and weights are shown in the following formula:

[0141] Pred=Pred mode1 ×w1+Pred mode2 ×w2

[0142]

[0143] Among them, Pred is the prediction result of the current block using TIMD fusion mode, Pred mode1 Use the prediction result of mode1 for the current block, Predmode2 The prediction result of mode2 is used for the current block, and w1 and w2 are the weights calculated according to cost1 and cost2.

[0144] DIMD uses the reconstructed pixel values ​​surrounding the current block as a template. The Sobel operator is used to scan each 3x3 region on the template and calculate the horizontal and vertical gradients. The horizontal and vertical gradients Dx and Dy are then calculated. Based on Dx and Dy, the amplitude value Amp = abs(Dx) + abs(Dy) and the angle value angular = arctan(Dy / Dx) are calculated at each location. The angle value at each location on the template is mapped to a traditional angle pattern, and the amplitude values ​​for the same angle pattern are accumulated to form a histogram of the amplitude and angle patterns. When the two angle patterns with the highest and second highest amplitude values ​​exist, the prediction values ​​of the two angle patterns and the planar mode are weighted to obtain the final prediction result for the current block using DIMD. This prediction mode combines three intra-frame prediction modes: the planar mode and the two angle patterns with the highest and second highest amplitude values. This is referred to as the DIMD fusion mode in this paper. When the two angle patterns with the highest and second highest amplitude values ​​do not exist, the prediction using DIMD is equivalent to the planar mode prediction.

[0145] In HEVC, intra prediction uses the row and column immediately above and to the left of the current block as references. If the reconstructed values ​​of these rows and columns differ significantly from the original pixel values, the prediction quality of the current block can be significantly affected. To address this issue, VVC employs multiple reference line (MRL) intra prediction technology. In addition to using reference line 0 (reference line 0), VVC can also use reference lines 1 (reference line 1) and 2 (reference line 2) as extended reference lines for intra prediction. To reduce coding complexity, MRL is only used in non-planar modes within the MPM. When predicting each angular mode, the encoder tries all three reference lines. Through rate-distortion optimization, the reference line with the lowest rate-distortion cost (RD cost) is selected for the current block. The index of this selected reference line is encoded and sent to the decoder. The decoder decodes the reference line index and then uses it to determine the selected reference line for the current block, which is then used for prediction.

[0146] The example shown in FIG5 shows four reference lines for the current block, including reference line 0 221 adjacent to the current block, i.e., a reference line with an index of 0; reference line 1 222 one line away from the current block, i.e., a reference line with an index of 1; reference line 2 223 two lines away from the current block, i.e., a reference line with an index of 2; and reference line 3 224 three lines away from the current block, i.e., a reference line with an index of 3. The current block can have more reference lines, i.e., reference lines with an index of 4 or more. Prediction can use only the reconstructed values ​​of a portion of the reference lines. In this document, the reference line indices are numbered as shown in FIG5 .

[0147] In this article, although the reference row is called a "row", this is for convenience of expression. A reference row actually includes one row and one column. Generally, the reconstructed value of the reference row used in prediction also includes the reconstructed value of one row and one column, which is the same as the usual description method in the industry.

[0148] In ECM, the MRL mode can use more reference lines. In order to encode the reference line selected by the current block, the indices of multiple candidate reference lines are filled into a list. This list is called a multiple reference line index list, abbreviated as MRL index list, and can also be called a multiple reference line list, a candidate reference line list, a reference line index list, etc. When the current block does not use TIMD, the length of the MRL index list is 6, that is, there are 6 positions in total, which can be filled with the indexes of 6 reference lines. The indexes of these 6 reference lines and their order are fixed, which are 0, 1, 3, 5, 7, 12, respectively, and can be expressed as follows: MULTI_REF_LINE_IDX[6] = {0, 1, 3, 5, 7, 12}. In this MRL index list, the index filled in the first position is 0, which is the index of the reference line closest to the current block. The indices filled in the second to sixth positions are 1, 3, 5, 7, 12, respectively, which are the indices of the 5 extended reference lines arranged in order from near to far in terms of distance from the current block.

[0149] When the reference row selected by the current block is in the MRL index list, a multi-reference row index (multiRefIdx) is used to indicate the position of the reference row selected by the current block in the MRL index list, and the encoded MRL index can indicate the selected reference row. Take the MRL index list {0,1,3,5,7,12} as an example, where the MRL indexes corresponding to the 1st to 6th positions are 0 to 5 respectively. Assuming that the current block selects the reference row with index 0, the MRL index is 0. Assuming that the current block selects the reference row with index 7, the MRL index is 4, and so on for other cases. The MRL index can be encoded using a unary truncated code based on the context model, and after encoding, multiple binary identifiers based on the context model are obtained. The binary identifier can also be called a binary identifier bit, a binary symbol, a binary bit, etc. The smaller the value of the MRL index, the shorter the code length and the faster the decoding.

[0150] The MRL mode can also be used simultaneously with the TIMD mode. When using TIMD, the length of the MRL index list is 3, which can be filled with the indexes of 3 reference lines and the order between the indexes is fixed, expressed as MULTI_REF_LINE_IDX[3]={0,1,3}.

[0151] It should be noted that the same technology may have different names in different standards. For example, a technology like MPM, which uses blocks surrounding the current block to derive a list of most likely modes, is called Adaptive Intra Mode Coding (AIMC) in AV2 (AVM), and is called Frequency-based Intra Mode Coding (FIMC) in AVS3 for screen content coding. In non-screen content coding, technologies similar to MPM are always enabled. And a technology like MRL, which uses multiple reference lines for intra prediction, is called Multiple Reference Line Selection for Intra Prediction (MRLS) in AV2 (AVM). However, this is merely a difference in name; when this embodiment uses terms such as MPM and MRL, it should also cover these essentially identical technologies in other standards.

[0152] One embodiment provides a template-based multiple reference line intra-frame prediction (Multiple Reference Line & Intra_Intra Prediction) mode, abbreviated as TMRL mode. The TMRL mode is a prediction mode that constructs a candidate list based on a combination of extended reference lines and intra-frame prediction modes, and encodes and decodes the combination of extended reference lines and intra-frame prediction modes.

[0153] The video encoding method of this embodiment is applied to an encoder, as shown in FIG6 , and includes:

[0154] Step 110 , constructing a candidate list of the TMRL mode of the current block, wherein the candidate list is filled with combinations of extended reference rows and intra prediction modes of the current block candidates;

[0155] Step 120 , through rate-distortion optimization, the current block selects a combination of a reference row and an intra prediction mode for intra prediction;

[0156] In this article, the reference lines include the reference line with index 0 and the extended reference lines. The combination of a reference line and an intra-frame prediction mode selected by the current block may be a combination of the reference line with index 0 and an intra-frame prediction mode, or a combination of an extended reference line and an intra-frame prediction mode.

[0157] Step 130: When the coding condition of the TMRL mode of the current block is satisfied, encode the TMRL mode flag of the current block to indicate that the current block uses the TMRL mode, and encode the TMRL mode index of the current block to indicate the position of the selected combination in the candidate list;

[0158] The encoding condition at least includes: the selected combination is in the candidate list.

[0159] Herein, the candidate list filled with combinations of extended reference rows of current block candidates and intra prediction modes may also be referred to as a candidate list of the TMRL mode.

[0160] In this paper, the candidate list is populated with combinations of extended reference rows and intra prediction modes for the current block candidate. This means that the combinations in the candidate list must participate in the rate-distortion optimization of the current block, that is, the mode selection process that selects the current block prediction mode based on the rate-distortion cost. This ensures that all combinations in the candidate list have a chance of being selected.

[0161] In this article, N, M, and K, which represent the number of parameters, are all positive integers and need no additional explanation.

[0162] The candidate list of TMRL modes constructed by this embodiment is filled with the combination of the extended reference row and the intra-frame prediction mode of the current block candidate, and is no longer a single list of candidate extended reference rows or a list of candidate intra-frame prediction modes. When the combination of the reference row and the intra-frame prediction mode selected by the current block is in the candidate list (in this case, the current block selects the extended reference row), that is, the selected combination is a combination in the candidate list and other coding conditions are met, the TMRL mode flag of the current block is encoded to indicate that the current block uses the TMRL mode, and the TMRL mode index of the current block is encoded to indicate the position of the selected combination in the candidate list; the decoding end can determine the extended reference row and intra-frame prediction mode selected by the current block based on the TMRL mode flag and the TMRL mode index. The combined coding and decoding method of this embodiment can reduce coding cost and improve coding performance.

[0163] The TMRL mode is N×M combinations obtained based on the N extended reference rows and M intra-frame prediction modes of the current block. The template area of ​​the current block is predicted respectively, and the error between the reconstructed value of the template area and the predicted value is calculated; the corresponding K combinations are filled into the candidate list of the TMRL mode of the current block in the order of error from small to large, 1≤K≤N×M.

[0164] In one example, a template region is predicted based on 25 combinations of extended reference lines and intra-frame prediction modes in the ECM. These 25 combinations are sorted in ascending order of error, and the 12 combinations with the smallest errors, i.e., the most likely selections, are added to the candidate list of TMRL modes. This allows the TMRL mode index to be encoded using fewer coding bits. Even if some of the extended reference lines {1, 3, 5, 7, 12} (such as those with indices 7 and 12) are outside the CTU boundary, the 15 combinations can be sorted and the 12 most likely selections can be added to the candidate list, allowing the coding bits of the TMRL mode index to be fully and efficiently utilized. Sorting by ascending error ensures that the reference lines and prediction modes that are more likely to be selected for prediction are retained in the candidate list. Combinations that are more likely to be selected are placed closer to the top of the list, reducing coding cost. The creation of the TMRL mode candidate list will be described in more detail below.

[0165] In an exemplary embodiment of this invention, the encoding condition further includes: the current block does not use TIMD; and the method further includes:

[0166] When the current block uses TIMD, the encoding of the TMRL mode flag and TMRL mode index of the current block is skipped;

[0167] When the current block does not use TIMD but the selected combination is not in the candidate list, the TMRL mode flag of the current block is encoded to indicate that the current block does not use the TMRL mode, and encoding of the TMRL mode index of the current block is skipped.

[0168] This embodiment is based on the situation where the TIMD mode is encoded and decoded before the TMRL mode. If the current block uses the TIMD mode, there is no need to use the TMRL mode, so the encoding of the TMRL mode flag and the TMRL mode index is skipped. If the current block does not use the TIMD mode, there are two situations: the combination selected by the current block is in the candidate list of the TMRL mode, or it is not in the candidate list of the TMRL mode. If the selected combination is not in the candidate list. It is necessary to encode the TMRL mode flag to indicate that the current block does not use the TMRL mode, and skip the encoding of the TMRL mode index. If the selected combination is in the candidate list, both the TMRL mode flag and the TMRL mode index need to be encoded.

[0169] When the current block does not use TIMD, the TMRL mode flag and TMRL mode index provided in this embodiment can replace the original multi-reference row index multiRefIdx. However, when the current block uses TIMD, the multi-reference row index can still be used to indicate the selected reference row and encode the multi-reference row index.

[0170] In an example of this embodiment, constructing a candidate list for the TMRL mode of the current block includes: constructing the candidate list only when all conditions for allowing the current block to use the TMRL mode are met, and the conditions for allowing the use of the TMRL mode include any one or more of the following:

[0171] Condition 1: the current block is a block in a luma frame; that is, the TMRL mode is only used for luma frames (i.e., luma images);

[0172] Condition 2: The current block is not located at the upper boundary of the coding tree unit (CTU). If it is located at the upper boundary of the CTU, no reference rows above the current block can be used. Therefore, this embodiment uses not being located at the upper boundary of the CTU as a condition for allowing the use of the TMRL mode.

[0173] Condition 3: The current block allows the use of multiple reference lines (MRL), that is, the TMRL mode is allowed only when the multiple reference line mode is allowed.

[0174] Condition 4: The size of the current block is no larger than the maximum size of the current block for which the TMRL mode can be used; this maximum size can be preset. Larger blocks are generally flatter and less likely to have angular details. For such large blocks, the use of the TMRL mode can be limited.

[0175] Condition 5: The aspect ratio of the current block meets the requirements of the TMRL mode for the aspect ratio of the current block. For example, the TMRL mode is allowed only when the aspect ratio of the current block is not greater than a preset value.

[0176] In an example of this embodiment, the TMRL pattern index is encoded using the Columbus-Rice coding method. Using the Columbus-Rice coding method can more reasonably classify the candidate combinations into categories with different codeword lengths for encoding and decoding, thereby improving coding efficiency.

[0177] In an example of this embodiment, when the encoding conditions of the TMRL mode of the current block are met, the method also includes: skipping the encoding of syntax elements of any one or more of the following modes: MPM mode, intra-frame sub-block partitioning ISP mode, multi-transform selection MTS mode, low-frequency non-divisible transform LFNST mode, TIMD mode.

[0178] For example, when the TMRL mode is placed before the TIMD mode for encoding and decoding, the encoding conditions of the TMRL mode no longer include that the current block does not use TIMD. When the current block uses the TMRL mode, the current block is not allowed to use TIMD, and the encoding of the TIMD mode syntax elements can be skipped.

[0179] For example, when the current block uses the TMRL mode, the TMRL mode flag and the TMRL mode index can simultaneously indicate the reference row and intra prediction mode selected by the current block. In this case, there is no need to encode or decode the MPM-related syntax elements.

[0180] For example, in a specific transform mode, the TMRL mode may be restricted from being used simultaneously with the multiple transform selection (MTS) mode and / or the low frequency non-separable transform (LFNST) mode.

[0181] One embodiment provides a method for constructing a TMRL mode candidate list, which can be applied to both an encoder and a decoder. As shown in FIG7 , the method includes:

[0182] Step 210 , obtaining N×M combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block, where N≥1, M≥1, and N×M≥2;

[0183] Step 220 , predicting the template region of the current block according to the N×M combinations, and calculating the error between the reconstructed value of the template region and the predicted value;

[0184] The error of this step can be expressed by the sum of absolute difference (SAD) or the sum of absolute transformed difference (SATD), but is not limited to this. It can also be expressed by the sum of squared difference (SSD), mean absolute difference (MAD), mean squared error (MSE), etc.

[0185] Step 230 , filling K combinations corresponding to the errors into a candidate list of the TMRL mode of the current block in order from small to large, 1≤K≤N×M.

[0186] The candidate list created in this embodiment can implement combined coding of extended reference lines and intra-frame prediction modes, improving coding efficiency. Furthermore, by using different combinations to predict the template region and sorting the errors, it is possible to select K combinations with a high probability of being selected from N×M combinations based on the similarity in distribution characteristics between the current block and the current block template region. These combinations with a high probability of being selected are placed at the front of the candidate list, resulting in a smaller TMRL mode index for the selected combination during coding, reducing the actual coding cost.

[0187] In one example of this embodiment, the template region of the current block is set on a reference line closest to the current block; alternatively, the template region of the current block is set on multiple reference lines closest to the current block, and the N extended reference lines participating in the combination are extended reference lines located outside the template region. In Figure 8A, the template region of the current block is set on reference line 30 with an index of 0. In the candidate list for constructing the TMRL mode, the N available extended reference lines are selected from a predefined list of extended reference lines with indices {1, 3, 5, 7, 12}. If there are more than 13 reference lines between the current block and the CTU boundary, five extended reference lines with indices {1, 3, 5, 7, 12} are selected for combination. If there are six or seven reference lines between the current block and the CTU boundary, three extended reference lines with indices {1, 3, 5} are selected for combination, and so on.

[0188] In this example, the template region of the current block is set on the reference row with an index of 0. The reference row with an index of 0 is called the reference row where the template region is located, and the reference rows with indexes 1 to 3 are called reference rows located outside the template region. If the template region of the current block is set on the reference rows with indexes 0 and 1, and the reference rows where the template region is located include extended reference rows, the reference rows with indexes 0 and 1 are the reference rows where the template region is located, and the reference rows with indexes 2 and 3 are the reference rows outside (upper and left sides) of the template region.

[0189] Figure 8A shows five extended reference rows involved in the combination: reference row 31 with index 1, reference row 33 with index 3, reference row 35 with index 5, reference row 37 with index 7, and reference row 39 with index 12. Unlike Figure 8A , in the example shown in Figure 8B , the template region 40 of the current block is set on two reference rows with indexes 0 and 1, while the five extended reference rows involved in the combination are reference row 42 with index 2, reference row 43 with index 3, reference row 45 with index 5, reference row 47 with index 7, and reference row 49 with index 12. This example selects N available extended reference rows from the predefined extended reference rows with indexes {2, 3, 5, 7, 12}. There are many other options for selecting template regions and extended reference rows, such as setting the template region of the current block on three reference rows with indexes 0, 1, and 2, or on four reference rows with indexes 0 through 3, and so on. When the template area is wider, the prediction is relatively more accurate.

[0190] In an example of this embodiment, the N extended reference rows of the current block are predefined N max Among the extended reference lines, the extended reference lines are located outside the template area of ​​the current block and do not exceed the CTU boundary; where N max The maximum number of extended reference lines that can be used in TMRL mode. This embodiment limits the N extended reference lines used for combination to those outside the template area of ​​the current block and not exceeding the CTU boundary. However, if the hardware supports it, extended reference lines that exceed the CTU boundary can also be selected for combination.

[0191] In an example of this embodiment, N max =5, the predefined 5 extended reference rows are reference rows with indexes {1, 3, 5, 7, 12} or {2, 3, 5, 7, 12}. In another example, the predefined N max The extended reference rows are the N closest to the current block, indexed starting from 1. max extended reference rows; or, the N nearest reference rows to the current block whose indexes start at 1 and are odd numbers. maxextended reference rows; or, the N closest to the current block with an even index starting from 2. max Selecting odd or even reference rows can simplify the calculation.

[0192] In an example of this embodiment, the M intra prediction modes are only allowed to be selected from the angular mode, or only allowed to be selected from the angular mode and the DC mode, or allowed to be selected from the angular mode, the DC mode, and the Planar mode.

[0193] In one example, the M intra prediction modes are selected in the following manner, where M ≥ 5:

[0194] Step 1: Determine the intra prediction modes used by the prediction blocks at the five neighboring positions around the current block, select the allowed intra prediction modes in turn and remove the duplicate modes;

[0195] The five adjacent positions are upper left, upper, upper right, left and lower left, as shown in FIG3 .

[0196] When the number of intra-frame prediction modes selected in the first step is equal to M, the process ends; when the number of intra-frame prediction modes selected in the first step is less than M and includes an angle mode, the process proceeds to the second step;

[0197] In the second step, starting from the first angle mode selected, the angle mode expansion operation is performed in sequence to obtain the extended angle mode, and the extended angle mode that is different from all the selected angle modes is selected until the total number of the selected intra-frame prediction modes is equal to M.

[0198] In this embodiment, if no angle mode is selected in the first step, or the total number of intra-frame prediction modes selected after the second step is still less than M, the third step is performed: determining the intra-frame prediction modes allowed to be selected from the predefined intra-frame prediction mode set, and sequentially selecting intra-frame prediction modes that are different from all the selected intra-frame prediction modes from the determined intra-frame prediction modes allowed to be selected, until the total number of selected intra-frame prediction modes is equal to M.

[0199] In one example, in the process of selecting M intra-frame prediction modes, only the angle mode is allowed to be selected. In another example, in the process of selecting M intra-frame prediction modes, only the angle mode and the DC mode are allowed to be selected; in another example, in the process of selecting M intra-frame prediction modes, the angle mode, the DC mode and the Planar mode are allowed to be selected. The Planar mode has limited effect in combination with the extended reference row and may not be used in the combination. The situation of the DC mode is similar. However, if the increased computational complexity can be accepted, the Planar mode and the DC mode may be added to the candidate list for combination.

[0200] In one example, the extended operation of the angle mode includes any one or more of the following operations: addition 1 and subtraction 1 operations on the angle mode; addition 2 and subtraction 2 operations on the angle mode; and addition 3 and subtraction 3 operations on the angle mode.

[0201] In an example of this embodiment, the M intra-frame prediction modes use some or all of the intra-frame prediction modes in the MPM except the Planar mode; or, use some or all of the intra-frame prediction modes in the MPM and the second MPM except the Planar mode; or, use some or all of the intra-frame prediction modes in the most likely mode MPM except the Planar mode and the DC mode; or, use some or all of the intra-frame prediction modes in the MPM and the second MPM except the Planar mode and the DC mode; or, use some or all of the intra-frame prediction modes in the most likely mode MPM except the Planar mode, the DC mode and the DIMD mode; or, the M intra-frame prediction modes use some or all of the intra-frame prediction modes in the MPM and the second MPM except the Planar mode, the DC mode and the DIMD mode.

[0202] In this example, all intra-frame prediction modes except the Planar mode in the MPM are used as predefined intra-frame prediction modes, or all intra-frame prediction modes except the Planar mode in the MPM and the second MPM are used as predefined intra-frame prediction modes. When the reference lines of the current block include all predefined extended reference lines, all predefined intra-frame prediction modes are used. When the reference lines of the current block include only some predefined extended reference lines, some predefined intra-frame prediction modes are used.

[0203] In an example of this embodiment, the M intra prediction modes are selected as follows:

[0204] Select M' intra prediction modes;

[0205] Predicting the templates of the current block according to the reference rows of the current block outside the template and the M' intra prediction modes, and calculating errors between reconstructed values ​​of the templates and predicted values, to obtain M' errors;

[0206] Select the M intra-frame prediction modes with the smallest corresponding errors from the M' intra-frame prediction modes as the M intra-frame prediction modes participating in the combination, M <M’。

[0207] In this example, the template of the current block is used to select M intra prediction modes from M' intra prediction modes, and the template area of ​​the current block is used to select K combinations from N×M combinations. The two can be different, but can also occupy the same area.

[0208] When selecting M' intra prediction modes in this example, various methods for selecting M intra prediction modes in the above embodiments can be used. For example, directly select from the list of MPM and the second MPM, or select through the first step using the method of the foregoing embodiment, or select through the first step and the second step, or select through the first step, the second step, and the third step, and so on.

[0209] In an example of this embodiment, N ≤ N max , 2 ≤ N max ≤ 12; 2 ≤ M ≤ 18; K ≤ K max , 6 ≤ K max ≤ 36; where, N max is the maximum number of extended reference lines allowed to be used in the TMRL mode, and K max is the maximum number of candidate combinations allowed to be used in the TMRL mode. Although the value ranges of the relevant parameters of N, M, and K are given here, this is only exemplary.

[0210] In an example of this embodiment, there are at least two sets of values for the N, M, and K. The first set of values is N1, M1, K1, and the second set of values is N2, M2, K2, where N1 ≤ N2, M1 ≤ M2, K1 ≤ K2, and N1 × M1 < N2 × M2; the first set of values is used when constructing the candidate list for the current block of the first size, and the second set of values is used when constructing the candidate list for the current block of the second size, and the first size is smaller than the second size. Here, the first size and the second size can represent various sizes respectively. For example, the first size can include 4×4, 4×8, 8×8, etc., and the second size can include 16×8, 16×16, 8×16, etc. In this example, different N, M, and K are used for current blocks of different sizes. When the size of the current block is small, smaller values are used to construct the candidate list of the TMRL mode. When the size of the current block is large, larger values are used to construct the candidate list of the TMRL mode. A better balance can be achieved between the complexity and performance of the operation.

[0211] In an example of this embodiment, the predicting the template region of the current block according to the N×M combinations respectively includes: when the current block is located at the left boundary of the picture, predicting the upper template region of the current block according to the N×M combinations respectively, and not predicting the left template region of the current block. This embodiment can simplify the operation and reduce the operation time required without affecting the performance.

[0212] In an example of this embodiment, the predicting the template region of the current block according to the N×M combinations respectively includes: for each of the N×M combinations, predicting in the following manner:

[0213] Calculating an initial prediction value of the template region according to a reconstruction value of the extended reference line in the combination and an intra-frame prediction mode in the combination, wherein the reconstruction value of the extended reference line is an original reconstruction value of the extended reference line or a filtered reconstruction value;

[0214] The initial prediction value of the template area is subjected to 4-tap filtering or 6-tap filtering, and the filtering result is used as the prediction value of the template area obtained according to the combined prediction.

[0215] When predicting the template area of ​​the current block, this embodiment may not filter the reconstructed value of the extended reference row, but use the original reconstructed value for calculation. It may also use a filter with shorter taps (such as a 4-tap filter) to reduce the calculation complexity and speed up the calculation speed.

[0216] In an example of this embodiment, predicting the template region of the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template region and the predicted value obtained by prediction, includes:

[0217] According to K combinations, the entire template area of ​​the current block is predicted respectively, and the corresponding K errors are formed into an error set, and the maximum error in the error set is recorded as D max ;

[0218] For each of the remaining combinations, first predict the single-side template area of ​​the current block according to the combination, and calculate the error D1 between the reconstructed value and the predicted value of the single-side template area of ​​the current block, such as D1≥D max , complete the prediction of the combination, such as D1 <D max , and then predict the template area on the other side of the current block based on the combination, and calculate the error D2 between the reconstructed value and the predicted value of the entire template area of ​​the current block, such as D2 <D max , add D2 to the error set, and D max Delete the maximum error D in the error set from the error set and update max , such as D2≥D max , then the prediction of the combination is completed;

[0219] After completing the prediction of the N×M combinations, the K combinations corresponding to the K errors in the error set are taken as the K combinations with the smallest corresponding errors.

[0220] In one example, the errors in the error set can be arranged in ascending order. When D2 is added to the error set, D2 is inserted at a position such that the errors in the error set are still arranged in ascending order. However, in other examples, the K errors in the error set can be sorted after the prediction of the N×M combinations is completed.

[0221] This example does not need to perform prediction and error calculation on the entire template area for all combinations. The combinations can be sorted, which can reduce computational complexity and speed up computational speed.

[0222] In one example of this embodiment, the step of filling the candidate list of the TMRL mode for the current block with the K combinations corresponding to the errors in ascending order of the errors includes: starting from the first position in the candidate list, filling the candidate list with the K combinations corresponding to the errors in ascending order of the errors. The candidate list of the TMRL mode in this example only contains combinations of extended reference lines and intra-frame prediction modes. The combination of a reference line indexed as 0 and an intra-frame prediction mode is indicated by other conventional modes, such as MPM.

[0223] In an example of this embodiment, starting from the i-th position of the candidate list, K combinations corresponding to the errors are filled into the candidate list in order from small to large; the i-th position of the candidate list is filled with a combination of a reference row with an index of 0 and one or more intra-frame prediction modes, i≥2. The candidate list of the TMRL mode in this example is not only filled with a combination of an extended reference row and an intra-frame prediction mode, but also with a combination of a reference row with an index of 0 and an intra-frame prediction mode. In this case, when the combination selected by the current block is a combination of a reference row with an index of 0 and an intra-frame prediction mode, it can also be represented by a TMRL mode index. At this time, the TMRL mode flag can still be used.

[0224] One embodiment provides a video decoding method related to the TMRL mode, which is applied to a decoder. As shown in FIG9 , the method includes:

[0225] Step 310 , decoding the multi-reference row intra prediction TMRL mode flag of the current block to determine whether the current block uses the TMRL mode;

[0226] Step 320: If it is determined that the current block uses the TMRL mode, continue decoding the TMRL mode index of the current block and construct a candidate list of the TMRL mode of the current block, wherein the candidate list is filled with combinations of extended reference rows and intra prediction modes of the current block candidates;

[0227] Step 330 , determining a combination of an extended reference row and an intra prediction mode selected for the current block based on the candidate list and the TMRL mode index, and predicting the current frame based on the selected combination;

[0228] The TMRL mode index is used to indicate the position of the selected combination of the extended reference row and the intra prediction mode in the candidate list.

[0229] This embodiment, after decoding the TMRL mode flag to determine that the current block uses the TMRL mode, enters the combination of the extended reference row and the intra-frame prediction mode into the TMRL mode candidate list. The decoded TMRL mode index and the candidate list are used to determine the selected combination for the current block and perform prediction. This means that the TMRL mode index can simultaneously indicate the selected extended reference row and intra-frame prediction mode for the current block, eliminating the need for two indexes. This reduces coding overhead.

[0230] In an example of this embodiment, before decoding the TMRL mode flag of the current block, the method further includes: when all conditions for allowing the current block to use the TMRL mode are met, decoding the TMRL mode flag of the current block again, wherein the conditions for allowing the use of the TMRL mode include any one or more of the following:

[0231] The current block is a block in the luma frame;

[0232] The current block allows the use of MRL;

[0233] The current block is not located at the upper boundary of the coding tree unit CTU;

[0234] The current block does not use template-based intra prediction mode to derive TIMD.

[0235] In this example, if one of the above conditions is met, TMRL mode is not permitted, and decoding of the TMRL mode flag and TMRL mode index can be skipped. However, this is not necessarily the case in other embodiments. For example, when TMRL mode is encoded and decoded before TIMD, the use of TIMD by the current block cannot be used as a condition for prohibiting the use of TMRL mode. For another example, if future hardware supports obtaining reference rows outside the CTU boundary, the current block being located at the upper boundary of the CTU will no longer be a condition for prohibiting the use of TMRL mode.

[0236] In an example of this embodiment, the method further includes: decoding a multiple reference row index for the current block, where it is determined through decoding that the current block allows the use of MRL, the current block is not located at the upper boundary of a CTU, and the current block uses TIMD, where the multiple reference row index indicates the position of a reference row selected for the current block in a multiple reference row index list. In this example, when the current block uses TIMD, the TMRL mode is not allowed, but the use of MRL is still allowed. Thus, the selected reference row for the current block can still be determined by decoding the multiple reference row index for the current block, and then, combined with the TIMD mode selected for the current block, prediction can be performed on the current block.

[0237] In an example of this embodiment, when it is determined that the current block uses the TMRL mode according to the TMRL mode flag, the method further includes: skipping decoding of syntax elements of any one or more of the following modes: MPM mode, sub-partition mode, multi-transform selection (MTS) mode, low-frequency non-divisible transform (LFNST) mode, and TIMD mode. Correspondingly, on the encoder side, if one or more of the above modes are skipped when the current block uses the TMRL mode flag during encoding, then on the decoder side, if it is determined through decoding that the current block uses the TMRL mode flag, decoding of these modes is also skipped.

[0238] One embodiment further provides a video decoding method, primarily involving the decoding process of intra-frame prediction, and also provides a corresponding description of the encoder. In this embodiment, the encoder constructs a candidate list of TMRL modes. When a combination in the candidate list is selected by the current block through mode selection, the syntax elements of the TMRL mode are encoded and decoded by combining the extended reference line and the intra-frame prediction mode.

[0239] This embodiment constructs a template based on N predefined extended reference lines and M intra-frame prediction modes, using reference line 0 (indexed as reference line 0). See template region 30 in FIG8A . (x, -1) and (-1, y) are the coordinates relative to the top-left corner (0, 0) of the current block. Five predefined extended reference lines are also shown, indexed as {1, 3, 5, 7, 12}.

[0240] When constructing the candidate list of the TMRL mode, the SAD between the predicted value and the reconstructed value of N×M combinations on the template is calculated, and the corresponding combinations are sorted in ascending order of SAD. The K combinations with the smallest SAD among the NxM combinations are filled into the candidate list of the TMRL mode in ascending order of SAD, N×M ≥ 2.

[0241] Constructing a candidate list for the TMRL mode is an operation that both the encoder and decoder must perform. When the encoding conditions are met, such as the combination selected by the current block being in the candidate list, the encoder encodes a TMRL mode flag to indicate the use of the TMRL mode. The encoder also determines the TMRL mode index based on the position of the selected combination in the candidate list. For example, if the selected combination is in the first position, the TMRL mode index is 0; if it is in the second position, the TMRL mode index is 1, and so on. The TMRL mode index can be encoded using, but is not limited to, Golomb-Rice coding.

[0242] The following description is made using an example where N = 5, M = 6 and K = 12. The indices of the predefined five extended reference rows are {1, 3, 5, 7, 12}, and the six intra prediction modes are selected step by step.

[0243] The video decoding method of this embodiment includes:

[0244] Step 1: Decode the syntax elements related to the TMRL mode;

[0245] The decoder parses syntax elements related to intra-frame prediction modes, including those related to TIMD, MRL, and other modes. The TMRL mode proposed in this embodiment can be considered an evolution of the MRL mode, and the syntax elements of the TMRL mode can also be considered part of the syntax elements of the MRL mode. Of course, the two modes can also be considered as two different modes.

[0246] When the current block uses the TIMD mode, the decoding method of the MRL mode syntax elements remains unchanged. When the current block does not use the TIMD mode, the syntax elements of the TMRL mode need to be decoded. The syntax related to the current block decoding is shown in the following table:

[0247]

[0248]

[0249] The "cu_tmrl_flag" in the table is the TMRL mode flag. When it is equal to 1, it means that the current block uses the TMRL mode, which also defines that the intra-frame prediction type of the current luminance sample is a template-based multi-reference row frame prediction mode; when "cu_tmrl_flag" is equal to 0, it means that the current block does not use the TMRL mode, which also defines that the intra-frame prediction mode type of the current luminance sample is not a template-based multi-reference row frame prediction mode.

[0250] The "tmrl_idx" field in the table represents the TMRL mode index, which indicates the position of the selected extended reference row and intra prediction mode combination within the TMRL mode candidate list. This field also defines the index of the selected combination within the TMRL mode-sorted candidate list (indicating the position of the combination). The "tmrl_idx" field can be encoded and decoded using the Columbine method, which is not detailed here.

[0251] As can be seen from the table above, before decoding cu_tmrl_flag, the following conditions are first determined: the current block allows the use of MRL (i.e., sps_mrl_enabled_flag is 1), the current block is not located at the upper boundary of the CTU (i.e., (y0%CtbSizeY)>0), and the current block does not use TIMD. If these conditions are met, cu_tmrl_flag is decoded. If the other two conditions are met and the current block uses TIMD, the multi-reference row index intra_luma_ref_idx of the current block is decoded.

[0252] The ISP mode flag (intra_subpartitions_mode_flag) in the table is decoded after the TMRL mode-related syntax elements. If the current block does not use the TMRL mode (!cu_tmrl_flag is set), then the intra_subpartitions_mode_flag is decoded. Similarly, if the current block does not use the TMRL mode (!cu_tmrl_flag is set), then the MPM-related syntax elements are decoded.

[0253] Step 2: construct a candidate list of TMRL modes, and determine the extended reference row and intra prediction mode selected for the current block according to the TMRL mode index and the candidate list;

[0254] After the parsing stage, before predicting the current block, if the current block uses the TMRL mode, it is necessary to construct a candidate list of the TMRL mode, and determine the extended reference row and intra-frame prediction mode selected by the current block according to the TMRL mode index and the candidate list.

[0255] To construct a candidate list for the TMRL mode, it is necessary to first determine the candidate extended reference rows and the candidate intra prediction modes.

[0256] ■Determine candidate extension reference lines

[0257] Candidate extended reference lines are selected from predefined extended reference lines. The position of the current block in the image determines which of the predefined extended reference lines can be used. In principle, the upper reference line that can be used by the current block should not exceed the upper CTU boundary. In one example, all extended reference lines with indexes {1, 3, 5, 7, 12} that do not exceed the CTU boundary are added to the candidate extended reference lines. To achieve better encoding and decoding performance or reduce complexity, more or fewer extended reference lines may be used.

[0258] ■Determine candidate intra prediction modes

[0259] In this embodiment, the TMRL mode is not bound to the MPM (it may be bound in other embodiments), but rather a candidate list of intra prediction modes is constructed, and the intra prediction mode used for the combination is selected from this candidate list. The candidate list is derived as follows:

[0260] First, among the 67 traditional prediction modes, the Planar mode and the DC mode are removed, or only the Planar mode is removed while the DC mode is retained. The removed modes are not added to the candidate list, that is, they are not used as intra-frame prediction modes participating in the combination in the TMRL mode.

[0261] In this embodiment, the candidate prediction mode list to be constructed is 6 in length. First, the candidate prediction mode list is populated by sequentially selecting non-overlapping intra-frame prediction modes from the intra-frame prediction modes used by the prediction blocks at the five neighboring locations surrounding the current block. Then, the angular mode expansion operation is performed on the modes already populated in the list. Specifically, the angular modes are added or subtracted by one, and non-overlapping expanded angular modes are selected and sequentially populated into the candidate prediction mode list. When the number of modes in the candidate list reaches 6, the expansion is stopped.

[0262] The specific operation process of adding one and subtracting one is shown in the following table:

[0263]

[0264]

[0265] The above "angle mode - 1" refers to the angle mode obtained by subtracting 1 from the index of the filled angle mode. For example, if the filled angle mode is mode 3, angle mode - 1 is angle mode 2. The above "angle mode + 1" refers to the angle mode obtained by adding 1 to the index of the filled angle mode. For example, if the filled angle mode is mode 3, angle mode + 1 is angle mode 4.

[0266] If angle pattern -1 is smaller than angle pattern 2, for example, angle pattern -1 results in angle pattern 1 (angle patterns are indexed from 2, and angle pattern 1 does not exist), then the angle pattern in the direction opposite to -1 is selected. Assuming there are 65 angle patterns, the angle pattern in the opposite direction is angle pattern 66. If the angle pattern after angle pattern +1 is larger than angle pattern 66, then the angle pattern in the direction opposite to +1 is selected. For example, if the filled angle pattern is angle pattern 66, then angle pattern +1 does not exist, and the angle pattern in the direction opposite to +1 is selected, which is angle pattern 2.

[0267] When the angle mode is expanded as described above, the angle mode is added by 1 or subtracted by 1. In other embodiments, the addition and subtraction operations can be expanded from addition and subtraction to addition and subtraction of X or subtraction of X. Assuming that X=3, the angle mode can be added by 1 or subtraction of 1, added by 2 or subtraction of 2, and added by 3 or subtraction of 3 until the candidate prediction mode list is filled.

[0268] If the candidate prediction pattern list is not full after the angle pattern expansion operation is performed on the patterns that have been filled in the list, the candidate prediction pattern list is filled with non-repeated patterns in the predefined pattern set until the candidate prediction pattern list is full. The pattern set includes some angle patterns selected according to statistical laws, as follows:

[0269] mpm_default[]={DC_IDX,VER_IDX,HOR_IDX,VER_IDX-4,VER_IDX+4,14,22,42,58,10,26,38,62,6,30,34,66,2,48,52,16};

[0270] Among them, DC_IDX indicates DC mode, VER_IDX indicates vertical mode, HOR_IDX indicates horizontal mode, and the remaining numbers indicate the angle mode corresponding to the number.

[0271] In this embodiment, the length of the candidate prediction mode list is 6. For performance reasons, the length can be set to a value greater than 6 to try more angle modes, or it can be set to a value less than 6 to try fewer modes to reduce complexity.

[0272] When determining candidate intra-frame prediction modes, this embodiment excludes the Planar mode and the DC mode, or only the Planar mode. However, if complexity is not considered, these two modes may not be excluded, that is, the Planar mode, the DC mode, and all angle modes may all be used as candidate intra-frame prediction modes and participate in the combination with the extended reference row.

[0273] ■Build a candidate list of TMRL models

[0274] After determining the candidate extended reference lines and intra-frame prediction modes, all combinations in the extended reference line list and candidate prediction mode list can be tried one by one. These combinations are predicted separately on the template area (Template as shown below) where reference line 0 is located, see Figure 8A. The error between the reconstructed value of the template area and the predicted value obtained by each combination is calculated. The K combinations with the smallest errors are filled into the candidate list of the TMRL mode in ascending order of error.

[0275] During the prediction process, this embodiment only limits the use of the TMRL mode when the current block occupies the first row of the CTU. When the current block is located at the left edge of the image, the TMRL mode can still be used. In this case, since the reference line 0 on the left is already outside the image boundary, the left template is not used during prediction, that is, only the upper template area is predicted.

[0276] The prediction process on the template area can be completely consistent with other normal intra-frame angle prediction processes, that is, the reconstructed value of the reference row is first filtered and then used as the initial prediction value of the template area. After the template area is predicted based on the reconstructed value after filtering the reference row and the intra-frame prediction mode in the combination, the initial prediction result is subjected to 4 or 6 tap filtering and then used as the predicted value. Considering the complexity of the operation, the filtering step of the reconstructed value of the reference row can be omitted, or a filter with shorter taps can be used. In this embodiment, when predicting the prediction value of the template area, the reconstructed value of the reference row pixel is not filtered, and the initial prediction result is a 4-tap interpolation filter with 1 / 32 accuracy at a non-integer angle.

[0277] Finally, the template region is predicted based on the angle, reference line, and filter in the current combination. The SAD between the predicted value and the reconstructed value of the template region is calculated, and the predicted values ​​are sorted in ascending order. The 12 combinations with the smallest SAD are selected and added to the candidate list of the TMRL model.

[0278] A fast algorithm can be used in the sorting process. In the above process, it is necessary to try 5 reference rows and 6 prediction modes, totaling 30 combinations, but only the 12 combinations with the smallest SAD need to be selected. After predicting the first 12 combinations and obtaining the corresponding SAD, this embodiment only needs to maintain and update the 12 combinations with the smallest error (also called cost) starting from the 13th combination. Starting from the 13th combination, only the upper template area is predicted and the corresponding SAD is calculated. When the SAD calculated based on the upper template is already greater than the combination with the largest error among the 12 combinations with the smallest error, the prediction and error calculation for the left template area can be skipped. For details, please refer to the above embodiment.

[0279] Step three: determine the combination of the extended reference row and intra prediction mode selected for the current block according to the constructed TMRL mode candidate list and the decoded TMRL mode index, and perform intra prediction on the current block according to the selected combination.

[0280] When the TMRL mode is used, the reference row index refIdx and the variable predModeIntra define the mode used for intra prediction, both of which are determined according to the TMRL mode index "tmrl_idx" and the candidate list of the TMRL mode.

[0281] Using the method described in this embodiment on the ECM-4.0 reference software, with N=5 (five extended reference rows, 1, 3, 5, 7, and 12 respectively), M=6 (six predefined prediction modes, excluding the Planar and DC modes), and K=12 (selecting only the top 12 ranked combinations with the smallest SAD among all combinations), the following results were measured under the AI ​​configuration:

[0282]

[0283] With N=5 (5 extended reference rows are 1, 3, 5, 7, 12), M=8 (8 predefined prediction modes) and K=16, the results are as follows under AI configuration:

[0284]

[0285] With N=5 (5 extended reference lines are 1, 3, 5, 7, 12), M=12 (12 predefined prediction modes), and K=24, the results measured under AI configuration are as follows:

[0286]

[0287] With N=5 (5 extended reference lines are 1, 3, 5, 7, and 12), M=8 (8 predefined prediction modes, excluding Planar and including DC modes), and K=16, the results are as follows under the AI ​​configuration:

[0288]

[0289] The meanings of the parameters in the table are as follows:

[0290] EncT: Encoding Time, 10X% means that after the reference row sorting technology is integrated, the encoding time is 10X% compared with before integration, which means that the encoding time increases by X%.

[0291] DecT: Decoding Time, 10X% means that after the reference row sorting technology is integrated, the decoding time is 10X% compared with before integration, which means that the decoding time increases by X%.

[0292] ClassA1 and ClassA2 are test video sequences with a resolution of 3840x2160, ClassB is a test sequence with a resolution of 1920x1080, ClassC is 832x480, ClassD is 416x240, and ClassE is 1280x720; ClassF is a screen content sequence with several different resolutions.

[0293] Y, U, V are the three color components. The columns where Y, U, V are located represent the BD-rate of the test results on Y, U, V. Indicator, the smaller the value, the better the encoding performance.

[0294] All intra indicates the test configuration of all intra-frame configurations.

[0295] It can be seen that using the TMRL mode of this embodiment for intra-frame prediction coding and decoding can achieve a significant improvement in coding performance.

[0296] In this embodiment, both the extended reference line and the prediction mode use a template area of ​​one row and one column, sorted and filtered in ascending SAD order. For the extended reference line, if all extended reference lines (including reference line 1) are sorted, only the template area of ​​one row and one column can be used. However, for prediction mode screening, to select a more appropriate prediction mode, using more reference rows, as in the TIMD mode, can yield more accurate results. In other embodiments, the method by which the TMRL mode determines the candidate prediction mode list can also be varied.

[0297] For example, when it is necessary to construct a candidate prediction mode list of length 6, a list of length greater than 6 can be first constructed using the same construction and filling method as in this embodiment. Then, the 4 rows and 4 columns closest to the current block are used as a template. The template is predicted using the 5th reference row and the intra-frame prediction mode in the candidate prediction mode list. The error (SAD or SATD) between the predicted value and the reconstructed value of the template is calculated. The six intra-frame prediction modes with the smallest errors are sorted in ascending order of error, and the six intra-frame prediction modes are selected as the intra-frame prediction modes in the TMRL mode candidate list of length 6 to be constructed. In addition, the length of the TMRL mode candidate list of 6 is only an example, and the numerical value can be adjusted according to the situation.

[0298] The angle modes in this embodiment take 65 as an example, but in other embodiments, the angles can be expanded to 129 or more to obtain better performance. When expanded to more angles, the number of filters for intra-frame prediction should also be increased accordingly. For example, when there are 129 angles, a filter with 1 / 64 precision is used.

[0299] One embodiment also provides an intra prediction fusion (IPF) technology. IPF allows the angle mode to use the prediction results of two adjacent reference lines for weighting to obtain the final prediction result of the current block. The weighting method is shown in the following formula:

[0300] p fusion =w a ×p a +w b ×p b

[0301] Among them, p a is the result of predicting the current block using the reference line with index a and the angle mode, pb is the result of predicting the current block using the reference line with index a+1 and the angle mode, p fusion is the fusion prediction result; w a When p is weighted by l a The weight, w b When p is weighted b The weight, w a is 3 / 4,w b is 1 / 4.

[0302] When the current block uses IPF, the above fused prediction result is used as the final prediction result of the current block.

[0303] In this embodiment, no identifier is set to indicate whether IPF is used (i.e., whether it is turned on). Instead, it is turned on by default for the angle mode selected for the current block when the following conditions are met:

[0304] The angle mode of the current block selection is not an integer slope angle mode;

[0305] The width and height of the current block is greater than 16

[0306] The intra-block partitioning ISP mode is not selected for the current block.

[0307] In other words, IPF is not allowed if at least one of the following restrictions holds:

[0308] The angle mode selected by the current block is the integer slope angle mode;

[0309] The width and height of the current block is less than or equal to 16;

[0310] The current block selects ISP mode.

[0311] When the angle value (intraPredAngle) of the angle mode is divided by 32 with a remainder of 0, the angle mode is an integer slope angle mode. The correspondence between intraPredAngle and the angle mode can be seen in Table 1 above.

[0312] When the current mode meets the IPF condition, the intra prediction mode selected for the current block may be the DIMD fusion mode (fusion of planar mode and two angular modes) selected by DIMD, or the TIMD fusion mode selected by TIMD. From a hardware implementation perspective, the less fusion, the better. When the DIMD fusion mode is selected, three intra prediction results are already fused on the current block. If IPF is used, too many fusions will increase the complexity of the prediction stage.

[0313] An embodiment of the present disclosure provides an intra-frame prediction fusion method, which can be applied to an encoder or a decoder. As shown in FIG10 , the method includes:

[0314] Step 410: When the intra prediction mode selected for the current block includes the angle mode, determining whether a restriction condition for using the intra prediction fusion IPF for the current block is met;

[0315] Step 430: When at least one of the restriction conditions is satisfied, restrict the use of the IPF when performing intra-frame prediction on the current block.

[0316] In this step, the use of IPF is restricted when performing intra-frame prediction on the current block. This may be to prohibit the use of IPF, or to allow IPF fusion for only some of the angle modes when the selected intra-frame prediction mode includes multiple angle modes that meet the IPF usage conditions. In one example, the angle mode that meets the IPF usage conditions is an angle mode that is not an integer slope, or an angle mode other than -45°, 0°, 45°, 90°, and 135°.

[0317] In this paper, disallowing IPF when a certain constraint condition holds is a sufficient condition for not using IPF when predicting the current block. Allowing IPF when a certain constraint condition does not hold is a necessary condition for using IPF when predicting the current block. If other constraints hold, IPF may still not be allowed.

[0318] In an exemplary embodiment of the present disclosure, the restriction condition includes the following restriction condition on the number of modes: using IPF will result in the prediction of the current block requiring the fusion of N or more intra-frame prediction modes, where N is an integer greater than or equal to 3.

[0319] In an example of this embodiment, when N=3, when the intra-frame prediction mode selected for the current block includes the angle mode, determining whether a restriction condition for using the IPF for the current block is met; when at least one of the restriction conditions is met, restricting the use of the IPF when performing intra-frame prediction on the current block includes:

[0320] When the TIMD fusion mode is selected for the current block and the TIMD fusion combines two angle modes that meet the IPF usage condition, it is determined that the mode quantity restriction condition is met, and only one of the two angle modes is allowed to be IPF-fused when performing intra-frame prediction on the current block;

[0321] When the TIMD fusion mode is selected for the current block and only one of the two intra-frame prediction modes of TIMD fusion is an angle mode that meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode when intra-frame prediction is performed on the current block.

[0322] In an exemplary embodiment of the present disclosure, the restriction condition includes the following restriction condition on the number of modes: using IPF will result in the prediction of the current block requiring the fusion of M or more angle modes, where M is an integer greater than or equal to 2.

[0323] In an example of this embodiment, when M=3, when the intra-frame prediction mode selected for the current block includes the angle mode, determining whether a restriction condition for using the IPF for the current block is met; when at least one of the restriction conditions is met, restricting the use of the IPF when performing intra-frame prediction on the current block includes:

[0324] When the TIMD fusion mode is selected for the current block and the TIMD fusion combines two angle modes that meet the IPF usage condition, it is determined that the mode quantity restriction condition is met, and only one of the two angle modes is allowed to be IPF-fused when performing intra-frame prediction on the current block. For example, only the angle mode with the smallest cost or the second smallest cost among the two angle modes is allowed to be IPF-fused;

[0325] When the TIMD fusion mode is selected for the current block and only one of the two intra-frame prediction modes of TIMD fusion is an angle mode that meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode when intra-frame prediction is performed on the current block.

[0326] In an example of this embodiment, when M=3, when the intra-frame prediction mode selected for the current block includes the angle mode, determining whether a restriction condition for using the IPF for the current block is met; when at least one of the restriction conditions is met, restricting the use of the IPF when performing intra-frame prediction on the current block includes:

[0327] When the DIMD fusion mode is selected for the current block and both angle modes of DIMD fusion meet the IPF usage conditions, it is determined that the mode quantity restriction condition is met, and only one of the two angle modes is allowed to be IPF fused when intra-frame prediction is performed on the current block. For example, only the angle mode with the highest amplitude value or the second highest amplitude value among the two angle modes is allowed to be IPF fused.

[0328] When the current block uses the DIMD fusion mode and only one of the two angle modes of DIMD fusion meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode that meets the IPF usage conditions when performing intra-frame prediction on the current block.

[0329] In an example of this embodiment, when M=2, when the intra-frame prediction mode selected for the current block includes the angle mode, determining whether a restriction condition for using the IPF for the current block is met; when at least one of the restriction conditions is met, restricting the use of the IPF when performing intra-frame prediction on the current block includes:

[0330] When the TIMD fusion mode is selected for the current block and the TIMD fuses two angle modes, it is determined that the mode quantity restriction condition holds true, and IPF is not allowed to be used when performing intra prediction on the current block;

[0331] When the TIMD fusion mode is selected for the current block, only one of the two intra-frame prediction modes of TIMD fusion is an angle mode, and the angle mode meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode that meets the IPF usage conditions when intra-frame prediction is performed on the current block.

[0332] In the above embodiment, the intra-frame prediction of the current block allows IPF fusion for an angle mode, including: taking a weighted sum of two or more prediction results as the final prediction result of the current block; the two or more prediction results include a prediction result obtained by predicting the current block based on a first reference row selected by the current block and the angle mode, and a prediction result obtained by predicting the current block based on a second reference row different from the first reference row and the angle mode. For example, the second reference row can be a reference row adjacent to the first reference row or a reference row with an index of 0.

[0333] In an exemplary embodiment of the present disclosure, the restriction condition includes any one or more of the following restriction conditions:

[0334] Constraint 1: The current block selects TIMD fusion mode;

[0335] Constraint 2: The current block selects DIMD fusion mode;

[0336] Constraint 3: The current block uses multiple reference rows (MRL).

[0337] Constraint 4: The index of the reference row selected by the current block is greater than or equal to K, where K is an integer greater than or equal to 3;

[0338] Constraint 5: The width of the current block is less than or equal to the set value;

[0339] Constraint 5: The height of the current block is less than or equal to the set value;

[0340] Constraint seven: The width multiplied by the height of the current block is less than or equal to the set value;

[0341] Constraint 8: The current block uses the intra-frame sub-block division ISP mode;

[0342] Constraint nine: The angle of the angle mode selected by the current block is any one of -45°, 0°, 45°, 90°, and 135°, or the angle mode selected by the current block is an angle mode with an integer slope;

[0343] Constraint 10: The current frame to which the current block belongs is an inter-frame;

[0344] Constraint 11: The current frame to which the current block belongs is a chroma frame, that is, IPF is only used for luma frames, and IPF is not used for chroma frames;

[0345] When at least one of the constraints one to eleven is satisfied, IPF is not allowed to be used when performing intra-frame prediction on the current block.

[0346] In an example of this embodiment, the restriction condition includes restriction condition four, wherein K=3 or 5 or 7 or 12.

[0347] The above-mentioned embodiments of the present disclosure limit the use of IPF. For example, this embodiment can limit the number of fused intra-frame prediction modes, limit the number of fused angle modes, limit the simultaneous use of IPF with prediction modes such as TIMD and DIMD that may fuse multiple modes, etc., so as to avoid excessive fusion during prediction, which leads to an inappropriate increase in the complexity of the prediction stage.

[0348] This embodiment can limit the size of the current block and only use IPF when the current block size is larger than a set value. This is because when the current block is smaller than a certain size, it usually has more texture, and the performance improvement of using fusion prediction is limited. This embodiment does not allow the use of IPF when the index of the reference row selected by the current block is greater than or equal to K. In other words, IPF is not allowed when the extended reference row selected by the current block is far from the current block. In this case, the performance improvement of using IPF is limited.

[0349] This embodiment reduces computational complexity by limiting the simultaneous use of IPF and MRL. This embodiment does not allow the use of IPF when the current frame is an inter-frame (such as a B frame or a P frame), thereby reducing the cost of inter-frame encoding and decoding.

[0350] In an exemplary embodiment of the present disclosure, the method further includes: when none of the restriction conditions are met, using the IPF when performing intra-frame prediction on the current block; the using the IPF when performing intra-frame prediction on the current block includes:

[0351] When there is only one intra-frame prediction mode selected for the current block and it is an angle mode, the weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block; wherein, the first prediction result is the result of predicting the current block based on the first reference row selected by the current block and the first angle mode selected by the current block, the second prediction result is the result of predicting the current block based on the second reference row and the first angle mode, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

[0352] It should be noted that when none of the aforementioned restrictions are met, IPF is used for intra-frame prediction of the current block. This does not mean that IPF is only allowed for intra-frame prediction of the current block when none of the restrictions are met. For example, when the aforementioned mode number restriction is met, IPF fusion can still be performed on some angle modes that meet the IPF usage conditions.

[0353] In an example of this embodiment, the weighted sum of the first prediction result and the second prediction result is calculated according to the following formula:

[0354] p fusion =(w a p a +w b p b )>>shift formula one

[0355] or

[0356] p fusion =(w a p a +w b p b +offset)>>shift formula 2

[0357] Among them, p a is the first prediction result, p b For the second prediction result, p fusion The final prediction result of the current block, w a For p a The weight, w b For p b The weight of

[0358] Among them, w a +w b =(1< <shift),offset=1<<(shift-1),shift> =1, shift and offset are set parameters.

[0359] The above algorithm can avoid the occurrence of decimals during the calculation process and improve the efficiency of the algorithm.

[0360] In an exemplary embodiment of the present disclosure, the method further includes: when none of the restriction conditions are met, using the IPF when performing intra-frame prediction on the current block; the using the IPF when performing intra-frame prediction on the current block includes:

[0361] When the intra-frame prediction mode selected for the current block is a fusion mode, a weighted sum of more than three prediction results is used as the final prediction result of the current block; wherein the more than three prediction results include: multiple prediction results obtained by predicting the current block according to the first reference row selected for the current block and each intra-frame prediction mode in the fusion mode, and one or more prediction results obtained by predicting the current block according to the second reference row and one or more angle modes in the fusion mode that allow IPF fusion; the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

[0362] In the above embodiment of the present disclosure, when calculating the weighted sum of the first prediction result and the second prediction result, the weight assigned to the first prediction result is greater than the weight assigned to the second prediction result. For example, when the first reference behavior index is a reference row of 1 and the second reference behavior index is a reference row of 0, the first prediction result is assigned a weight of 3 / 4 and the second prediction result is assigned a weight of 1 / 4.

[0363] In the above embodiment of the present disclosure, when the index of the first reference row is greater than or equal to K, the second reference row is adjacent to the first reference row and is closer to the current block than the first reference row, where K is an integer greater than or equal to 1.

[0364] For example, the second reference row may be determined according to at least one of the following methods:

[0365] When the index of the first reference row is 1, determining the index of the second reference row to be 0;

[0366] When the index of the first reference row is 3, determining the index of the second reference row to be 2;

[0367] When the index of the first reference row is 5, determining that the index of the second reference row is 4;

[0368] When the index of the first reference row is 7, determining that the index of the second reference row is 6;

[0369] When the index of the first reference row is 12, the index of the second reference row is determined to be 11.

[0370] When the index of the first reference row is greater than or equal to K, setting the second reference row to a reference row that is adjacent to the first reference row and closer to the current block than the first reference row can improve the accuracy of the second reference row's prediction of the current block, thereby improving the accuracy of the fused prediction result.

[0371] In an exemplary embodiment of the present disclosure, the IPF can be used simultaneously with the TMRL mode. That is, when the current block selects an extended reference row and an angle mode in the candidate list of the TMRL mode, and none of the restriction conditions are met, a first prediction result of predicting the current block based on the extended reference row and the angle mode, and a second prediction result of predicting the current block based on another reference row and the angle mode are calculated, and a weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block, where the other reference row is a reference row with an index of 0 or an adjacent row of the extended reference row.

[0372] An embodiment of the present disclosure further provides a video encoding method, which is applied to an encoder, as shown in FIG11 , and includes:

[0373] Step 510, determining the reference row and intra prediction mode selected for the current block through mode selection;

[0374] Step 520: predict the current block according to the intra-frame prediction fusion method described in any embodiment of the present disclosure to obtain a predicted value of the current block;

[0375] In this step, when IPF is used when performing intra-frame prediction on the current block, the prediction value of the current block is obtained according to the final prediction result of the current block.

[0376] Step 530: Determine the residual of the current block according to the original value and the predicted value of the current block.

[0377] The video encoding method of this embodiment uses the intra-frame prediction fusion method of any embodiment of the present disclosure to predict the current block, and can achieve various effects of the intra-frame prediction fusion method.

[0378] An embodiment of the present disclosure further provides a video decoding method, which is applied to a decoder, as shown in FIG12 , and includes:

[0379] Step 610: Decode the code stream and determine the reference row and intra prediction mode selected for the current block;

[0380] Step 620: predict the current block according to the intra-frame prediction fusion method described in any embodiment of the present disclosure to obtain a predicted value of the current block;

[0381] In this step, when IPF is used when performing intra-frame prediction on the current block, the prediction value of the current block is obtained according to the final prediction result of the current block.

[0382] Step 630: Determine the reconstructed value of the current block according to the predicted value of the current block.

[0383] The video encoding method of this embodiment uses the intra-frame prediction fusion method of any embodiment of the present disclosure to predict the current block, and can achieve various effects of the intra-frame prediction fusion method.

[0384] The intra-frame prediction method using the template-based multiple reference line intra-frame prediction (TMRL) mode used in the above embodiments constructs a candidate list based on a combination of extended reference lines and intra-frame prediction modes, and then encodes and decodes the combination of extended reference lines and intra-frame prediction modes. However, regardless of whether it is template-based or sorted, prediction is performed using a single reference line, which often contains noise, affecting prediction accuracy.

[0385] Therefore, one embodiment of the present disclosure proposes a method for using IPF based on the TMRL model. Using IPF based on the TMRL model involves performing IPF fusion on a selected angle mode. Specifically, the prediction result for the current block based on the selected reference row and the angle mode is weighted with the prediction result for the current block based on another reference row and the angle mode to obtain the final prediction result for the current block. By fusing the prediction results from different reference rows, prediction accuracy is improved.

[0386] When using IPF based on the TMRL mode, the flag bit does not need to be set to indicate whether the current block uses IPF. Instead, the encoder and decoder make the judgment based on the agreed conditions.

[0387] In one embodiment of the present disclosure, when IPF can be used simultaneously with the TMRL mode, when the current block selects an extended reference row and an angle mode in the candidate list of the TMRL mode, and none of the restriction conditions are met, a first prediction result of predicting the current block based on the extended reference row and the angle mode, and a second prediction result of predicting the current block based on another reference row and the angle mode are calculated, and a weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block, where the other reference row is a reference row with an index of 0 or an adjacent row of the extended reference row.

[0388] The TMRL mode of this embodiment can use the method of the aforementioned embodiment when constructing the candidate list and encoding and decoding. The difference lies in the stage of generating the prediction value of the current block, which is to weight the prediction value generated by the selected angle mode and the selected reference row and the prediction value generated by the selected angle mode and another reference row to obtain the final prediction value of the current block.

[0389] This embodiment can limit the IPF fusion in TMRL mode, for example, IPF fusion is performed only when the selected reference behavior is close to certain reference lines of the current block. Specifically, fusion is allowed only when the reference behavior is reference line 1, or reference behavior reference line 1,3, or reference behavior reference line 1,3,5, or reference behavior reference line 1,3,5,7, or reference behavior reference line 1,3,5,7,12. In addition, it can also be limited that IPF is not used when the current block uses certain angle modes, for example, IPF is not allowed when the angle mode is horizontal angle, vertical angle, -45 degree angle, 45 degree angle, 135 degree angle. In addition, when the size of the current block is less than or equal to a certain size, IPF, i.e., fusion prediction, may also not be allowed because small blocks usually have more textures.

[0390] IPF is used on the basis of the TMRL mode, and a flag may be added to the syntax element of the TMRL mode to indicate whether IPF is used.

[0391] An embodiment of the present disclosure provides a video decoding method, which is applied to a decoder. As shown in FIG13 , the method includes:

[0392] Step 710 , when it is determined by decoding that the current block uses the template-based multiple reference line intra prediction TMRL mode, continue decoding the TMRL mode index and TMRL fusion flag of the current block;

[0393] Step 720: construct a candidate list of TMRL modes for the current block, and determine the extended reference row and intra prediction mode selected for the current block according to the candidate list and the TMRL mode index;

[0394] This step may construct the candidate list and determine the extended reference row and intra prediction mode selected for the current block in the same manner as in the above embodiment.

[0395] Step 730: When the TMRL fusion flag indicates using intra-frame prediction fusion IPF, a weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block;

[0396] In this step, the weighted sum of the first prediction result and the second prediction result can be calculated according to the aforementioned formulas 1 and 2.

[0397] The first prediction result is obtained by predicting the current block based on the extended reference row and the intra-frame prediction mode, and the second prediction result is obtained by predicting the current block based on another reference row and the intra-frame prediction mode, where the intra-frame prediction mode is an angular mode. The weighting of the two prediction results can be calculated using the formula of the aforementioned embodiment.

[0398] In this embodiment, when the TMRL fusion flag indicates that IPF is not used, the current block is predicted according to the extended reference row and the intra prediction mode to obtain a final prediction result of the current block.

[0399] Based on the TMRL mode, this embodiment sets a TMRL fusion flag to indicate whether IPF is used. At the encoding end, the TMRL fusion flag can be encoded according to whether the same or similar restrictions as in the previous embodiment are met. For example, when it is determined that IPF is used, the TMRL fusion flag is set to 1, and when it is determined that IPF is not used, the TMRL fusion flag is set to 0. At the decoding end, there is no need to combine these restrictions for judgment. Based on the TMRL fusion flag, it can be determined whether to perform IPF fusion on the angle mode selected when using the TMRL mode. This can simplify the processing at the decoding end and increase the flexibility of setting the IPF usage conditions. Using IPF on the basis of the TMRL mode can improve the accuracy of prediction and enhance the performance of video encoding.

[0400] In an example of this embodiment, the other reference line is a reference line that is adjacent to the extended reference line and located inside the extended reference line, or a reference line that is adjacent to the extended reference line and completely outside the extended reference line, or a reference line with an index of 0. The inside refers to a side close to the current block, and the outside refers to a side away from the current block.

[0401] In one example of this embodiment, a condition for decoding the TMRL fusion flag may be added, that is, continuing to decode the TMRL mode index and the TMRL fusion flag of the current block includes: after decoding the TMRL mode index of the current block, determining whether the size of the current block meets a set condition; if the set condition is met, decoding the TMRL fusion flag of the current block; if the set condition is not met, skipping decoding the TMRL fusion flag of the current block;

[0402] The setting conditions include any one or more of the following conditions:

[0403] The width of the current block is greater than the set value;

[0404] The height of the current block is greater than the set value;

[0405] The width and height of the current block are greater than the set value;

[0406] The current frame to which the current block belongs is not an inter-frame; that is, intra-frame blocks using the fusion mode are prohibited from appearing in B-frames and P-frames.

[0407] Because small blocks typically have more texture, the improvement of using IPF is limited. Therefore, this embodiment sets the current block size greater than a set value as a condition for decoding the TMRL fusion flag. When the current block size is not greater than the set value, the TMRL fusion flag may not be encoded or decoded to improve coding efficiency.

[0408] In an example of this embodiment, constructing a candidate list of TMRL modes for the current block includes:

[0409] Obtain N×M combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block, where N≥1, M≥1, and N×M≥2;

[0410] Predicting the template area of ​​the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template area and the predicted value;

[0411] Filling K combinations corresponding to the errors into a candidate list of a template-based multi-reference row intra prediction (TMRL) mode for the current block in ascending order of the errors, where 1≤K≤N×M;

[0412] The M intra-frame prediction modes are selected from angle modes other than angle modes of -45°, 0°, 45°, 90°, and 135°.

[0413] When constructing the candidate list of the TMRL mode, this embodiment limits the M intra-frame prediction modes involved in the combination to other angle modes except for some specific angles, which is conducive to the combination with the IPF to improve the prediction accuracy.

[0414] An embodiment of the present disclosure provides a video decoding method. During the stage of parsing syntax elements at the CU level, an additional decoding identifier tmrl_fusion_flag, ie, a TMRL fusion flag, is required.

[0415] For example, in the following table,

[0416]

[0417] The sps_mrl_enabled_flag in the table above is a sequence-level flag. A value of 1 indicates that MRL can be used in the current sequence, while a value of 1 indicates that MRL is not used. (y0% CtbSizeY) > 0 indicates that the current CU position is not the first row of a CTU. A value of 1 for tmrl_fusion_flag indicates that the subsequent TMRL mode will use fusion mode; otherwise, fusion mode is not used.

[0418] In another embodiment, whether to decode tmrl_fusion_flag can also be based on the size of the current coding block (Cb). For example, in the example in the table below, tmrl_fusion_flag is decoded only when the width of the coding block cbWidth>N and the height of the coding block cbHeight>M

[0419]

[0420]

[0421] In yet another embodiment, as shown in the example of the following table, tmrl_fusion_flag is decoded only when the coding block cbWidth*cbHeight>L.

[0422]

[0423] M, N, and L in the table are set values ​​of the dimensions, all of which are positive integers, and M can be equal to N.

[0424] When constructing a candidate list of TMRL modes based on template sorting, this embodiment traverses NxM combinations of N reference lines and M intra-frame prediction modes that can be used, generates a prediction value in the template area, and calculates the error with the reconstructed value of the template area. The top K are selected in ascending order according to the size of the error to form a candidate list of {reference line, prediction mode} combinations. The error value can be the sum of absolute difference (SAD), etc. This embodiment can limit the use of IPF to only some modes. When constructing the candidate list, one or more of the following intra-frame prediction modes can be excluded: PLANAR, DC, angle mode with a horizontal angle, angle mode with a vertical angle, angle mode with an angle of -45 degrees, angle mode with an angle of 45 degrees, and angle mode with an angle of 135 degrees.

[0425] The fusion in the prediction stage needs to be based on the value of tmrl_fusion_flag. When tmrl_fusion_flag is 1, the selected intra prediction mode and the selected reference row generate the prediction signal p a The selected intra prediction mode and reference line 0 generate the prediction signal p b , p a With p b Fusion generates the final prediction signal. Otherwise, directly use p a as the final prediction signal.

[0426] Accordingly, an embodiment of the present disclosure further provides a video encoding method, which is applied to an encoder, as shown in FIG18 , and includes:

[0427] Step 1110 , constructing a candidate list of template-based multiple reference line intra prediction (TMRL) modes for the current block, wherein the candidate list is filled with combinations of candidate extended reference lines and intra prediction modes for the current block;

[0428] Step 1120 , selecting a reference row and intra prediction mode combination for the current block through rate-distortion optimization;

[0429] Step 1130: When the coding condition of the TMRL mode of the current block is satisfied, encode the TMRL mode flag of the current block to indicate that the current block uses the TMRL mode, and encode the TMRL mode index of the current block to indicate the position of the selected combination in the candidate list;

[0430] Step 1140 , encoding the TMRL fusion flag of the current block to indicate whether the current block uses intra prediction fusion IPF or does not use IPF;

[0431] The encoding condition at least includes: the selected combination is in the candidate list.

[0432] Parts of the video encoding method of this embodiment that are not related to fusion may be the same as those of the aforementioned embodiment of the video encoding method using TMRL but not performing fusion, such as the setting of the encoding conditions in step 1130 may be the same, and so on.

[0433] In an example of this embodiment, encoding a TMRL fusion flag of the current block to indicate whether the current block uses intra prediction fusion IPF or does not use IPF includes: when the current block selects a combination including an angle mode in the candidate list, if at least one of the set constraints is met, encoding the TMRL fusion flag of the current block to indicate not using IPF; if none of the set constraints are met, encoding the TMRL fusion flag of the current block to indicate using IPF, wherein:

[0434] The set restriction conditions also include any one or more of the following restriction conditions:

[0435] The TIMD fusion mode is selected for the current block;

[0436] The current block selects DIMD mode;

[0437] The index of the reference row selected by the current block is greater than or equal to K, where K is an integer greater than or equal to 3;

[0438] The width of the current block is less than or equal to the set value;

[0439] The height of the current block is less than or equal to the set value;

[0440] The width and height of the current block are less than or equal to the set value;

[0441] The current block uses the intra-frame sub-block division ISP mode;

[0442] The angle of the angle mode currently selected in the block is any one of -45°, 0°, 45°, 90°, and 135°, or the angle mode currently selected in the block is an angle mode with an integer slope;

[0443] The current frame to which the current block belongs is an inter-frame;

[0444] The current frame to which the current block belongs is a chroma frame.

[0445] In an example of this embodiment, when constructing the candidate list of TMRL modes for the current block, the intra prediction modes participating in the combination exclude angle modes of -45°, 0°, 45°, 90°, and 135°.

[0446] When the encoder selects a combination from the candidate list of TMRL modes for the current block and uses IPF, it predicts the current block based on the combination using IPF. The prediction result of the combination is weighted with the prediction result of the angle mode in the combination and another reference row to obtain the predicted value of the current block. This can then determine the reconstructed value of the current block.

[0447] In order to use IPF on the basis of TMRL, the fused prediction mode obtained based on the combination of the extended reference line and the angle mode may also be included as a combination in the candidate list when constructing the candidate list of TMRL.

[0448] An embodiment of the present disclosure provides a method for constructing a multi-reference row intra prediction mode candidate list, which can be applied to an encoder or a decoder. As shown in FIG14 , the method includes:

[0449] Step 810 , obtaining N×M original combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block;

[0450] Step 820 , predicting the template region of the current block according to the N×M original combinations, and calculating the error between the reconstructed value of the template region and the predicted value;

[0451] Step 830, for each of the K original combinations with the smallest error including the predetermined angle mode, determine whether fusion is required. If fusion is required, fill the fused combination corresponding to the original combination into the candidate list of the template-based multi-reference row intra-frame prediction TMRL mode of the current block. If fusion is not required, fill the original combination into the candidate list, where K, N, and M are set positive integers, 1≤K≤N×M.

[0452] In this embodiment, for each of the K original combinations with the smallest error that includes a predetermined angle mode, whether fusion is required is determined. If fusion is required, the fused combination corresponding to the original combination is added to the candidate list of the TMRL mode of the current block, including:

[0453] Predicting a template area of ​​the current block according to the fused combination corresponding to the original combination, and calculating an error between a reconstructed value of the template area and a predicted value obtained by prediction; when the error corresponding to the original combination is greater than the error corresponding to the fused combination, determining that fusion is required; and when the error corresponding to the original combination is less than or equal to the error corresponding to the fused combination, determining that fusion is not required;

[0454] When predicting the current block based on the fused combination, a weighted sum of the first prediction result and the second prediction result is used as a prediction value of the current block; wherein the first prediction result is a result of predicting the current block based on the extended reference row and the angle mode in the original combination, and the second prediction result is a result of predicting the current block based on the second reference row and the angle mode in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0;

[0455] The predetermined angle pattern includes all angle patterns, or includes other angle patterns except the angle pattern with an integer slope.

[0456] The weighted sum of the first prediction result and the second prediction result in this embodiment can be calculated according to the above formula 1 or formula 2.

[0457] This embodiment sorts the unfused modes to select K combinations as a candidate list. Next, a further determination is made for each of the K combinations regarding whether the reference lines should be fused. This determination is based on the error between the predicted and reconstructed signals generated for the selected reference lines in the fused and unfused modes. If the error in the fused mode is smaller, fusion is used for that combination; otherwise, no fusion is performed. This embodiment only requires prediction and error calculation for 5xM+K combinations across the template area.

[0458] Another embodiment of the present disclosure provides a method for constructing a multi-reference row intra prediction mode candidate list, which can be applied to an encoder or a decoder. As shown in FIG15 , the method includes:

[0459] Step 910 , obtaining N×M original combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block;

[0460] Step 920: performing a fusion process on the N×M original combinations, wherein the fusion process includes: for each original combination including a predetermined angle pattern, when a set condition is met, replacing the original combination with a corresponding fused combination;

[0461] Step 930 , predicting the template region of the current block according to the N×M combinations after the fusion process, and calculating the error between the reconstructed value of the template region and the predicted value;

[0462] Step 940 , filling K combinations corresponding to the errors into a candidate list of template-based multiple reference row intra prediction (TMRL) modes for the current block in ascending order of the errors;

[0463] Among them, K, N, M are set positive integers, 1≤K≤N×M;

[0464] In which, when predicting the current block based on the fusion combination corresponding to the original combination, the weighted sum of the first prediction result and the second prediction result is used as the prediction value of the current block; wherein, the first prediction result is the result of predicting the current block based on the extended reference row and angle pattern in the original combination, and the second prediction result is the result of predicting the current block based on the second reference row and the angle pattern in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

[0465] In an example of this embodiment, the set conditions include any one or more of the following conditions: the size of the current block is greater than NxM, N and M are set positive integers; and the intra-frame prediction mode selected by the current block is not an angle mode with an integer slope; the predetermined angle mode includes all angle modes, or includes other angle modes except the angle mode with an integer slope.

[0466] The weighted sum of the first prediction result and the second prediction result in this embodiment can be calculated according to the above formula 1 or formula 2.

[0467] This embodiment does not require the templates to be sorted by error cost and sorted simultaneously for reference rows to be fused or not fused. When one or more of the following conditions are met, the fusion mode of the adjacent reference rows is used to participate in the template sorting and generate the combined {reference row, prediction mode} list; otherwise, the fusion mode is not used:

[0468] When the size of the current block is larger than NxM;

[0469] When the current intra prediction mode is not PLANAR, DC, horizontal angle, vertical angle, -45 degree angle, 45 degree angle, 135 degree angle mode

[0470] In the prediction stage, when the current block uses TMRL, the selected reference row and prediction mode are determined based on the tmrl_idx obtained by decoding and the candidate list generated based on the template sorting, and the same conditions are used to determine whether to use adjacent reference rows to generate the fused prediction result. That is, when one or more of the following conditions are met, the selected angle mode is IPF fused:

[0471] When the size of the current block is larger than NxM;

[0472] When the current intra prediction mode is not PLANAR, DC, horizontal angle, vertical angle, -45 degree angle, 45 degree angle, or 135 degree angle mode.

[0473] Another embodiment of the present disclosure provides a method for constructing a multi-reference row intra prediction mode candidate list, which can be applied to an encoder or a decoder. As shown in FIG16 , the method includes:

[0474] Step 1010 , obtaining N×M original combinations of extended reference lines and angle modes according to the N extended reference lines and M intra prediction modes of the current block;

[0475] Step 1020: Obtain a corresponding fusion combination according to the combination including the predetermined angle pattern in the N×M original combinations;

[0476] Step 1030 , predicting the template region of the current block according to the N×M original combinations and the obtained fused combination, and calculating the error between the reconstructed value of the template region and the predicted value obtained by prediction;

[0477] Step 1040 , in order of the errors from small to large, fill the K combinations corresponding to the errors into the candidate list of the template-based multi-reference row intra prediction (TMRL) mode of the current block, where K, N, and M are set positive integers, and 1≤K≤N×M.

[0478] This embodiment obtains a corresponding fused combination based on a combination including a predetermined angle pattern among the N×M original combinations, including: obtaining a fused combination based on each original combination; when predicting a current block based on the fused combination, using a weighted sum of a first prediction result and a second prediction result as a prediction value of the current block; wherein the first prediction result is a result of predicting the current block based on an extended reference row and the angle pattern in the original combination, the second prediction result is a result of predicting the current block based on a second reference row and the angle pattern in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0;

[0479] The predetermined angle pattern includes all angle patterns, or includes angle patterns other than angle patterns with integer slopes.

[0480] In this embodiment, when all the intra-frame prediction modes involved in the combination are angle modes, it is necessary to perform prediction and error value calculation on the template area for 5xMx2 combinations.

[0481] The weighted sum of the first prediction result and the second prediction result in this embodiment can be calculated according to the above formula 1 or formula 2.

[0482] In the above-mentioned embodiment of the present disclosure, taking the reference behavior indexes {1, 3, 5, 7, 12} as an example, the fused combination corresponding to the original combination may be, for example, a combination of a reference row indexed 1 and a certain angle pattern, and the corresponding fused combination may be a combination of two reference rows indexed 1 and 2 and the angle pattern. When predicting the current block based on this fused combination, the prediction result based on the reference row indexed 1 and the angle pattern, and the prediction result based on the reference row indexed 2 and the angle pattern are weighted to obtain the final prediction result of the current block. In another example, if the second reference row selects an inner adjacent reference row, the corresponding fused combination may be a combination of two reference rows indexed 0 and 1 and the angle pattern.

[0483] When the method of constructing a candidate list of the TMRL mode according to the above embodiment is applied to video encoding and video decoding, if the current block selected is a fusion combination in the candidate list, the current block is predicted according to the prediction method of the fusion combination; if the original combination in the candidate list is selected, the current block is predicted according to the prediction method of the original combination, and IPF is not used.

[0484] An embodiment of the present disclosure further provides a code stream, wherein the code stream is generated by the video encoding method described in any embodiment of the present disclosure.

[0485] An embodiment of the present disclosure also provides a device for constructing a multi-reference row intra-frame prediction mode candidate list, as shown in Figure 17, comprising a processor 71 and a memory 73 storing a computer program, wherein the processor 71, when executing the computer program, can implement the method for constructing a multi-reference row intra-frame prediction mode candidate list described in any embodiment of this document.

[0486] An embodiment of the present disclosure further provides an intra-frame prediction fusion device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the intra-frame prediction fusion method as described in any embodiment of the present disclosure.

[0487] An embodiment of the present disclosure further provides a video decoding device, comprising a processor and a memory storing a computer program, wherein the processor can implement the video decoding method described in any embodiment of the present disclosure when executing the computer program.

[0488] An embodiment of the present disclosure further provides a video encoding device, including a processor and a memory storing a computer program, wherein the processor can implement the video encoding method described in any embodiment of the present disclosure when executing the computer program.

[0489] An embodiment of the present disclosure further provides a video encoding and decoding system, which includes the video encoding device described in any embodiment of the present disclosure and the video decoding device described in any embodiment of the present disclosure.

[0490] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the intra-frame prediction fusion method described in any embodiment of the present disclosure, the method for constructing a multi-reference row intra-frame prediction mode candidate list described in any embodiment of the present disclosure, or implements the video decoding method described in any embodiment of the present disclosure, or implements the video encoding method described in any embodiment of the present disclosure.

[0491] The processors of the above embodiments of the present disclosure may be general-purpose processors, including central processing units (CPUs), network processors (NPs), microprocessors, etc., or other conventional processors; the processors may also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), discrete logic or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other equivalent integrated or discrete logic circuits, or a combination of the above devices. That is, the processors of the above embodiments may be any processing device or device combination that implements the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. If the embodiments of the present disclosure are partially implemented in software, the instructions for the software may be stored in a suitable non-volatile computer-readable storage medium, and one or more processors may be used to execute the instructions in hardware to implement the methods of the embodiments of the present disclosure. The term "processor" used herein may refer to the above structure or any other structure suitable for implementing the technology described herein.

[0492] In one or more exemplary embodiments above, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, such as according to a communication protocol. In this way, a computer-readable medium may generally correspond to a non-transitory tangible computer-readable storage medium or a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in this disclosure. A computer program product may include a computer-readable medium.

[0493] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection may also be referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient (transient) media, but rather refer to non-transient tangible storage media. As used herein, disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, or Blu-ray disc, among others, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0494] In some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0495] The technical solutions of the embodiments of the present disclosure can be implemented in a wide variety of devices or equipment, including wireless mobile phones, integrated circuits (ICs), or a group of ICs (e.g., chipsets). Various components, modules, or units are described in the embodiments of the present disclosure to emphasize the functional aspects of the devices configured to perform the described techniques, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

Claims

1. An intra-frame prediction fusion method, comprising: In the case where the intra prediction mode selected by the current block includes an angle mode, determining whether a restriction condition for using the intra prediction fusion IPF for the current block is met; When at least one of the restriction conditions is met, the IPF is restricted from being used when performing intra-frame prediction on the current block.

2. The method of claim 1, wherein: The restriction condition includes a mode quantity restriction condition: using IPF will result in the prediction of the current block requiring the fusion of N or more intra-frame prediction modes, where N is an integer greater than or equal to 3.

3. The method of claim 2, wherein: In the case of N=3, when the intra prediction mode selected by the current block includes the angle mode, determining whether the restriction condition of using the IPF for the current block is met; When at least one of the restriction conditions is met, limiting the use of the IPF when performing intra-frame prediction on the current block includes: When the template-based intra-mode derivation TIMD fusion mode is selected for the current block and the TIMD is fused with two angle modes that meet the IPF usage condition, it is determined that the mode quantity restriction condition is met, and IPF fusion is only allowed for one of the two angle modes when performing intra prediction on the current block; When the TIMD fusion mode is selected for the current block and only one of the two intra-frame prediction modes of TIMD fusion is an angle mode that meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode when intra-frame prediction is performed on the current block.

4. The method of claim 1, wherein: The restriction condition includes a mode quantity restriction condition: using IPF will result in the prediction of the current block requiring the fusion of M or more angle modes, where M is an integer greater than or equal to 2.

5. The method of claim 4, wherein: In the case where M=3, when the intra prediction mode selected by the current block includes an angle mode, determining whether a restriction condition for using the IPF for the current block is met; When at least one of the restriction conditions is met, limiting the use of the IPF when performing intra-frame prediction on the current block includes: When the TIMD fusion mode is selected for the current block and the TIMD fusion combines two angle modes that meet the IPF usage condition, it is determined that the mode quantity restriction condition is met, and only one of the two angle modes is allowed to be IPF-fused when performing intra-frame prediction on the current block; When the TIMD fusion mode is selected for the current block and only one of the two intra-frame prediction modes of TIMD fusion is an angle mode that meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode when intra-frame prediction is performed on the current block.

6. The method according to claim 3 or 5, wherein: The allowing IPF fusion to be performed on only one of the two angle modes includes: allowing IPF fusion to be performed on only the angle mode with the smallest cost or the second smallest cost among the two angle modes.

7. The method of claim 4, wherein: In the case where M=3, when the intra prediction mode selected by the current block includes an angle mode, determining whether a restriction condition for using the IPF for the current block is met; When at least one of the restriction conditions is met, limiting the use of the IPF when performing intra-frame prediction on the current block includes: When the decoding-side intra prediction mode is selected for the current block and the DIMD fusion mode is derived, and both DIMD fusion angle modes meet the IPF usage condition, it is determined that the mode quantity restriction condition is met, and only one of the two angle modes is allowed to be fused with IPF when performing intra prediction on the current block; When the current block uses the DIMD fusion mode and only one of the two angle modes of DIMD fusion meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode that meets the IPF usage conditions when performing intra-frame prediction on the current block.

8. The method of claim 4, wherein: In the case of M=2, when the intra-frame prediction mode selected by the current block includes the angle mode, determining whether the restriction condition of using the IPF for the current block is met; When at least one of the restriction conditions is met, limiting the use of the IPF when performing intra-frame prediction on the current block includes: When the TIMD fusion mode is selected for the current block and the TIMD fuses two angle modes, it is determined that the mode quantity restriction condition holds true, and IPF is not allowed to be used when performing intra prediction on the current block; When the TIMD fusion mode is selected for the current block, only one of the two intra-frame prediction modes of TIMD fusion is an angle mode, and the angle mode meets the IPF usage conditions, it is determined that the mode quantity restriction condition does not hold, and IPF fusion is allowed for the angle mode that meets the IPF usage conditions when intra-frame prediction is performed on the current block.

9. The method of claim 3, 5, 7 or 8, wherein: The angle pattern that meets the IPF usage conditions is not an angle pattern with an integer slope.

10. The method of claim 3, 5, 7 or 8, wherein: The method allows IPF fusion of an angle mode when performing intra-frame prediction on the current block, including: taking a weighted sum of two or more prediction results as the final prediction result of the current block; the two or more prediction results include: a prediction result obtained by predicting the current block based on a first reference row selected by the current block and the angle mode, and a prediction result obtained by predicting the current block based on a second reference row different from the first reference row and the angle mode.

11. The method of claim 1 , wherein: The restriction conditions include any one or more of the following restriction conditions: Constraint 1: The current block selects TIMD fusion mode; Constraint 2: The current block selects DIMD fusion mode; Constraint 3: The current block uses multiple reference rows (MRL). Constraint 4: The index of the reference row selected by the current block is greater than or equal to K, where K is an integer greater than or equal to 3; Constraint 5: The width of the current block is less than or equal to the set value; Constraint 5: The height of the current block is less than or equal to the set value; Constraint seven: The width multiplied by the height of the current block is less than or equal to the set value; Constraint 8: The current block uses the intra-frame sub-block division ISP mode; Constraint nine: The angle of the angle mode selected by the current block is any one of -45°, 0°, 45°, 90°, and 135°, or the angle mode selected by the current block is an angle mode with an integer slope; Constraint 10: The current frame to which the current block belongs is an inter-frame; Constraint 11: The current frame to which the current block belongs is not a luma frame; When at least one of the constraints one to eleven is satisfied, IPF is not allowed to be used when performing intra-frame prediction on the current block.

12. The method of claim 11, wherein: The restriction condition includes restriction condition four, wherein K=3 or 5 or 7 or 12.

13. The method of claim 1, wherein: The method further includes: when none of the restriction conditions are met, using the IPF when performing intra-frame prediction on the current block; the using the IPF when performing intra-frame prediction on the current block includes: When there is only one intra-frame prediction mode selected for the current block and it is an angle mode, the weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block; wherein, the first prediction result is the result of predicting the current block based on the first reference row selected by the current block and the first angle mode selected by the current block, the second prediction result is the result of predicting the current block based on the second reference row and the first angle mode, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

14. The method of claim 1, wherein: The method further includes: when none of the restriction conditions are met, using the IPF when performing intra-frame prediction on the current block; the using the IPF when performing intra-frame prediction on the current block includes: When the intra-frame prediction mode selected for the current block is a fusion mode, a weighted sum of more than three prediction results is used as the final prediction result of the current block; wherein the more than three prediction results include: multiple prediction results obtained by predicting the current block according to the first reference row selected for the current block and each intra-frame prediction mode in the fusion mode, and one or more prediction results obtained by predicting the current block according to the second reference row and one or more angle modes in the fusion mode that allow IPF fusion; the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

15. The method of claim 13 or 14, wherein: When the index of the first reference line is greater than or equal to K, the second reference line is adjacent to the first reference line and is closer to the current block than the first reference line, where K is an integer greater than or equal to 1.

16. The method of claim 13 or 14, wherein: When calculating the weighted sum of the first prediction result and the second prediction result, the weight assigned to the first prediction result is greater than the weight assigned to the second prediction result.

17. The method of claim 13, wherein: The weighted sum of the first prediction result and the second prediction result is calculated according to the following formula: p fusion =(w a p a +w b p b )>>shift or p fusion =(w a p a +w b p b +offset)>>shift Among them, p a is the first prediction result, p b For the second prediction result, p fusion The final prediction result of the current block, w a For p a The weight, w b For p b The weight of Among them, w a +w b =(1< <shift),offset=1<<(shift-1),shift> =1, shift and offset are set parameters.

18. The method of claim 13, wherein: The determining whether a restriction condition for using intra-frame prediction fusion IPF for the current block is met when the intra-frame prediction mode selected by the current block includes an angle mode; When none of the above constraints hold true, IPF is used when intra-prediction is performed on the current block; including: When the current block selects an extended reference row and an angle mode in the candidate list of the TMRL mode, and none of the restriction conditions are met, a first prediction result of predicting the current block based on the extended reference row and the angle mode, and a second prediction result of predicting the current block based on another reference row and the angle mode are calculated, and a weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block, wherein the other reference row is a reference row with an index of 0 or an adjacent row of the extended reference row.

19. A video decoding method, comprising: Decode the bitstream and determine the reference row and intra prediction mode selected for the current block; Predicting the current block according to the intra-frame prediction fusion method according to any one of claims 1 to 18 to obtain a predicted value of the current block; A reconstructed value of the current block is determined according to the predicted value of the current block.

20. A video encoding method, comprising: Determine the reference row and intra prediction mode selected by the current block through mode selection; Predicting the current block according to the intra-frame prediction fusion method according to any one of claims 1 to 18 to obtain a predicted value of the current block; The residual of the current block is determined based on the original value and the predicted value of the current block.

21. A video decoding method, comprising: When decoding determines that the current block uses the template-based multi-reference line intra prediction TMRL mode, continue decoding the TMRL mode index and TMRL fusion flag of the current block; Constructing a candidate list of the TMRL mode of the current block, and determining the extended reference row and intra prediction mode selected for the current block according to the candidate list and the TMRL mode index; In the case where the TMRL fusion flag indicates that intra-frame prediction fusion IPF is used, a weighted sum of the first prediction result and the second prediction result is used as the final prediction result of the current block; The first prediction result is obtained by predicting the current block based on the extended reference row and the intra-frame prediction mode, and the second prediction result is obtained by predicting the current block based on another reference row and the intra-frame prediction mode, and the intra-frame prediction mode is an angle mode.

22. The method of claim 21, wherein: The other reference line is a reference line adjacent to the extended reference line and located inside the extended reference line, or a reference line adjacent to the extended reference line and completely outside the extended reference line, or a reference line with an index of 0.

23. The method of claim 21, wherein: The method further includes: when the TMRL fusion flag indicates that the IPF is not used, predicting the current block according to the extended reference row and the intra-frame prediction mode to obtain a final prediction result of the current block.

24. The method of claim 21, wherein: The continuing to decode the TMRL mode index and the TMRL fusion flag of the current block includes: after decoding the TMRL mode index of the current block, determining whether the size of the current block meets a set condition, decoding the TMRL fusion flag of the current block when the set condition is met, and skipping decoding the TMRL fusion flag of the current block when the set condition is not met; The setting conditions include any one or more of the following conditions: The width of the current block is greater than the set value; The height of the current block is greater than the set value; The width and height of the current block are greater than the set value; The current frame to which the current block belongs is not an inter frame.

25. The method of claim 21, wherein: The candidate list of the TMRL mode of the current block is constructed, including: Obtain N×M combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block, where N≥1, M≥1, and N×M≥2; Predicting the template area of ​​the current block according to the N×M combinations respectively, and calculating the error between the reconstructed value of the template area and the predicted value; Filling K combinations corresponding to the errors into a candidate list of a template-based multi-reference row intra prediction (TMRL) mode for the current block in ascending order of the errors, where 1≤K≤N×M; The M intra-frame prediction modes are selected from angle modes other than angle modes of -45°, 0°, 45°, 90°, and 135°.

26. The method of claim 21, wherein: The weighted sum of the first prediction result and the second prediction result is calculated according to the following formula: p fusion =(w a p a +w b p b )>>shift or p fusion =(w a p a +w b p b +offset)>>shift Among them, p a is the first prediction result, p b For the second prediction result, p fusion The final prediction result of the current block, w a For p a The weight, w b For p b The weight of Among them, w a +w b =(1< <shift),offset=1<<(shift-1),shift> =1, shift and offset are set parameters.

27. A video encoding method, comprising: Constructing a candidate list of template-based multiple reference line intra prediction (TMRL) modes for the current block, wherein the candidate list is filled with combinations of extended reference lines and intra prediction modes of the current block candidates; selecting a reference line and intra prediction mode combination for the current block through rate-distortion optimization; When the encoding condition of the TMRL mode of the current block is satisfied, encoding the TMRL mode flag of the current block to indicate that the current block uses the TMRL mode, and encoding the TMRL mode index of the current block to indicate the position of the selected combination in the candidate list; Encode the TMRL fusion flag of the current block to indicate whether the current block uses intra-frame prediction fusion IPF or not; The encoding condition at least includes: the selected combination is in the candidate list.

28. The method of claim 27, wherein: The encoding of the TMRL fusion flag of the current block to indicate whether the current block uses intra-frame prediction fusion IPF or does not use IPF includes: when the current block selects a combination including the angle mode in the candidate list, if at least one of the set restriction conditions is met, encoding the TMRL fusion flag of the current block to indicate not using IPF; if none of the set restriction conditions are met, encoding the TMRL fusion flag of the current block to indicate using IPF, wherein: The set restriction conditions also include any one or more of the following restriction conditions: The TIMD fusion mode is selected for the current block; The current block selects DIMD mode; The index of the reference row selected by the current block is greater than or equal to K, where K is an integer greater than or equal to 3; The width of the current block is less than or equal to the set value; The height of the current block is less than or equal to the set value; The width and height of the current block are less than or equal to the set value; The current block uses the intra-frame sub-block division ISP mode; The angle of the angle mode currently selected in the block is any one of -45°, 0°, 45°, 90°, and 135°, or the angle mode currently selected in the block is an angle mode with an integer slope; The current frame to which the current block belongs is an inter-frame; The current frame to which the current block belongs is a chroma frame.

29. The method of claim 27, wherein: When constructing the candidate list of the TMRL mode of the current block, the intra prediction modes participating in the combination exclude angle modes with angles of -45°, 0°, 45°, 90°, and 135°.

30. A method for constructing a multi-reference line intra prediction mode candidate list, comprising: Obtain N×M original combinations of extended reference lines and angle modes according to the N extended reference lines and M intra prediction modes of the current block; Obtaining a corresponding fusion combination according to a combination including a predetermined angle pattern in the N×M original combinations; Predicting the template region of the current block according to the N×M original combinations and the obtained fusion combination, and calculating the error between the reconstructed value of the template region and the predicted value; In order of the errors from small to large, K combinations corresponding to the errors are filled into a candidate list of the template-based multi-reference row intra prediction TMRL mode of the current block, where K, N, and M are set positive integers, and 1≤K≤N×M.

31. The method of claim 30, wherein: The obtaining of a corresponding fusion combination according to a combination including a predetermined angle pattern in the N×M original combinations includes: A fused combination is obtained based on each original combination including the predetermined angle pattern; when predicting a current block based on the fused combination, a weighted sum of the first prediction result and the second prediction result is used as a prediction value of the current block; wherein the first prediction result is a result of predicting the current block based on the extended reference row and the angle pattern in the original combination, the second prediction result is a result of predicting the current block based on the second reference row and the angle pattern in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0; The predetermined angle pattern includes all angle patterns, or includes angle patterns other than angle patterns with integer slopes.

32. The method of claim 30, wherein: The weighted sum of the first prediction result and the second prediction result is calculated according to the following formula: p fusion =(w a p a +w b p b )>>shift or p fusion =(w a p a +w b p b +offset)>>shift Among them, p a is the first prediction result, p b For the second prediction result, p fusion The final prediction result of the current block, w a For p a The weight, w b For p b The weight of Among them, w a +w b =(1< <shift),offset=1<<(shift-1),shift> =1, shift and offset are set parameters.

33. A method for constructing a multi-reference line intra prediction mode candidate list, comprising: Obtain N×M original combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block; Predicting the template area of ​​the current block respectively according to the N×M original combinations, and calculating the error between the reconstructed value of the template area and the predicted value; For each of the K original combinations with the smallest error, including a combination of predetermined angle modes, determine whether fusion is required. If fusion is required, fill the fused combination corresponding to the original combination into the candidate list of the template-based multi-reference row intra-frame prediction TMRL mode of the current block. If fusion is not required, fill the original combination into the candidate list, where K, N, and M are set positive integers, and 1≤K≤N×M.

34. The method of claim 33, wherein: The step of determining whether each of the K original combinations with the smallest error including the predetermined angle mode needs to be fused, and if fusion is required, filling the fused combination corresponding to the original combination into the candidate list of the TMRL mode of the current block includes: Predicting a template area of ​​the current block according to the fused combination corresponding to the original combination, and calculating an error between a reconstructed value of the template area and a predicted value obtained by prediction; when the error corresponding to the original combination is greater than the error corresponding to the fused combination, determining that fusion is required; and when the error corresponding to the original combination is less than or equal to the error corresponding to the fused combination, determining that fusion is not required; When predicting the current block based on the fused combination, a weighted sum of the first prediction result and the second prediction result is used as a prediction value of the current block; wherein the first prediction result is a result of predicting the current block based on the extended reference row and the angle mode in the original combination, and the second prediction result is a result of predicting the current block based on the second reference row and the angle mode in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0; The predetermined angle pattern includes all angle patterns, or includes other angle patterns except the angle pattern with an integer slope.

35. The method of claim 33, wherein: The weighted sum of the first prediction result and the second prediction result is calculated according to the following formula: p fusion =(w a p a +w b p b )>>shift or p fusion =(w a p a +w b p b +offset)>>shift Among them, p a is the first prediction result, p b For the second prediction result, p fusion The final prediction result of the current block, w a For p a The weight, w b For p b The weight of Among them, w a +w b =(1< <shift),offset=1<<(shift-1),shift> =1, shift and offset are set parameters.

36. A method for constructing a multi-reference line intra prediction mode candidate list, comprising: Obtain N×M original combinations of extended reference lines and intra prediction modes according to the N extended reference lines and M intra prediction modes of the current block; Performing a fusion process on the N×M original combinations, the fusion process comprising: for each original combination including a predetermined angle pattern, when a set condition is met, replacing the original combination with a corresponding fusion combination; Predicting the template area of ​​the current block according to the N×M combinations after the fusion process, and calculating the error between the reconstructed value of the template area and the predicted value; Filling K combinations corresponding to the errors into a candidate list of a template-based multiple reference row intra prediction (TMRL) mode for the current block in an ascending order of the errors; Among them, K, N, M are set positive integers, 1≤K≤N×M; In which, when predicting the current block based on the fusion combination corresponding to the original combination, the weighted sum of the first prediction result and the second prediction result is used as the prediction value of the current block; wherein, the first prediction result is the result of predicting the current block based on the extended reference row and angle pattern in the original combination, and the second prediction result is the result of predicting the current block based on the second reference row and the angle pattern in the original combination, and the second reference row is an adjacent row of the first reference row or a reference row with an index of 0.

37. The method of claim 36, wherein: The setting conditions include any one or more of the following conditions: The size of the current block is greater than NxM, where N and M are set positive integers; The intra prediction mode selected for the current block is not an angle mode with an integer slope; The predetermined angle pattern includes all angle patterns, or includes angle patterns other than angle patterns with integer slopes.

38. The method of claim 36, wherein: The weighted sum of the first prediction result and the second prediction result is calculated according to the following formula: p fusion =(w a p a +w b p b )>>shift or p fusion =(w a p a +w b p b +offset)>>shift Among them, p a is the first prediction result, p b For the second prediction result, p fusion The final prediction result of the current block, w a For p a The weight, w b For p b The weight of Among them, w a +w b =(1< <shift),offset=1<<(shift-1),shift> =1, shift and offset are set parameters.

39. A code stream, wherein The code stream is generated by the video encoding method according to any one of claims 20, 27 to 29.

40. An intra-frame prediction fusion device, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it is capable of implementing the intra-frame prediction fusion method according to any one of claims 1 to 18.

41. A device for constructing a candidate list of intra prediction modes for multiple reference lines, comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the method for constructing a multi-reference row intra prediction mode candidate list as described in any one of claims 30 to 38.

42. A video decoding device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the video decoding method according to any one of claims 19, 21 to 26.

43. A video encoding device comprising a processor and a memory storing a computer program, wherein: When the processor executes the computer program, it can implement the video encoding method according to any one of claims 20, 27 to 29.

44. A video encoding and decoding system, wherein: It includes the video encoding device as claimed in claim 36 and the video decoding device as claimed in claim 35.

45. A non-transitory computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it can implement the intra-frame prediction fusion method as described in any one of claims 1 to 18, or implement the video decoding method as described in any one of claims 19, 21 to 26, or implement the video encoding method as described in any one of claims 20, 27 to 29, or implement the method for constructing a multi-reference row intra-frame prediction mode candidate list as described in any one of claims 30 to 38.

Images