Multi-reference line index list sorting method, video encoding / decoding method, apparatus and system

Abstract

A multi-reference line index list sorting method, a video encoding / decoding method, apparatus, and system, which predict respective template regions according to at least some reference lines located outside the template region of the current block, calculate an error between the reconstructed value of the template region and the predicted value obtained by prediction, embed the index of the reference line corresponding to the error into the MRL index list of the current block in ascending order from the smaller error to the larger error, obtain an MRL index list sorted based on the current block, and when encoding / decoding, can sort the MRL index list of the current block, and decode or encode the multi-reference line index of the current block according to the sorted MRL index list. The embodiments of the present disclosure can save encoding overhead and improve encoding performance.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to video technology, but are not limited thereto. More specifically, the present disclosure relates to a multi-reference line index list sorting method, a video encoding / decoding method, apparatus, and system.

Background Art

[0002] Digital video compression technology mainly facilitates transmission and storage by compressing huge digital video data. Current common video encoding / decoding standards, such as H.266 / Versatile Video Coding (VVC), all use a block-based hybrid encoding framework. Each frame in a video is divided into square largest coding units (LCUs) of the same size (e.g., 128×128, 64×64, etc.). Each largest coding unit may be divided into rectangular coding units (CUs) according to rules. The coding unit may further be divided into a prediction unit (PU), a transform unit (TU), etc. The hybrid encoding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filter. The prediction module includes intra prediction and inter prediction for reducing or removing redundancy in the video. Intra blocks predict with pixels around the block as references, and inter blocks refer to information of spatially adjacent blocks and reference information in other frames. For the predicted signal, the residual information becomes a code stream through block-based transformation, quantization, and entropy coding. These technologies are described in the standards and are implemented in various fields related to video compression.

[0003] With the rapid increase of Internet videos and the growing demand for video clarity among people, although the existing digital video compression standards can save a lot of video data, in order to reduce the bandwidth and traffic pressure of digital video transmission, there is still a need to pursue better digital video compression technology.

Summary of the Invention

Means for Solving the Problems

[0004] The following is an overview of the subject matter described in detail in this specification. This overview is not intended to limit the scope of the claims.

[0005] In one embodiment of the present disclosure, a multi-reference line index list sorting method is provided, and the method includes:

[0006] predicting each of the template areas according to at least some of the reference lines located outside the template area of the current block, and calculating an error between a reconstructed value of the template area and a predicted value obtained by prediction;

[0007] embedding the index of the reference line corresponding to the error into the MRL index list of the current block in ascending order from the smallest to the largest of the errors, and obtaining an MRL index list sorted based on the current block.

[0008] In one embodiment of the present disclosure, a video decoding method is further provided, and the method includes:

[0009] decoding the intra-prediction related syntax elements of the current block in the code stream, and continuing to decode the multi-reference line index of the current block when the current block permits the use of multi-reference line MRL, where the multi-reference line index is used to indicate the position of the reference line selected by the current block in the MRL index list.

[0010] If it is determined to permit sorting of the MRL index list of the current block according to the decryption result, sort the MRL index list of the current block according to the method described in any embodiment of the present disclosure to obtain an MRL index list sorted based on the current block;

[0011] determining a reference line selected by the current block according to the multi-reference line index and the MRL index list sorted based on the current block, and performing intra prediction on the current block according to the reference line selected by the current block.

[0012] In one embodiment of the present disclosure, a video encoding method is further provided, and the method includes:

[0013] performing intra prediction on the current block and determining a reference line selected by the current block;

[0014] If it is determined to permit sorting of the MRL index list of the current block, sort the MRL index list of the current block according to the method described in any embodiment of the present disclosure to obtain an MRL index list sorted based on the current block;

[0015] determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the MRL index list sorted based on the current block.

[0016] In one embodiment of the present disclosure, a code stream is provided, the code stream includes a multi-reference line index of a current block, the multi-reference line index is used to indicate the position of the index of the reference line selected by the current block in the MRL index list, and the MRL index list is a sorted MRL index list based on the current block or a set MRL index list.

[0017] In one embodiment of the present disclosure, a multi-reference line index list sorting apparatus is further provided, the apparatus includes a processor and a memory storing a computer program, and when the processor executes the computer program, the multi-reference line index list sorting method described in any embodiment of the present disclosure can be realized.

[0018] In one embodiment of the present disclosure, a video decoding apparatus is further provided, the apparatus includes a processor and a memory storing a computer program, and when the processor executes the computer program, the video decoding method described in any embodiment of the present disclosure can be realized.

[0019] In one embodiment of the present disclosure, a video encoding apparatus is further provided, the apparatus includes a processor and a memory storing a computer program, and when the processor executes the computer program, the video encoding method described in any embodiment of the present disclosure can be realized.

[0020] In one embodiment of the present disclosure, a video encoding / decoding system is further provided, the video encoding / decoding system includes the video encoding apparatus described in any embodiment of the present disclosure and the video decoding apparatus described in any embodiment of the present disclosure.

[0021] In one embodiment of the present disclosure, a non - transitory computer - readable storage medium is further provided. A computer program is stored in the computer - readable storage medium. When the computer program is executed by a processor, the multi - reference - line index list sorting method described in any embodiment of the present disclosure is realized, or the video decoding method described in any embodiment of the present disclosure is realized, or the video encoding method described in any embodiment of the present disclosure is realized.

[0022] After reading and understanding the drawings and the detailed description, other aspects can be understood.

Brief Description of the Drawings

[0023] The drawings are used to provide an understanding of the embodiments of the present disclosure, constitute a part of the specification, and are used together with the embodiments of the present disclosure to explain the technical solutions of the present disclosure, and do not constitute a limitation of the technical solutions of the present disclosure.

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0024] The present disclosure describes a plurality of embodiments, but the description is exemplary and not limiting. As will be apparent to those skilled in the art, there may be more embodiments and implementation schemes within the scope included by the embodiments described in the present disclosure.

[0025] In the description of the present disclosure, terms such as "exemplarily" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment described as "exemplary" or "for example" in the present disclosure should not be construed as being more preferable or advantageous than other embodiments. "And / or" in this specification is an explanation of the relevant relationship of the relevant objects and can represent three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. "Plurality" refers to two or more. Also, in order to clarify and facilitate the description of the technical solutions of the embodiments of the present disclosure, terms such as "first" and "second" are used to distinguish the same item or similar items whose functions and roles are substantially the same. It will be understood by those skilled in the art that terms such as "first" and "second" do not limit the number and execution order, and are not necessarily limited to being different.

[0026] When describing representative exemplary embodiments, the specification may present a method and / or process as a particular sequence of steps. However, to the extent that the method or process does not depend on the particular order of the steps described herein, the method or process should not be limited to the steps in the particular order described. Other sequences of steps understandable to those skilled in the art are possible. Therefore, the particular order of steps described in the specification should not be construed as limiting the claims. Further, the claims for the method and / or process should not be limited to the steps of performing them in the order described, and it is readily understood by those skilled in the art that these orders can be changed and still remain within the spirit and scope of the embodiments of the present disclosure.

[0027] Provided by embodiments of the present disclosure Multi-reference line index list sorting The method and the video encoding / decoding method may be applied to various video encoding / decoding standards, for example, H.264 / Advanced Video Coding (High Efficiency Video Coding, AVC), H.265 / High Efficiency Video Coding (High Efficiency Video Coding, HEVC), H.266 / Versatile Video Coding (Versatile Video Coding, VVC), AVS (Audio Video coding Standard), and MPEG (Moving Picture Experts Group), AOM (Alliance for Open Media), other standards created by JVET (Joint Video Experts Team) and extensions of these standards, or any other customized other standards, etc.

[0028] FIG. 1A is a block diagram of a video encoding / decoding system according to an embodiment of the present disclosure. As shown in the figure, the system is divided into an encoding-side device 1 and a decoding-side device 2. The encoding-side device 1 generates a code stream. The decoding-side device 2 may decode the code stream. The decoding-side device 2 may receive the code stream from the encoding-side device 1 via a link 3. The link 3 includes one or more media or devices that enable the code stream to move from the encoding-side device 1 to the decoding-side device 2. In one example, the link 3 includes one or more communication media that enable the encoding-side device 1 to directly transmit the code stream to the decoding-side device 2. The encoding-side device 1 modulates the code stream according to a communication standard (e.g., a wireless communication protocol) and transmits the modulated code stream to the decoding-side device 2. The one or more communication media may include wireless and / or wired communication media and may form part of a packet network. In another example, the code stream may be output from an output interface 15 to a storage device, and the decoding-side device 2 may read the data stored in the storage device via code stream transmission or download.

[0029] As shown in the figure, the encoding-side device 1 includes a data source 11, a video encoding device 13, and an output interface 15. The data source 11 includes a video capture device (e.g., a video camera), an archive containing previously captured data, a feed interface for receiving data from a content provider, a computer graphics system for generating data, or a combination of these sources. The video encoding device 13 encodes the data from the data source 11 and then outputs it to the output interface 15. The output interface 15 may include at least one of a regulator, a modem, and a transmitter. The decoding-side device 2 includes an input interface 21, a video decoding device 23, and a display device 25. The input interface 21 includes at least one of a receiver and a modem. The input interface 21 may receive a code stream via link 3 or from a storage device. The video decoding device 23 decodes the received code stream. The display device 25 is used to display the decoded data. The display device 25 may be integrated with other devices of the decoding-side device 2 or installed separately. The display device 25 is selective for the decoding side. In other examples, the decoding side may include other devices or facilities that apply the decoded data.

[0030] Based on the video encoding / decoding system shown in FIG. 1A, various video encoding / decoding methods may be used to achieve video compression and decompression.

[0031] FIG. 1B is a block diagram of an exemplary video encoding apparatus according to an embodiment of the present disclosure. As shown in the figure, the video encoding apparatus 1000 includes a prediction unit 1100, a splitting unit 1101, a residual generation unit 1102 (represented by a circled plus sign after the splitting unit 1101 in the figure), a transform processing unit 1104, a quantization unit 1106, an inverse quantization unit 1108, an inverse transform processing unit 1110, a reconstruction unit 1112 (represented by a circled plus sign after the inverse transform processing unit 1110 in the figure), a filter unit 1113, a decoded picture buffer 1114, and an entropy encoding unit 1115. Here, the prediction unit 1100 includes an inter prediction unit 1121 and an intra prediction unit 1126, and the decoded picture buffer 1114 may be referred to as a decoded picture buffer, a decoded picture buffer, a decoded picture buffer, etc. Video Encoding device 1000 may include more, fewer, or different functional assemblies than the example, and for example, in some cases, may not include the transform processing unit 1104, the inverse transform processing unit 1110, etc.

[0032] The splitting unit 1101, in cooperation with the prediction unit 1100, splits the received video data into slices, coding tree units (CTUs), or other relatively large units. The video data received by the splitting unit 1101 may be a video sequence including video frames such as I-frames, P-frames, or B-frames.

[0033] The prediction unit 1100 may split a CTU into coding units (CUs) and perform intra prediction coding or inter prediction coding on the CUs. When performing intra prediction and inter prediction on a CU, the CU may be split into one or more prediction units (PUs).

[0034] The inter prediction unit 1121 may perform inter prediction on the PU and generate prediction data of the PU. The prediction data includes a predicted block of the PU, motion information of the PU, and various syntax elements. The inter prediction unit 1121 may include a motion estimation (ME) unit and a motion compensation (MC) unit. The motion estimation unit may generate a motion vector by being used for motion estimation, and the motion compensation unit may be used to obtain or generate a predicted block according to the motion vector.

[0035] The intra prediction unit 1126 may perform intra prediction on the PU and generate prediction data of the PU. The prediction data of the PU may include a predicted block of the PU and various syntax elements.

[0036] The residual generation unit 1102 may subtract the predicted block of the PU obtained by dividing the CU based on the original block of the CU to generate a residual block of the CU.

[0037] The transform processing unit 1104 may divide the CU into one or more transform units (TU), and the division between the prediction unit and the transform unit may be different. The TU-related residual block is a sub-block obtained by dividing the residual block of the CU. By applying one or more transforms to the TU-related residual block, a TU-related coefficient block is generated.

[0038] The quantization unit 1106 may quantize the coefficients in the coefficient block based on the selected quantization parameter and adjust the degree of quantization of the coefficient block by adjusting the quantization parameter (QP).

[0039] The inverse quantization unit 1108 and the inverse transform unit 1110 may apply inverse quantization and inverse transform to the coefficient block respectively to obtain a TU-related reconstructed residual block.

[0040] The reconstruction unit 1112 may add the reconstruction residual block and the prediction block generated by the prediction unit 1100 to generate a reconstructed image.

[0041] The filter unit 1113 executes a loop filter on the reconstructed image and stores the filtered reconstructed image in the decoded image buffer 1114 as a reference image. The intra prediction unit 1126 may perform intra prediction by extracting a reference image of a block in the vicinity of the PU from the decoded image buffer 1114. The inter prediction unit 1121 may perform inter prediction on the PU of the current frame image using the reference image of the previous frame cached in the decoded image buffer 1114.

[0042] The entropy encoding unit 1115 may perform an entropy encoding operation on the received data (for example, syntax elements, quantized coefficient blocks, motion information, etc.).

[0043] FIG. 1C is a block diagram of an exemplary video decoding apparatus according to an embodiment of the present disclosure. As shown in the figure, the video decoding apparatus 101 includes an entropy decoding unit 150, a prediction unit 152, an inverse quantization unit 154, an inverse transform processing unit 155 , a reconstruction unit 158 (represented by a circled plus sign after the inverse transform processing unit 155 in the figure), a filter unit 159, and a decoded image buffer 160. In other embodiments, the video Decoding device 101 may include more, fewer, or different functional assemblies. For example, in some cases, it may not include the inverse transform processing unit 155, etc.

[0044] The entropy decoding unit 150 may perform entropy decoding on the received code stream and extract syntax elements, quantized coefficient blocks, and motion information of the PU, etc. The prediction unit 152, the inverse quantization unit 154, the inverse transform processing unit 155The inverse quantization unit 154, the inverse transform processing unit, the prediction unit 152, the reconstruction unit 158, and the filter unit 159 may all perform corresponding operations based on the syntax elements extracted from the code stream.

[0045] The inverse quantization unit 154 may inverse quantize the quantized TU-related coefficient block.

[0046] Inverse transform processing unit 155 may facilitate the generation of the reconstruction residual block of the TU by applying one or more inverse transforms to the inverse quantized coefficient block.

[0047] The prediction unit 152 includes an inter prediction unit 162 and an intra prediction unit 164. When the PU uses intra prediction coding, the intra prediction unit 164 determines the intra prediction mode of the PU based on the syntax elements decoded from the code stream, and performs intra prediction according to the determined intra prediction mode and the reconstructed reference information in the vicinity of the PU obtained from the decoded picture buffer 160, and may generate the prediction block of the PU. When the PU uses inter prediction coding, the inter prediction unit 162 determines one or more reference blocks of the PU based on the motion information of the PU and the corresponding syntax elements, and may generate the prediction block of the PU based on the reference blocks obtained from the decoded picture buffer 160.

[0048] The reconstruction unit 158 may obtain a reconstructed image based on the TU-related reconstruction residual block and the prediction block of the PU generated by the prediction unit 152.

[0049] The filter unit 159 may perform a loop filter on the reconstructed image, and the loop-filtered reconstructed image is stored in the decoded picture buffer 160. The decoded picture buffer 160 may be used for subsequent motion compensation, intra prediction, inter prediction, etc. by providing a reference image, and output the filtered reconstructed image as decoded video data and display it on a display device.

[0050] Based on the above video encoding device and video decoding device, the following basic encoding and decoding processes may be executed. On the encoding side, an image of one frame is divided into blocks, and intra prediction, inter prediction, or other algorithms are performed on the current block to generate a predicted block of the current block. Using the original block of the current block, the predicted block is subtracted to obtain a residual block. The residual block is transformed and quantized to obtain quantization coefficients, and the quantization coefficients are entropy encoded to generate a code stream. On the decoding side, intra prediction or inter prediction is performed on the current block to generate a predicted block of the current block. Meanwhile, the quantization coefficients obtained by decoding the code stream are inverse quantized and inverse transformed to obtain a residual block. The predicted block and the residual block are added to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image. Based on the image or blocks, a loop filter is performed on the reconstructed image to obtain a decoded image. Similarly on the encoding side, a decoded image is obtained by the same operations as on the decoding side. The decoded image obtained by the encoding side is usually also referred to as a reconstructed image. The decoded image may also be used as a reference frame for performing inter prediction on subsequent frames. The block division information, mode information such as prediction, transformation, quantization, entropy encoding, loop filter, and parameter information determined by the encoding side may be written into the code stream as needed. The decoding side determines the same block division information, mode information such as prediction, transformation, quantization, entropy encoding, loop filter, and parameter information as the encoding side by decoding the code stream or analyzing according to existing information, thereby ensuring that the decoded image obtained by the encoding side is the same as the decoded image obtained by the decoding side.

[0051] The above takes the block-based hybrid encoding framework as an example, but the embodiments of the present disclosure are not limited thereto. With the development of technology, one or more modules in the framework and one or more steps in the process may be replaced or optimized.

[0052] In this specification, the current block may be a block-level encoding unit such as the current coding unit (CU) or the current prediction unit (PU) in the current image.

[0053] When the encoding side performs intra prediction, usually, with the help of various angular modes and non-angular modes, the current block is predicted to obtain a predicted block. According to the rate-distortion information calculated from the predicted block and the original block, the optimal prediction mode of the current block is filtered. After encoding the prediction mode, it is transmitted to the decoding side via the code stream. The decoding side obtains the prediction mode by decoding and performs intra prediction on the current block according to the prediction mode. Through the development of previous digital video encoding and decoding standards, the non-angular modes are relatively stable, including the average value mode (i.e., DC mode) and the planar mode (i.e., Planar mode). The angular modes continue to increase with the evolution of digital video encoding and decoding standards. Taking the international digital video encoding standard H series as an example, the H.264 / AVC standard has only 8 conventional angular prediction modes and 1 conventional non-angular prediction mode, while H.265 / HEVC is extended to 33 conventional angular prediction modes and 2 conventional non-angular prediction modes. In H.266 / VVC, as shown in FIG. 2, there are the conventional intra prediction modes Planar mode, DC mode, and 65 angular modes. Here, the DC mode is applied to large-area flat regions, and its predicted value is obtained by calculating the average value of the reference pixels on the left and / or above. The Planar mode is applied when the pixel changes gradually, that is, in regions where the pixel values change slowly.

[0054] When directly encoding the mode of a prediction block, 67 types of modes require 7 bits for encoding, resulting in a large amount of data. According to statistical characteristics, pixels in regions closer to the current block are more likely to select the same intra prediction mode as the current block. In response to this characteristic, HEVC, VVC, and the Enhanced Compression Model (ECM) all use the Most Probable Mode (MPM) technology. ECM is reference software that integrates various new tools based on the VTM-10.0 reference software to further explore encoding and decoding performance.

[0055] The MPM technology first constructs an MPM list, which is filled with the intra prediction modes most likely to be selected by the current block. If the prediction mode selected by the current block is in the MPM list, only its index number (requiring only 3 bits) needs to be encoded. If the prediction mode selected by the current block is not in the MPM and is in one of the 61 non-MPM modes, the prediction mode is encoded using a Truncated Binary Code (TBC) at the entropy encoding stage.

[0056] In VVC, regardless of whether multi-reference line (MRL) and intra sub-partitions (ISP) are applied or not, there are six prediction modes in the MPM list. On the other hand, the MPM in ECM is divided into MPM and Secondary MPM (second MPM), and MPM and Secondary MPM are a list with a length of 6 and a list with a length of 16, respectively. Among the six modes of the MPM list, the Planar mode is always in the first position of the MPM. The remaining five positions are sequentially performed in the following three steps until the five MPM positions are filled, and the extra modes automatically enter the Secondary MPM. The first step: As shown in Figure 3, the prediction modes used by the five prediction blocks adjacent to the current block are the prediction modes used by the prediction blocks at the upper left (AL), upper (A), upper right (AR), left (L), and lower left (BL) positions of the current block. The second step: The reconstructed pixels around the current block use the mode derived based on the gradient histogram. The third step: The angle mode close to the angle of the angle mode selected in the first step. The Secondary MP list is composed of several main angle modes other than the modes in the MPM.

[0057] Since the encoding / decoding order of the MPM identifier bits is after the MRL mode, the encoding / decoding of the MPM in ECM needs to depend on the MRL identifier bits. The following table is an example. JPEG2025521742000032.jpg244152 JPEG2025521742000033.jpg163151

[0058] Template based intra mode derivation (TIMD) is an intra prediction mode for a luminance frame, and the TIMD mode is generated by candidate modes in the MPM list and a template. In ECM, as shown in FIG. 4, the left adjacent region and the upper adjacent region of the current block (illustrated as Current CU) 11 constitute the template region 12 of the current block, and outside the template region 12 (i.e., on the left and upper sides), a template reference region 13 is provided, and exemplary dimensions and positions of each region are as shown in the figure. The template reference region 13 may be one row adjacent to the upper side of the template region, or one column adjacent to the left side.

[0059] TIMD assumes that the distribution characteristics of the current block and the template region of the current block are the same, uses the reconstructed value of the template reference region as the reference row, traverses all modes in the MPM and Secondary MPM, predicts the template region, and obtains a prediction result. Next, the sum of absolute transformed differences (SATD) between the reconstructed value on the template region and the prediction result of each mode is calculated, the mode with the minimum SATD, i.e., the optimal mode, is selected, and the mode is set as the TIMD mode of the current block. The decoder side may also derive the TIMD mode by the same derivation method. When the sequence permits the use of TIMD, each current block requires one flag bit to indicate whether to use TIMD. When the selected intra prediction mode for the current block is the TIMD mode, the current block predicts using the TIMD mode, and the decoding processes of the remaining luminance frames and syntax elements related to intra prediction such as ISP and MPM may all be skipped, thereby significantly reducing the coding bits of the mode.

[0060] After obtaining the SATD between the reconstructed value on the template area and each mode prediction result, the TIMD mode to be finally used may be determined according to the following method.

[0061] JPEG2025521742000034.jpg36149

[0062] If cost1 × 2 ≤ cost2, set mode1 as the TIMD mode of the current block.

[0063] If cost1 × 2 > cost2, set the prediction mode that weights the prediction results of mode1 and mode2 as the TMID mode of the current block, which is also called the TIMD fusion mode.

[0064] The weighting method and weights are as shown in the following formulas (1) and (2).

[0065] JPEG2025521742000035.jpg18109

[0066] JPEG2025521742000036.jpg2482

[0067] JPEG2025521742000037.jpg36149

[0068] In HEVC, for intra prediction, the upper row and the left column closest to the current block are used as references for prediction. If the reconstructed values of this row and column have a large error from the original pixel values, it will also have a significant impact on the prediction quality of the current block. To solve this problem, VVC adopts the MRL intra prediction technique. In addition to using the reference pixels of the closest row, that is, reference line 0 (Reference line0), VVC can further use reference line 1 (Reference line1) and reference line 2 (Reference line2) as extended reference lines for intra prediction. To reduce the encoding complexity, MRL is only used in the non-planar mode in MPM. When the encoder predicts each angular mode, after trying all three of these reference lines, the encoder selects the one reference line with the minimum rate-distortion cost (RD Cost). The index of the selected reference line is encoded and sent to the decoder side in the same way as the index of the selected intra prediction mode. The decoder decodes to obtain the index of the reference line and then selects the corresponding reference line according to the index of the reference line for prediction.

[0069] Figure 5 shows an example with four reference lines, which includes reference line 0 (reference line0) 221 adjacent to the current block, that is, the reference line with index 0, reference line 1 (reference line1) 222 one row away from the current block, that is, the reference line with index 1, reference line 2 (reference line2) 223 two rows away from the current block, that is, the reference line with index 2, and reference line 3 (reference line3) 224 three rows away from the current block, that is, the reference line with index 3. When predicting, only some of the reconstructed values of the reference lines may be used. In one example, when the area in reference line 0 is used as the template area of the current block, reference line 0 may be called the reference line where the template area is located, and reference lines 1 to 3 may be called the reference lines located outside the template area.

[0070] In this specification, the indexes of the reference lines are all numbered in the manner shown in FIG. 5. Except for the reference line with an index of 0, the other reference lines are extended reference lines.

[0071] In this specification, the reference lines are called "lines" for the convenience of explanation. In fact, one reference line actually includes one row and one column, and the reconstructed value of the reference line used for prediction also includes the reconstructed value of one row and one column, which is the same as the method generally described in this field.

[0072] In ECM, the decoding method related to the multi-reference line index is as shown in Table 1 and Table 2 below. Here, Table 1 is used when the current block does not use TIMD, and Table 2 is used when the current block uses TIMD.

[0073]

Table 1

[0074] In ECM, the MRL tool may use more reference lines. To encode the reference lines selected by the current block (i.e., the reference lines selected by the current block), the indices of multiple candidate reference lines are embedded into one list. In this specification, this list is called the multi-reference line index list, abbreviated as the MRL index list, but may also be called the multi-reference line list, candidate reference line list, reference line index list, etc. When the current block does not use TIMD, the length of the multi-reference line index list is 6, that is, the indices of 6 reference lines may be embedded in a total of 6 positions. The indices and their order of these 6 reference lines are fixed and are represented as MULTI_REF_LINE_IDX[6] = {0, 1, 3, 5, 7, 12}. In this multi-reference line index list, the index embedded in the first position is 0, which is the index of the reference line closest to the current block, and the indices embedded in the second to sixth positions are 1, 3, 5, 7, and 12 respectively, which are the indices of 5 extended reference lines from the closest to the farthest from the current block.

[0075] When encoding the reference lines selected by the current block, due to the existence of the multi-reference line index list, instead of directly encoding the index of the reference line, the index of the position of the index of the reference line in the multi-reference line index list is encoded, which is called the multi-reference line index (multiRefIdx). The multi-reference line index is used to indicate the position of the index of the reference line selected by the current block in the multi-reference line index list. Taking the MRL index list {0, 1, 3, 5, 7, 12} as an example, the indices from the first to the sixth positions are 0 to 5 respectively. If the index of the reference line selected by the current block is 7 and the index 7 is in the fifth position in the multi-reference line index list, the multi-reference line index is 4.

[0076] The multi-reference line index (multiRefIdx) may be encoded using a method such as a unary truncation code based on the context model, but is not limited thereto. The smaller the value of the multi-reference line index, the shorter the code length and the faster the decoding.

[0077] The MRL mode may be used simultaneously with the TIMD mode. When TIMD is used, the length of the MRL index list is 3, embedding the indexes of three reference lines, and the order between the indexes is fixed, represented as MULTI_REF_LINE_IDX[3] = {0, 1, 3}. Here, 0 is the index of the reference line in the template area, and 1 and 3 are the indexes of the extended reference lines.

[0078] The encoding and decoding methods of the multi-reference line index are as shown in Table 2.

[0079]

Table 2

[0080] For ease of understanding, in the above, when MRL uses TIMD and when MRL does not use TIMD, they are exemplified in two separate tables, and the above two tables may be integrated into one table.

[0081] In different standards, the technique of deriving a list of one most likely prediction modes (MPM) using the blocks around the current block may have different names. For example, in AV2 (AVM), it is called Adaptive Intra Mode Coding (AIMC), and in AVS3, for screen content coding, it is called Frequency-based Intra Mode Coding (FIMC). In the case of non-screen content coding, this technique like MPM is always effective. On the other hand, the multi-reference line technique (MRL) for performing intra prediction using multiple reference lines is called Multiple reference line selection for intra prediction (MRLS) in AV2 (AVM). This is simply a difference in names. When this embodiment uses MPM and MRL, it similarly covers these substantially the same techniques in other standards.

[0082] Compared with VVC, ECM extends the number of reference line candidates for MRL, but only brings a 0.09% improvement in coding performance in intra frame coding. According to research, this is due to the increase in the number of MRL candidate index lines. More identification positions are required to code the MRL mode, resulting in an increase in the additional burden. Also, as can be seen from the MRL decoding methods shown in Table 1 and Table 2 above, the multi-reference line index list is composed of indexes in a specified fixed order. The closer the position of the index of the selected reference line is to the top in the table, the lower the cost of coding the multi-reference line index.

[0083] In one embodiment of the present disclosure, a method for sorting a multi-reference line index list is provided, and this method may be used on the encoding side or the decoding side. As shown in FIG. 6, the method includes the following steps.

[0084] Step 110: Predict each of the template areas according to at least some of the reference lines located outside the template area of the current block, and calculate the error between the reconstruction value of the template area and the predicted value obtained by prediction.

[0085] The above error may be represented by the absolute error and SAD, or may be represented by the absolute value of error conversion and SATD, but is not limited thereto, and may also be represented by the sum of squared differences (SSD), the mean absolute difference (MAD), the mean squared error (MSE), etc.

[0086] Step 120: Embed the index of the reference line corresponding to the error into the MRL index list of the current block in ascending order from the smaller to the larger of the errors, and obtain an MRL index list sorted based on the current block.

[0087] In previous embodiments, the MRL index list is set by an encoding / decoding system, and the indexes and order of the reference lines embedded in the MRL index list are fixed. Embodiments of the present disclosure use a template-based method to predict each template region according to at least some of the reference lines located outside the template region of the current block, calculate the error between the reconstructed value of the template region and the predicted value obtained by prediction, and embed the indexes of the reference lines corresponding to the errors into the MRL index list of the current block in ascending order of the errors from the smallest to the largest, so as to obtain an MRL index list sorted based on the current block. Therefore, what is constructed according to the embodiments of the present disclosure is an MRL index list at the block level. Here, since the indexes and order of the reference lines are determined based on the distribution characteristics of the template region of the current block, the reference lines selected by the current block are likely to appear in the MRL index list and are likely to appear at a position closer to the beginning of the current MRL index list. That is, the values of the multi-reference line indexes that need to be encoded are likely to be relatively small values, so the code length is short, the bandwidth occupancy is small, the decoding speed is fast, and thereby the encoding performance is improved.

[0088] In an exemplary embodiment of the present disclosure, the step of predicting each template region according to at least some reference lines located outside the template region of the current block includes, for each reference line among the at least some reference lines, using the reconstructed value of the reference line and the intra prediction mode selected by the current block to predict the template region. When the method of this embodiment is used on the encoding side, after the selection in the intra prediction mode is completed and one intra prediction mode (here, referring to conventional intra prediction modes such as the average mode, the planar mode, the angular mode, etc.) is selected for the current block, based on the intra prediction mode selected by the current block, it may be associated with each reference line participating in the socheet to predict the template region. When the method of this embodiment is used on the decoding side, after decoding the syntax elements related to the intra prediction of the current block and determining the intra prediction mode selected by the current block, the above prediction is performed. After obtaining the MRL index list socketed based on the current block, the reference line selected by the current block may be determined according to the multi-reference line index obtained by decoding.

[0089] In an exemplary embodiment of the present disclosure, the step of embedding the index of the reference line corresponding to the error into the MRL index list of the current block means embedding the index of the reference line corresponding to the error position by position starting from the K1-th position of the MRL index list and embedding up to the K2-th position at most. Here, 2 ≤ K1 ≤ K2 ≤ M, and M is the length of the MRL index list. Actually, the index to be embedded in the MRL index list may be less than or equal to M.

[0090] In the MRL index list of the current block, for example, the reference line shown in FIG. 5 2 21 (whose index is 0), etc., the reference line where the template region is located Index of may be embedded, or for example, the reference line shown in FIG. 5 2 22, 2 23, and 2Reference lines outside the template area, such as 24 (the indices of these three reference lines are 1, 2, and 3 respectively). Index of It may be embedded in Index of . When sorting the MRL index list of the current block, since the reference lines where the template area is located are not used for prediction, the indices of the reference lines where the template area is located do not participate in the sorting, and these reference lines are likely to be selected. In this embodiment, it is directly embedded at one or more positions close to the head of the MRL index list, and the indices of the reference lines participating in the sorting are embedded starting from the K1-th position for each position. For example, K1 may be 2, 3, but is not limited thereto.

[0091] In an exemplary embodiment of the present disclosure, the method further includes embedding the indices of the reference lines where the template area of the current block is located from the first position that does not participate in the sorting to the (K1 - 1)-th position in the MRL index list sorted based on the current block, where the index of the reference line embedded in the first position is 0. In this embodiment, when the template area is limited to the row and column closest to the current block, it is only necessary to embed the index 0 of the reference line closest to the current block. However, in other embodiments, the template area may be expanded. For example, it may be expanded to the second row above and the second column to the left of the current block, or expanded to a larger area, that is, installed in a plurality of reference lines closest to the current block. In this case, more reference line indices may be embedded in the MRL index list, but not all the indices of the reference lines overlapping with the template area need to be embedded in the MRL index list, and a part of them may be selected and embedded in the MRL index list.

[0092] In an exemplary embodiment of the present disclosure, the step of embedding the index of the reference line corresponding to the error into the MRL index list of the current block means embedding the index of the reference line corresponding to the error position by position starting from the first position of the MRL index list and embedding it up to the K2-th position at most. Here, 1 ≤ K2 ≤ M, and M is the length of the MRL index list. Compared with the foregoing embodiment, this embodiment has a difference that it is not necessary to embed the index of the reference line where the template area of the current block is located into the MRL index list. For example, assuming that there is only one reference line where the template area is located and the index of the reference line selected by the current block is 0, that is, when the extended reference line is not used, the index of the selected reference line does not need to be represented using the multi-reference line index and may be represented by setting the MRL usage flag to false. Therefore, the MRL index list does not necessarily need to embed index 0, but the indexes of the reference lines participating in the sorting may be embedded starting from the first position of the MRL index list in ascending order from the smallest error to the largest.

[0093] In an exemplary embodiment of the present disclosure, when K2 < M, from the (K2 + 1)-th position to the M-th position of the MRL index list, the indexes of M - K2 reference lines are embedded position by position in descending order of the distance from the current block, and the M - K2 reference lines are located outside the template area, specified by the system, and the indexes are different from the indexes embedded in the MRL index list.

[0094] In this embodiment, instead of embedding the indexes of the reference rows corresponding to the errors in one or more positions close to the rear of the MRL index list in ascending order from the smallest to the largest error, the designated reference rows outside the template area are embedded. For example, the length M of the MRL index list is 6, and the indexes of the reference rows sorted according to the error are embedded only in the second and third positions among them. In one example, the set MRL index list is {0, 1, 3, 5, 7, 12}, and 1, 3, 5, 7, 12 are the indexes of the designated reference rows. In this embodiment, predictions are made according to the reference rows with indexes from 1 to 12. Assuming that the two rows with the smallest errors are the reference rows with indexes 5 and 7, and since these two reference rows are the originally designated reference rows, the MRL index list sorted based on the current block is {0, 5, 7, 1, 3, 12}, that is, the indexes 1, 3, 12 of the designated reference rows in the set MRL index list are embedded in positions close to the rear of the sorted MRL index list. If the two rows with the smallest errors are the reference rows with indexes 4 and 6, since these two rows are not the designated reference rows, the list of the sorted MRL indexes is {0, 4, 6, 1, 3, 5}, that is, the indexes of the designated reference rows are embedded position by position in descending order from the closer to the farther distance from the current block of the reference rows.

[0095] In an exemplary embodiment of the present disclosure, the template area of the current block is set in one reference row closest to the current block, that is, in the reference row with index 0. Or, the template area of the current block is set in a plurality of reference rows closest to the current block. For example, it is set in two reference rows with indexes 0 and 1, or in three reference rows with indexes 0, 1, and 2, or in four reference rows with indexes 0, 1, 2, and 3, etc.

[0096] In an exemplary embodiment of the present disclosure, at least some of the reference rows located outside the template area of the current block are

[0097] 1) The N' reference lines closest to the current block outside the template area, or

[0098] 2) The odd-numbered lines among the N' reference lines closest to the current block outside the template area, or

[0099] 3) The even-numbered lines among the N' reference lines closest to the current block outside the template area, or

[0100] 4) N reference lines selected by comparing the errors between adjacent lines among the N' reference lines closest to the current block outside the template area, N ≤ N', or

[0101] 5) N specified reference lines among the N' reference lines closest to the current block outside the template area, including

[0102] where N' ≤ N max and N max is the maximum number of reference lines that allow participation in the sorting.

[0103] When sorting, first, it is necessary to determine the number of reference lines participating in the sorting and specifically which reference lines to select. This embodiment provides multiple selection methods.

[0104] In this embodiment, N set when the current block uses TIMD max and N set when the current block does not use TIMD maxAssume that both are 12, and the template area is in the reference line closest to the current block, that is, the reference line with index 0, and the reference line of the current block is not allowed to exceed the upper boundary of the CTU where it is located. In this case, the first method selects to participate in the sorting of the N' reference lines closest to the current block outside the template area. When the number of reference lines from outside the template area to the upper boundary of the CTU is 12 or more, N' = 12, that is, select the 12 reference lines closest to the current block outside the template area to participate in the sorting. If there are no 12 reference lines, N' is equal to the number of reference lines from outside the template area to the upper boundary of the CTU. The second method and the third method simplify the operation by selecting odd and even lines from the reference lines determined by the first method respectively.

[0105] In one example of this embodiment, the fourth method of this embodiment, that is, the method of selecting N reference lines by comparing the errors between adjacent lines from the N' reference lines closest to the current block outside the template area, may be implemented using the following method. Select the first reference line selected as the reference line closest to the current block outside the template area, and compare the newly selected reference line with its adjacent reference lines one by one. Select one adjacent reference line that is closest to the reference line and the error between the two reference lines is greater than the set threshold as the newly selected reference line, and loop like this until N' reference lines are compared and selected. Here, the error refers to the error between the reconstruction values of two reference lines (the two reference lines participating in the comparison), or the error between the prediction values obtained by predicting the template area with the reconstruction values of the two reference lines as references.

[0106] The following uses one example to illustrate the comparison process. For the convenience of explanation, the reference lines outside the template area are numbered in ascending order of the distance from the current block, with the reference line closest to the current block being called the first reference line outside the template area, abbreviated as the first reference line, the reference line next closest to the current block being called the second reference line outside the template area, abbreviated as the second reference line, and so on by analogy. Further, assume that the number of reference lines from the template area to the upper boundary of the CTU is greater than 12.

[0107] The comparison process is as follows. After selecting the first reference line outside the template area, compare the error between the first reference line and its adjacent line, that is, the second reference line. If the error is smaller than the set threshold, continue to compare the error between the first reference line and the third reference line. If the error is greater than the set threshold, select the third reference line as the newly selected line. Then, compare the error between the third reference line and its adjacent line (i.e., the fourth reference line). If the error is greater than the set threshold, select the fourth reference line as the newly selected line. Then, compare the error between the fourth reference line and its adjacent line (i.e., the fifth reference line), and continue by analogy until the comparison of the twelfth reference line is completed. If the error between the twelfth reference line and the most recently selected reference line is greater than the set threshold, select the twelfth reference line to participate in the sorting; otherwise, do not select the twelfth reference line.

[0108] When using the above four methods, it can be seen that the MRL index list sorting method of the embodiments of the present disclosure is not limited to the sorting itself, and by selecting the reference lines participating in the sorting, it is also possible to select the reference lines whose indexes are embedded in the MRL index list. This is advantageous for finding the optimal reference lines and improving the efficiency of video coding.

[0109] However, the present disclosure may sort using the specified reference lines. In the fifth method above, the reference lines participating in the sorting are the specified N reference lines from the N' reference lines closest to the current block outside the template region. For example, the reference lines with specified indexes 1, 3, 5, 7, 12. In this case, it is equivalent to sorting the reference lines in the set MRL index list, and no new reference lines are introduced.

[0110] In one example of this embodiment, the N' reference lines closest to the current block outside the template region are reference lines that do not exceed the upper boundary of the coding tree unit CTU where the current block is located.

[0111] In one example of this embodiment, N set when the current block uses the template-based intra mode to derive TIMD max and N set when the current block does not use TIMD max are both 3 or more. In one example, N set when the current block uses TIMD max is 12, and when the current block does not use TIMD, N max may also be set to 12. Regarding the specific value of N max the present disclosure is not limited, but for example, values greater than 15 such as 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, and 18, 24, 36 may be used.

[0112] In an embodiment of the present disclosure, a video decoding method is further provided. As shown in FIG. 7, the method includes the following steps.

[0113] Step 210, decode the intra prediction related syntax elements of the current block in the code stream, and if the current block permits the use of multi-reference lines MRL, continue to decode the multi-reference line indexes of the current block.

[0114] As described above, the multi-reference line index is used to indicate the position of the index of the reference lines selected by the current block in the MRL index list.

[0115] Step 220, when it is determined to permit sorting of the MRL index list of the current block according to the decoding result, sort the MRL index list of the current block according to the method described in any embodiment of the present disclosure, and obtain an MRL index list sorted based on the current block.

[0116] Step 230, determine the reference lines selected by the current block according to the multi-reference line index and the MRL index list sorted based on the current block, and perform intra prediction on the current block according to the reference lines selected by the current block.

[0117] In this embodiment, in the decoding process, when it is determined to permit sorting of the MRL index list of the current block, first, sort the MRL index list of the current block according to the method of the above embodiment of the present disclosure to obtain an MRL index list sorted based on the current block, and then determine the reference lines selected by the current block according to the multi-reference line index obtained by decoding and the MRL index list sorted based on the current block. Further, perform intra prediction on the current block according to the reference lines selected by the current block. This embodiment uses the MRL index list sorted based on the current block to decode the multi-reference line index, so that the reference lines selected by the current block are likely to be located at positions closer to the head of the MRL index list, thereby saving coding overhead and improving coding performance.

[0118] In an exemplary embodiment of the present disclosure, when decoding a syntax element related to intra prediction of a current block in a code stream, by decoding the intra prediction mode selected by the current block, it is used to sort the MRL index list of the current block. According to the reference line selected by the current block, when performing intra prediction on the current block, intra prediction is performed on the current block according to the reference line selected by the current block and the intra prediction mode selected by the current block.

[0119] In an exemplary embodiment of the present disclosure, if the method determines not to permit sorting of the MRL index list of the current block according to the decoding result, according to the multi-reference line index and the set MRL index list, determine the reference line selected by the current block, and further include the step of performing intra prediction on the current block according to the reference line selected by the current block. This embodiment determines the reference line pointed to by the multi-reference line index using the set MRL index list when it is determined not to permit sorting of the MRL index list of the current block.

[0120] In one example of this embodiment, when it is determined that the current block does not use TIMD, the set MRL index list is {0, 1, 3, 5, 7, 12}, and when it is determined that the current block uses TIMD, the set MRL index list is {0, 1, 3} or {0, 1, 2}.

[0121] Regarding whether to permit sorting of the MRL index list of the current block, it may be determined according to the decoding of the multi-reference line index, or may be directly represented using a flag.

[0122] In an exemplary embodiment of the present disclosure, if the method determines to permit sorting of the MRL index list of the current block, it further includes the step of setting a flag indicating permission to sort the MRL index list of the current block to true.

[0123] In another exemplary embodiment of the present disclosure, any one or more of the following conditions are sufficient conditions that do not permit sorting of the MRL index list of the current block.

[0124] The current block does not use a multi-reference line MRL.

[0125] The multi-reference line index indicates a position that does not participate in sorting in the MRL index list, for example, the first position.

[0126] The current frame is a non-luminance frame.

[0127] The current block is located at the upper boundary of the coded tree unit CTU.

[0128] The intra prediction mode selected by the current block is a specified mode, where the specified mode includes any one or more of a planar mode, a TIMD fusion mode in which one mode is a planar mode, an average mode, a horizontal angle mode, a vertical angle mode, and a wide angle mode.

[0129] The size of the current block does not satisfy the set conditions.

[0130] The aspect ratio of the current block does not satisfy the set conditions.

[0131] If none of the conditions that do not permit sorting of the MRL index list of the current block (not limited to the conditions exemplified above) are satisfied, it is determined to permit sorting of the MRL index list of the current block.

[0132] In one example, either the case where the current block uses TIMD or the case where the current block does not use TIMD may be a sufficient condition for not permitting sorting of the MRL index list of the current block. That is, sorting of the MRL index list is permitted only when the current block uses TIMD or when the current block does not use TIMD.

[0133] In an exemplary embodiment of the present disclosure, when the current block uses MRL, the method further includes decoding some syntax elements of the transform modes that can be used simultaneously with the MRL of the current block and skipping some syntax elements of the transform modes that cannot be used simultaneously with the MRL.

[0134] In this embodiment, some transform modes may be skipped when the current block uses MRL. The MRL mode is located relatively close to the rear in all intra prediction modes. For example, it is necessary to determine whether it is MRL after determining that the current mode is not DIMD, BDPCM, MIP, or TIMD. In VVC, the residual signal may use primary transformation and secondary transformation to facilitate encoding of the transformed residual. Here, the primary transformation includes DCT2 transformation and DST7. On the other hand, the secondary transformation further transforms the residual after the primary transformation using DCT2 for the primary transformation. In ECM, the mode of the primary transformation is also further extended. Considering that additional bits are required to represent when different modes are selected, resulting in an overhead of the codeword, and at the same time, the encoding time also increases when the encoding end traverses these transform modes, when the intra prediction selects the multi-reference line mode, its primary transformation may be limited to using only the DCT2 mode and using the secondary transformation, or may be limited to using only the primary transformation, or may be limited to using only the DCT2 transformation. In this way, when encoding the index of the corresponding transform mode, the encoding of the transform mode can be skipped according to whether the MRL mode is selected.

[0135] In an exemplary embodiment of the present disclosure, when the current block uses the MRL, the method further includes allowing decoding of syntax elements related to the second MPM in the process of decoding the intra prediction mode selected by the current block. This embodiment allows the simultaneous use of the MRL and the second MPM mode.

[0136] In an exemplary embodiment of the present disclosure, the step of decoding the multi-reference line index of the current block includes decoding the multi-reference line index of the current block according to different MRL index lists when the current block uses the TIMD and when the current block does not use the TIMD.

[0137] In an embodiment of the present disclosure, a video encoding method is further provided. As shown in FIG. 8, the method includes the following steps.

[0138] Step 310, perform intra prediction on the current block and determine the reference line selected by the current block.

[0139] Step 320, if it is determined to allow sorting of the MRL index list of the current block, Embodiment of the present application sort the MRL index list of the current block according to the method described in any of the above, and obtain an MRL index list sorted based on the current block.

[0140] Step 330, determine and encode the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the MRL index list sorted based on the current block.

[0141] In this embodiment, when it is determined to permit sorting of the MRL index list of the current block in the encoding process, first, according to the method of the above embodiment of the present disclosure, the MRL index list of the current block is sorted to obtain the MRL index sorted based on the current block, and then, according to the position of the index of the reference line selected by the current block in the MRL index list sorted based on the current block, the multi-reference line index of the current block is determined and encoded. This embodiment uses the MRL index list sorted based on the current block to encode the multi-reference line index, so that the reference line selected by the current block is likely to be located closer to the head of the MRL index list, thereby saving the encoding overhead of the multi-reference line index and improving the encoding performance.

[0142] In an exemplary embodiment of the present disclosure, before sorting the MRL index list of the current block, the method further includes the step of determining the intra prediction mode selected by the current block.

[0143] In an exemplary embodiment of the present disclosure, when it is determined not to permit sorting of the MRL index list of the current block, the method further includes the step of determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the set MRL index. In this embodiment, when it is determined not to permit sorting of the MRL index list of the current block, the position of the selected reference line in the set MRL index list is used to determine the multi-reference line index of the current block.

[0144] In an exemplary embodiment of the present disclosure, when it is determined to permit sorting of the MRL index list of the current block, the method further includes the step of setting to true a flag indicating permission to sort the MRL index list of the current block. In this embodiment, whether to permit sorting of the MRL index list of the current block is directly represented by using one flag.

[0145] In an exemplary embodiment of the present disclosure, any one or more of the following conditions are sufficient conditions for not permitting sorting of the MRL index list of the current block.

[0146] The current block does not use the MRL.

[0147] The reference line selected by the current block is located in the template area of the current block.

[0148] The current frame is a non-luminance frame.

[0149] The current block is located at the upper boundary of the coding tree unit CTU.

[0150] The intra prediction mode selected by the current block is a specified mode, and the specified mode includes any one or more of a planar mode, a TIMD fusion mode in which one mode is a planar mode, an average mode, a horizontal angle mode, a vertical angle mode, and a wide angle mode.

[0151] The size of the current block does not meet the set conditions.

[0152] The aspect ratio of the current block does not meet the set conditions.

[0153] The current block is either a case where TIMD is used or a case where TIMD is not used.

[0154] If none of the conditions that do not permit sorting of the MRL index list of the current block (not limited to the conditions exemplified above) are satisfied, determine to permit sorting of the MRL index list of the current block.

[0155] In the exemplary embodiments of the present disclosure, different MRL index lists are used when the current block uses TIMD and when the current block does not use TIMD.

[0156] The step of determining and encoding the multi-reference line index of the current block includes the following steps.

[0157] When the current block uses TIMD, determine and encode the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the first MRL index list.

[0158] When the current block does not use TIMD, determine and encode the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the second MRL index list.

[0159] Here, the length of the second MRL index list is greater than or equal to the length of the first MRL index list.

[0160] In the exemplary embodiments of the present disclosure, when the current block uses MRL, the method further includes the step of encoding some syntax elements of the conversion mode that can be used simultaneously with the MRL of the current block and skipping the encoding of some conversion modes that cannot be used simultaneously with the MRL.

[0161] In an exemplary embodiment of the present disclosure, when the current block uses the MRL and the mode selected by the current block is a non-TIMD mode, after encoding the multi-reference line index, the method continues to encode the intra prediction mode selected by the current block, and further includes the step of permitting encoding of syntax elements related to the second MPM in the process of encoding the intra prediction mode selected by the current block.

[0162] In one embodiment of the present disclosure, a code stream is further provided. Here, the code stream includes the multi-reference line index of the current block, and the multi-reference line index is used to indicate the position of the index of the reference line selected by the current block in the MRL index list. The MRL index list is a sorted MRL index list based on the current block or a set MRL index list. The code stream of this embodiment may be generated by the video encoding method described in any embodiment of the present disclosure.

[0163] In one embodiment of the present disclosure, a multi-reference line index list sorting device is further provided. As shown in FIG. 9, the device includes a processor 71 and a memory 73 storing a computer program. When the processor 71 executes the computer program, it can implement the multi-reference line index list sorting method described in any embodiment of the present disclosure.

[0164] In one embodiment of the present disclosure, a video decoding device is further provided. As shown in FIG. 9, the device includes a processor and a memory storing a computer program. When the processor executes the computer program, it can implement the video decoding method described in any embodiment of the present disclosure.

[0165] In one embodiment of the present disclosure, a video encoding device is further provided. As shown in FIG. 9, the device includes a processor and a memory storing a computer program. When the processor executes the computer program, the video encoding method described in any embodiment of the present disclosure can be realized.

[0166] The processor in the above embodiment of the present disclosure may be a general-purpose processor including a Central Processing Unit (CPU for short), a Network Processor (NP for short), a microprocessor, etc., or other ordinary processors. The processor may further be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), discrete logic or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware assemblies, or a combination of the above devices. That is, the processor in the above embodiment may be any processing device or combination of devices that realizes each method, step, and logical block diagram disclosed in the embodiments of the present invention. When implementing the embodiments of the present disclosure partially in software, the software instructions used in the software may be stored in a suitable non-volatile computer-readable storage medium, and the instructions may be executed in hardware using one or more processors to implement the method of the embodiments of the present disclosure.

[0167] In one embodiment of the present disclosure, a video encoding / decoding system is further provided. The system includes a video encoding device described in any embodiment of the present disclosure and a video decoding device described in any embodiment of the present disclosure.

[0168] In one embodiment of the present disclosure, a non-volatile computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the multi-reference row index list sorting method described in any embodiment of the present disclosure is realized, or the video decoding method described in any embodiment of the present disclosure is realized, or the video encoding method described in any embodiment of the present disclosure is realized.

[0169] In one embodiment of the present disclosure, a video decoding method is further provided. In the decoding process, the multi-reference row index list of the current block is sorted. In this embodiment, sorting based on the template area is an operation that both the encoder and the decoder need to perform. It is used for the MRL index list, represents the position of the selected reference row in the MRL index list by the multi-reference row index, combines with the MRL index list sorted based on the current block, commonly determines the reference row selected by the current block, and uses the determined reference row and the selected intra prediction mode (conventional prediction mode) for prediction.

[0170] In this embodiment, a specific example is used to explain how to sort the MRL index list using the sum of absolute differences (SAD) between the predicted value (pred) and the reconstructed value (reco) in the template area. (x, -1) and (-1, y) are the coordinates relative to the upper left corner (0, 0) position of the current block respectively. In this embodiment, for the template area of one row and one column of the current block, refer to the cross-hatched area shown in FIG. 10. Six reference rows outside the template area of the current block are shown in the figure.

[0171] The method of this embodiment includes the following steps.

[0172] The first step

[0173] The decoder analyzes the intra prediction mode, the decoder analyzes the MRL-related syntax elements, confirms whether the current block uses or does not use MRL, and also decodes the multi-reference line index of the current block.

[0174] The syntax table related to this embodiment is as shown in the following table. JPEG2025521742000042.jpg205168 JPEG2025521742000043.jpg193169 JPEG2025521742000044.jpg232170 JPEG2025521742000045.jpg153170

[0175] In the table, intra_luma_mrl_tab_idx defines the multi-reference line index when the current block does not use TIMD (i.e.,!intra_timd_flag), indicates the position of the index of the reference line selected by the current block in the sorted MRL index list, and the range of its value is from 0 to 5. intra_timd_luma_ref_idx defines the multi-reference line index when the current block uses TIMD, indicates the position of the index of the reference line selected by the current block in the set MRL index list, and the range of its value is from 0 to 2. For the multi-reference line indexes in both cases, truncation binary coding and decoding based on the context model may be used.

[0176] In this example, when the current block uses TIMD, the set MRL index list is not sorted. However, in other examples, when the current block uses TIMD, the MRL index list of the current block may be sorted. When sorting the MRL index list when the current block uses TIMD, only the reference line sorting embedded in the set MRL index list where the index is set may be sorted, or more reference lines may be introduced and, for example, the same reference lines as when the current block does not use TIMD may be included in the sorting. In this case, instead of intra_timd_luma_ref_idx, a syntax element similar to one intra_luma_mrl_tab_idx may be used.

[0177] The timd in the tables in this specification is the same concept as TIMD in the text, and the same applies to other concepts.

[0178] Second step

[0179] After the decoding stage is completed and before predicting the current block, if the multi-reference line index of the current block is not 0, that is, if the index of the selected reference line is not 0, instead of using the set MRL index list, it is necessary to sort the MRL index list of the current block. The sorting is performed only on the extended reference lines (in this embodiment, reference lines with non-zero indices).

[0180] The sorting process is as follows.

[0181] First, determine the maximum number of extended reference lines permitted at the position of the current block in the image. In principle, the upper reference lines available to the current block shall not exceed the upper CTU boundary. In ECM, the CTU may be up to 256×256. When the vertical coordinate y at the position of the upper left corner of the current block in the image has a remainder when divided by 256, the obtained remainder is the number of reference lines within the upper CTU boundary. At the same time, when setting the MRL to be able to use a maximum of 13 reference lines (reference line 0~12) in the non-TIMD mode, the reference lines that can participate in the sorting are reference line 1~N, that is, the reference lines with indexes 1~N. Here, N = min(12, (x%256)-1)). In other embodiments, if the hardware supports it, the reference lines used may also exceed the CTU boundary.

[0182] The N reference lines of reference line 1~N are outside the template area of the row where reference line 0 is located. For each of these reference lines, the template area is predicted using the selected intra prediction mode. Since the template area of reference line 0 has been reconstructed, in this case, the SAD may be obtained according to the reconstructed value and the predicted value obtained by prediction. Sort the N SAD values in ascending order, and embed the indexes of the M reference lines with the smallest SAD calculated correspondingly into the MRL index list.

[0183] Specifically, when the current block does not use TIMD, the MRL index list has a length of 6, and when TIMD is used, the length is 3. The filled values set for these two lists are {0, 1, 3, 5, 7, 12} and {0, 1, 3} respectively, representing the reference lines with indexes 0, 1, 3, 5, 7, 12 and the reference lines with indexes 0, 1, 3 respectively. The sorting in this embodiment is to sort reference line 1~N, so the index 0 embedded in the first position in the MRL index list never changes, and only the extended reference lines after sorting are embedded in the subsequent positions. When the current block does not use TIMD, a maximum of 5 indexes can be embedded, and when the current block uses TIMD, a maximum of 2 indexes can be embedded.

[0184] In the process of sorting, there may be a possibility that two adjacent reference lines to be sorted have similar reconstruction values. In this case, the results predicted using these two lines are similar. In this case, redundancy may be removed by skipping one of these two lines, and this case should be considered. Here, there may be two skipping methods.

[0185] Method 1. Calculate the SAD value between the reconstruction values at the corresponding positions of two adjacent reference lines. If the SAD value is smaller than a preset threshold, it is considered that the similarity of these two reference lines is too large and one of the reference lines should be skipped. When the adjacent reference line is skipped, the selected reference line continues to compare with the next closest reference line for the threshold.

[0186] Method 2. Predict the template area using two reference lines respectively, calculate the SAD value between the prediction results in the template area. If the SAD value is smaller than a preset threshold, it is considered that the similarity of these two lines is too large and one of the lines should be skipped.

[0187] The above skipping method may skip any one of these two lines. However, usually considering that it is preferable to use the reference line closer to the current block as the reference line, in this embodiment, the farther reference line is skipped.

[0188] In the ECM, the MRL, TIMD, and MPM technologies may be used simultaneously. In the case of TIMD, the determined intra prediction mode may be the TIMD fusion mode. However, since the common effect of MRL and the Planar mode usually does not bring additional performance improvement, when MRL is selected, if the intra prediction mode is Planar or one of the modes is the TIMD fusion mode with Planar, sorting may not be necessary. Except for the Planar mode, sorting may not be necessary for some specific angles or modes. For example, sorting is not performed in the DC mode, sorting is not performed at wide angles, sorting is not performed in the horizontal and vertical angle modes, etc. The wide angle mode may be equivalent to not sorting the rectangular prediction block.

[0189] When not sorting, no SAD-based resorting is performed on the reference lines in these lists, and the set initial reference line list, for example, {1, 3, 5, 7, 12}, is directly used.

[0190] This embodiment may skip some conversion modes. The current MRL is encoded at a relatively rear position in all intra modes. In the ECM, before determining whether it is the MRL, it is necessary to determine that the current mode is not DIMD, BDPCM, MIP, or TIMD. Considering that more identifier bits are needed to identify the primary conversion and the secondary conversion in the subsequent conversion mode encoding, in order to reduce the encoding overhead in this regard, the MRL may be restricted to be used only in combination with some conversion modes. For example, it may be used only with the primary conversion of DCT2, or it may be used only with the primary conversion and the secondary conversion of DCT2. For example, under the primary conversion restriction, if the MRL is restricted to be used only jointly with DCT2, it should be checked whether the MRL is selected to decode the syntax element of the conversion mode. If it is selected, the primary conversion mode should not be decoded.

[0191] The corresponding syntax is as follows. JPEG2025521742000046.jpg207170 JPEG2025521742000047.jpg236170 JPEG2025521742000048.jpg33170

[0192] In the column described as "intra_luma_ref_tab_idx != 0 &&" in the above table, by defining several conditions, it is determined whether to decode mts_idx (the conversion mode index selected by the primary conversion), among which includes that the multi-reference line index is not 0.

[0193] In another example, under the primary conversion, if the MRL is restricted not to be used with the secondary conversion, it should be checked whether the MRL is selected to decode the syntax element of the conversion mode. If it is selected, the secondary conversion mode should not be decoded. The corresponding syntax is as follows. JPEG2025521742000049.jpg237170 JPEG2025521742000050.jpg237170

[0194] In the column described as "intra_luma_ref_tab_idx != 0 &&" in the above table, by defining several conditions, it is determined whether to decode lfnst_idx (the transform mode index selected by the secondary transform), among which it includes that the multi-reference line index is not 0.

[0195] The third step

[0196] According to the re-sorted multi-reference line index list and the list index obtained by decoding, the reference line is determined and the current block is predicted.

[0197] In this embodiment, when intra prediction is sorted using the multi-reference line index list, by using the error between the predicted value and the reconstructed value of the template area and sorting in ascending order, a multi-reference line index list sorted based on the current block is obtained, whereby the reference line selected for prediction is more likely to be arranged at a position earlier than the multi-reference line index list, and the coding overhead can be saved. In this embodiment, in the process of sorting, by setting a threshold, the reference lines to be sorted that may overlap can be skipped, the redundancy can be reduced, and the sorting efficiency can be improved. In this embodiment, the reference line where the template area is located and the reference lines smaller than the reference line cannot participate in the sorting and can maintain their original positions in the candidate list.

[0198] Based on this embodiment, sortable reference lines may be added. In this embodiment, to participate in the sorting, up to 12 extended reference lines are used. In fact, when complexity is not considered, by removing this limit, the reference lines for sorting may be reference lines other than the reference lines exceeding 12 lines, whereby better coding and decoding performance can be obtained.

[0199] Furthermore, the size of the template area may be increased, and the template area is not limited to the reference line closest to the current block and may continue to expand. For example, it expands to the right of the upper template area, below the left template area, and the expansion of the gray intersecting part at the upper left corner. It may continue to expand above the upper template area and to the left of the left template area. As a result, a larger area of reference lines is used. For example, if the area of the further used reference line 1 is used as the template area, reference line 1 itself cannot participate in the sorting.

[0200] In the ECM-4.0 reference software, using the method described in this embodiment to re-sort the MRL index list when the current block does not use TIMD can further bring about performance improvements as follows on the reference software. JPEG2025521742000051.jpg55117

[0201] Using the above method of this embodiment to sort the MRL index lists both when the current block uses TIMD and when it does not use TIMD can further bring about performance improvements as follows on the reference software. JPEG2025521742000052.jpg55118

[0202] In the above test, when the current block uses TIMD, reference lines 1 to 12 are also used to participate in the sorting, and only the two reference lines line_a and line_b with the smallest SAD among them are retained and embedded in the sorted MRL index list: {0, line_a, line_b}.

[0203] The parameters in the table are as follows.

[0204] EncT: Encoding Time, which is the symbolization time. 10X% means that after integrating the reference line sorting technology, the encoding time is 10X% compared to before integration, which means that the encoding time increases by X%.

[0205] DecT: Decoding Time, which is the decoding time. 10X% means that after integrating the reference line sorting technology, the decoding time is 10X% compared to before integration, which means that the decoding time increases by X%.

[0206] ClassA1 and ClassA2 are test video sequences with a resolution of 3840×2160, ClassB is a test sequence with a resolution of 1920×1080, ClassC is 832×480, ClassD is 416×240, ClassE is 1280×720, and ClassF is screen content sequences with several different resolutions.

[0207] All intra represents the test configuration of the all-intra frame structure. In the first aspect, a multi-reference line index list sorting method is provided, the method comprising: predicting each of the template regions according to at least some of the reference lines located outside the template region of the current block, and calculating an error between a reconstructed value of the template region and a predicted value obtained by prediction; embedding indexes of the reference lines corresponding to the errors into a multi-reference line MRL index list of the current block in ascending order of the errors from the smallest to the largest, to obtain an MRL index list sorted based on the current block; and including. In the second aspect, according to the method of the first aspect, the step of predicting each template region according to at least some of the reference lines located outside the template region of the current block comprises: for each of the at least some of the reference lines, predicting the template region by using a reconstructed value of the reference line and an intra prediction mode selected by the current block. In the third aspect, according to the method of the first aspect, the step of embedding the index of the reference line corresponding to the error into the MRL index list of the current block means embedding the index of the reference line corresponding to the error 1 starting from the K 2 -th position of the MRL index list position by position, and embedding up to the K 1 ≦K 2 -th position at most, where 2 ≤ K ≤ M, and M is the length of the MRL index list. In the fourth aspect, according to the method of the third aspect, the method further comprises: 1 embedding indexes of the reference lines where the template region of the current block is located from the first position not participating in the sorting to the (K -1)-th position in the MRL index list sorted based on the current block, wherein the index of the reference line embedded in the first position is 0. In the fifth aspect, according to the method of the first aspect, the step of embedding the index of the reference line corresponding to the error into the MRL index list of the current block means embedding the index of the reference line corresponding to the error 2 to embed up to the K-th position, where 1 ≤ K 2 ≤ M, and M is the length of the MRL index list. In the sixth aspect, according to the method of the third or fifth aspect, the method K 2 when K < M, from the (K + 1)-th position to the M-th position of the MRL index list, in the order from the closer to the farther distance from the current block, for each position, 2 embed the indexes of M - K reference lines, where the M - K 2reference lines are located outside the template region, specified by the system, and have indexes different from the indexes embedded in the MRL index list. 2 The method further includes the step of In the seventh aspect, according to the method of the first aspect, at least some of the reference lines located outside the template region of the current block are the N' reference lines closest to the current block outside the template region, or the odd-numbered lines among the N' reference lines closest to the current block outside the template region, or the even-numbered lines among the N' reference lines closest to the current block outside the template region, or N reference lines selected by comparing the errors between adjacent lines among the N' reference lines closest to the current block outside the template region, where N ≤ N', or N specified reference lines among the N' reference lines closest to the current block outside the template region, where N' ≤ N max and N max is the maximum number of reference lines allowed to participate in the sorting. In the eighth aspect, according to the method of the seventh aspect, the N' reference lines closest to the current block outside the template region are reference lines that do not exceed the upper boundary of the coding tree unit CTU where the current block is located, N set when the current block uses an intra mode based on a template to derive TIMD, and max N set when the current block does not use TIMD max are both 3 or more. In the ninth aspect, according to the method of the seventh aspect, the step of selecting N reference lines by comparing the errors between adjacent lines Select the first reference line closest to the current block outside the template area as the selected first reference line, compare the newly selected reference line and its adjacent reference lines one by one, and select one adjacent reference line that is closest to the reference line and the error between the two reference lines is greater than the set threshold as the newly selected reference line. Loop like this until N' reference lines are compared and selected. The error refers to the error between the reconstructed values of two reference lines, or refers to the error between the predicted values obtained by predicting the template area with the reconstructed values of two reference lines as references, and use this method. In item 10, according to the method of item 1 or item 9, the error is represented by the absolute error and SAD, or represented by the absolute value of error conversion and SATD. In item 11, according to the method of item 1, The template area of the current block is set in the reference line closest to the current block, or The template area of the current block is set in a plurality of reference lines closest to the current block. In item 12, a video decoding method is provided, and the method is Decoding the intra prediction related syntax elements of the current block in the code stream, and continuing to decode the multi-reference line index of the current block when the current block permits the use of multi-reference line MRL, where the multi-reference line index is used to indicate the position in the MRL index list of the indexes of the reference lines selected by the current block. When it is determined to permit sorting of the MRL index list of the current block according to the decoding result, sort the MRL index list of the current block according to the method described in any one of items 1 to 11 to obtain the MRL index list sorted based on the current block. Determine the reference lines selected by the current block according to the multi-reference line index and the MRL index list sorted based on the current block, and perform intra prediction on the current block according to the reference lines selected by the current block. including. In item 13, according to the method of item 12, the method is If it is determined not to permit sorting of the MRL index list of the current block according to the decryption result, according to the multi-reference line index and the set MRL index list, determine the reference line selected by the current block, and further include the step of performing intra prediction on the current block according to the reference line selected by the current block. In item 14, according to the method of item 12, a sufficient condition for permitting sorting of the MRL index list of the current block includes setting to true a flag indicating permission to sort the MRL index list of the current block. In item 15, according to the method of item 13, the condition that the current block does not use the multi-reference line MRL, the condition that the multi-reference line index indicates a position not participating in sorting in the MRL index list, the condition that the current frame is a non-luminance frame, the condition that the current block is located at the upper boundary of the coded tree unit CTU, the condition that the intra prediction mode selected by the current block is a specified mode, and the specified mode includes any one or more of a planar mode, a TIMD fusion mode in which one mode is a planar mode, an average mode, a horizontal angle mode, a vertical angle mode, and a wide angle mode, the condition that the size of the current block does not meet the set conditions, the condition that the aspect ratio of the current block does not meet the set conditions, the condition that the current block is one of the cases where TIMD is used and the case where TIMD is not used, any one or more of the above conditions are used as sufficient conditions for not permitting sorting of the MRL index list of the current block. In item 16, according to the method of item 12, when the current block uses MRL, the method includes the step of decrypting the syntax elements of some transform modes that can be used simultaneously with the MRL of the current block and skipping the syntax elements of some transform modes that cannot be used simultaneously with the MRL, and / or the step of permitting decryption of the syntax elements related to the second most likely mode MPM in the process of decrypting the intra prediction mode selected by the current block, is further included. In item 17, according to the method of item 12, the step of decrypting the intra prediction related syntax elements of the current block in the code stream is including steps used to sort the MRL index list of the current block by decoding the intra prediction mode currently selected by the block The step of performing intra prediction on the current block according to the reference line selected by the current block is including the step of performing intra prediction on the current block according to the reference line selected by the current block and the intra prediction mode selected by the current block In item 18, according to the method of item 17, the step of decoding the multi-reference line index of the current block is including the step of decoding the multi-reference line index of the current block according to different MRL index lists when the current block uses TIMD and when the current block does not use TIMD In item 19, according to the method of item 13, when it is determined that the current block does not use TIMD, the set MRL index list is {0, 1, 3, 5, 7, 12}, and when it is determined that the current block uses TIMD, the set MRL index list is {0, 1, 3} or {0, 1, 2} In item 20, a video encoding method is provided, and the method is performing intra prediction on the current block and determining the reference line selected by the current block when it is determined to permit sorting of the MRL index list of the current block, sorting the MRL index list of the current block according to the method described in any one of items 1 to 11 to obtain a sorted MRL index list based on the current block determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the sorted MRL index list based on the current block including In item 21, according to the method of item 20, the method is when it is determined not to permit sorting of the MRL index list of the current block, further including the step of determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the set MRL index In item 22, according to the method of item 20, the method is If it is determined to permit sorting of the MRL index list of the current block, the method further includes the step of setting to true a flag indicating permission to sort the MRL index list of the current block. In claim 23, according to the method of claim 21, the condition that the current block does not use the multi-reference line MRL, the condition that the reference line selected by the current block is located in the template area of the current block, the condition that the current frame is a non-luminance frame, the condition that the current block is located at the upper boundary of the coded tree unit CTU, the condition that the intra prediction mode selected by the current block is a specified mode, where the specified mode includes any one or more of a planar mode, a TIMD fusion mode in which one mode is a planar mode, an average mode, a horizontal angle mode, a vertical angle mode, and a wide angle mode, the condition that the size of the current block does not meet the set condition, the condition that the aspect ratio of the current block does not meet the set condition, the condition that one of the case where the current block uses TIMD and the case where the current block does not use TIMD, if none of the conditions for not permitting sorting of the MRL index list of the current block are satisfied, the condition for determining to permit sorting of the MRL index list of the current block, any one or more of the above conditions are used as sufficient conditions for not permitting sorting of the MRL index list of the current block. In claim 24, according to the method of claim 20, the method the step of determining and encoding the multi-reference line index of the current block, when the current block uses TIMD, determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the first MRL index list, when the current block does not use TIMD, determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the second MRL index list, where the length of the second MRL index list is greater than or equal to the length of the first MRL index list, further includes. In claim 25, according to the method of claim 20, when the current block uses MRL, the method Further comprising the step of encoding some of the syntax elements of the conversion modes that can be used simultaneously with the MRL of the current block and skipping the encoding of some of the conversion modes that cannot be used simultaneously with the MRL. In claim 26, according to the method of claim 20, when the current block uses the MRL and the mode selected by the current block is a non-TIMD mode, the method is After encoding the multi-reference line index, continue to encode the intra prediction mode selected by the current block, and further comprising the step of permitting the encoding of the syntax elements related to the second MPM in the process of encoding the intra prediction mode selected by the current block. In claim 27, according to the method of claim 20, before sorting the MRL index list of the current block according to the method described in any one of claims 1 to 11, the method is Further comprising the step of determining the intra prediction mode selected by the current block. In claim 28, a code stream is provided, the code stream includes the multi-reference line index of the current block, the multi-reference line index is used to indicate the position of the index of the reference line selected by the current block in the MRL index list, and the MRL index list is a sorted MRL index list based on the current block or a set MRL index list. In claim 29, according to the method of claim 28, the code stream is generated by the video encoding method described in any one of claims 20 to 27. In claim 30, a multi-reference line index list sorting device is provided, comprising a processor and a memory storing a computer program, and when the processor executes the computer program, it realizes the multi-reference line index list sorting method described in any one of claims 1 to 11. In claim 31, a video decoding device is provided, comprising a processor and a memory storing a computer program, and when the processor executes the computer program, it realizes the video decoding method described in any one of claims 12 to 19. In claim 32, a video encoding device is provided, comprising a processor and a memory storing a computer program, and when the processor executes the computer program, it realizes the video encoding method according to any one of claims 20 to 27. In claim 33, a video encoding / decoding system is provided, comprising the video encoding device according to claim 32 and the video decoding device according to claim 31. In claim 34, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, it realizes the multi-reference row index list sorting method according to any one of claims 1 to 11, or realizes the video decoding method according to any one of claims 12 to 19, or realizes the video encoding method according to any one of claims 20 to 27.

[0208] In one or more of the above exemplary embodiments, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on a computer-readable medium as one or more instructions or codes, or may be transmitted via a computer-readable medium and executed by a processing unit based on hardware. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or may include a communication medium such as a computer program, any medium that facilitates transmission from one place to another according to a communication protocol. Thus, the computer-readable medium may typically correspond to a non-transitory tangible computer-readable storage medium, or a communication medium such as a signal or a carrier. The data storage medium may be any usable medium for storing and invoking the technical instructions, codes, and / or data structures described in the present disclosure by one or more computers or one or more processors. A computer program product may include a computer-readable medium.

[0209] For example, such a computer-readable storage medium may include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can store the desired program code in the form of instructions or data structures and is accessible by a computer. Also, any connection may be referred to as a computer-readable medium. For example, when transmitting instructions from a website, server, or other remote source using coaxial cables, optical fiber cables, twisted pairs, digital subscriber lines (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cables, optical fiber cables, twisted pairs, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other volatile media, but rather are directed to non-volatile tangible storage media. As used herein, magnetic disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, or Blu-ray disks, etc., where magnetic disks typically reproduce data magnetically and optical disks reproduce data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0210] For example, instructions may be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, as used herein, the term "processor" may refer to any of the foregoing structures, or any other structure suitable for implementing the techniques described herein. In some aspects, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated in a packaged codec. Further, the techniques may be implemented entirely in one or more circuits or logic elements.

[0211] The technical solutions of the embodiments of the present disclosure may be implemented in a variety of devices or apparatuses including a wireless mobile phone, an integrated circuit (IC), or a set of ICs (e.g., a chipset). In the embodiments of the present disclosure, various assemblies, modules, or units are described, emphasizing the functional aspects of the devices that implement the described techniques by configuration, but not necessarily implemented by different hardware units. Rather, as described above, the various units may be combined into codec hardware units, or provided by a combination of interoperable hardware units (including the one or more processors described above) and appropriate software and / or firmware.

Claims

1. Predicting each of the template regions according to at least some of the reference lines located outside the template region of the current block, and calculating an error between the reconstructed value of the template region and the predicted value obtained by prediction; Embedding the index of the reference line corresponding to the error into the multi-reference line MRL index list of the current block in ascending order from the smaller error to the larger error, and obtaining an MRL index list sorted based on the current block; including A multi-reference line index list sorting method.

2. The step of predicting each template region according to at least some of the reference lines located outside the template region of the current block includes, for each reference line among the at least some of the reference lines, using the reconstructed value of the reference line and the intra prediction mode selected by the current block to predict the template region. The method according to claim 1.

3. The step of embedding the index of the reference line corresponding to the error into the MRL index list of the current block means that the index of the reference line corresponding to the error is embedded position by position starting from the K １ -th position of the MRL index list, and is embedded up to the K ２ -th position at most, where 2 ≤ K １ ≤ K ２ ≤ M, and M is the length of the MRL index list The method according to claim 1.

4. In the MRL index list sorted based on the current block, from the first position that does not participate in the sorting to the K １ - 1st position, embedding the index of the reference line where the template area of the current block is located, and the index of the reference line embedded in the first position is 0, and further including the step of The method according to claim 3.

5. The step of embedding the index of the reference line corresponding to the error into the MRL index list of the current block means embedding the index of the reference line corresponding to the error position by position starting from the first position of the MRL index list, up to the maximum of K ２ th position, where 1 ≤ K ２ ≤ M, and M is the length of the MRL index list The method according to claim 1.

6. The method is K ２ In the case of <M>, from the (K + 1)-th position to the M-th position of the MRL index list, for each position, in the order from the nearer to the farther distance from the current block, embed the indices of M - K ２ reference lines, wherein the M - K ２ reference lines are located outside the template area, specified by the system, and have indices different from the indices embedded in the MRL index list, further including the step of ２ embedding. The method according to claim 3 or 5.

7. At least some of the reference lines located outside the template region of the current block are the N' reference lines closest to the current block outside the template region, or the odd-numbered lines among the N' reference lines closest to the current block outside the template region, or the even-numbered lines among the N' reference lines closest to the current block outside the template region, or N reference lines selected by comparing the errors between adjacent lines among the N' reference lines closest to the current block outside the template region, N ≤ N', or N specified reference lines among the N' reference lines closest to the current block outside the template region including N' ≤ N ｍａｘ where N ｍａｘ is the maximum number of reference lines that permit participation in the sorting The method according to claim 1.

8. The N' reference lines closest to the current block outside the template region are reference lines that do not exceed the upper boundary of the coding tree unit CTU where the current block is located. N that is set when the current block derives TIMD using the intramode based on the template ｍａｘ and N that is set when the current block does not use TIMD ｍａｘ are both 3 or more The method according to claim 7.

9. The step of selecting N reference lines by comparing the errors between adjacent lines is Select the first reference line closest to the current block outside the template area as the selected first reference line, compare the newly selected reference line with its adjacent reference line one by one, and select a reference line adjacent to the current reference line that is closest to the current reference line and has an error between the two reference lines greater than the set threshold as the newly selected reference line. Keep looping like this until N' reference lines are compared and selected. The error refers to the error between the reconstructed values of two reference lines, or using the reconstructed values of two reference lines as references, it refers to the error between the predicted values obtained by predicting the template area respectively. The method according to claim 7.

10. The error is represented by an absolute error and SAD, or represented by an error conversion absolute value and SATD. The method according to any one of claims 1 or 9.

11. The template area of the current block is set in the reference line closest to the current block, or The template area of the current block is set in a plurality of reference lines closest to the current block. The method according to claim 1.

12. Decoding the intra-prediction related syntax elements of the current block in the code stream, and when the current block permits the use of multi-reference lines MRL, continuously decoding the multi-reference line index of the current block, where the multi-reference line index is used to indicate the position in the MRL index list of the indexes of the reference lines selected by the current block. If it is determined according to the decoding result that sorting of the MRL index list of the current block is permitted, sort the MRL index list of the current block according to the method according to any one of claims 1 to 11 to obtain the MRL index list sorted based on the current block. Determine the reference line selected by the current block according to the multi-reference line index and the MRL index list sorted based on the current block, and perform intra-prediction on the current block according to the reference line selected by the current block. Including Video decoding method.

13. When the method determines not to permit sorting of the MRL index list of the current block according to the decryption result, according to the multi-reference row index and the set MRL index list, determine the reference row selected by the current block, and according to the reference row selected by the current block, further include the step of performing intra prediction on the current block. The method according to claim 12.

14. A sufficient condition for permitting sorting of the MRL index list of the current block includes setting to true a flag indicating permission to sort the MRL index list of the current block. The method according to claim 12.

15. The condition that the current block does not use the multi-reference row MRL, The condition that the multi-reference row index indicates a position not participating in sorting in the MRL index list, The condition that the current frame is a non-luminance frame, The condition that the current block is located at the upper boundary of the coded tree unit CTU, The condition that the intra prediction mode selected by the current block is a specified mode, where the specified mode includes any one or more of a planar mode, a TIMD fusion mode in which one mode is a planar mode, an average mode, a horizontal angle mode, a vertical angle mode, and a wide angle mode, The condition that the size of the current block does not meet the set condition, The condition that the aspect ratio of the current block does not meet the set condition, The condition that the current block is one of the cases where TIMD is used and the case where TIMD is not used, Any one or more of the above conditions are sufficient conditions for not permitting sorting of the MRL index list of the current block. The method according to claim 13.

16. When the current block uses MRL, the method includes: Decrypting some syntax elements of the conversion modes that can be used simultaneously with the MRL of the current block and skipping some syntax elements of the conversion modes that cannot be used simultaneously with the MRL, and / or In the process of decrypting the intra prediction mode selected by the current block, permitting the decryption of the syntax elements related to the second most possible mode MPM. Further included. The method according to claim 12.

17. The step of decoding the intra prediction related syntax elements of the current block in the code stream includes the step of sorting the MRL index list of the current block by decoding the intra prediction mode selected by the current block. The step of performing intra prediction on the current block according to the reference line selected by the current block includes the step of performing intra prediction on the current block according to the reference line selected by the current block and the intra prediction mode selected by the current block. The method according to claim 12.

18. The step of decoding the multi-reference line index of the current block includes the step of decoding the multi-reference line index of the current block according to different MRL index lists when the current block uses TIMD and when the current block does not use TIMD. The method according to claim 17.

19. When it is determined that the current block does not use TIMD, the set MRL index list is {0, 1, 3, 5, 7, 12}. When it is determined that the current block uses TIMD, the set MRL index list is {0, 1, 3} or {0, 1, 2}. The method according to claim 13.

20. The step of performing intra prediction on the current block and determining the reference line selected by the current block. When it is determined to permit sorting of the MRL index list of the current block, sort the MRL index list of the current block according to the method according to any one of claims 1 to 11 to obtain a sorted MRL index list based on the current block. Determine and encode the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the sorted MRL index list based on the current block. including Video encoding method.

21. When it is determined that sorting of the MRL index list of the current block is not permitted, the method further includes the step of determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the set MRL index. The method according to claim 20.

22. When it is determined to permit sorting of the MRL index list of the current block, the method further includes a step of setting to true a flag indicating permission to sort the MRL index list of the current block, The method according to claim 20.

23. The condition that the current block does not use a multi-reference line MRL, The condition that the reference line selected by the current block is located in the template area of the current block, The condition that the current frame is a non-luminance frame, The condition that the current block is located at the upper boundary of the coding tree unit CTU, The condition that the intra prediction mode selected by the current block is a specified mode, where the specified mode includes any one or more of a planar mode, a TIMD fusion mode in which one mode is a planar mode, an average mode, a horizontal angle mode, a vertical angle mode, and a wide angle mode, The condition that the size of the current block does not satisfy the set condition, The condition that the aspect ratio of the current block does not satisfy the set condition, The condition that the current block is either using TIMD or not using TIMD, When none of the conditions for not permitting sorting of the MRL index list of the current block are satisfied, the condition for determining to permit sorting of the MRL index list of the current block, Any one or more of the above conditions are used as sufficient conditions for not permitting sorting of the MRL index list of the current block, The method according to claim 21.

24. The method, The step of determining and encoding the multi-reference line index of the current block is, When the current block uses TIMD, determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the first MRL index list, When the current block does not use TIMD, determining and encoding the multi-reference line index of the current block according to the position of the index of the reference line selected by the current block in the second MRL index list, where the length of the second MRL index list is greater than or equal to the length of the first MRL index list, further includes the step of, The method according to claim 20.

25. When the current block uses an MRL, the method further includes encoding the syntax elements of some of the transformation modes that can be used simultaneously with the MRL of the current block and skipping the encoding of some of the transformation modes that cannot be used simultaneously with the MRL. The method according to claim 20. **Claim 26** When the current block uses an MRL and the mode selected by the current block is a non-TIMD mode, the method further includes encoding the multi-reference line index, then continuing to encode the intra prediction mode selected by the current block, and in the process of encoding the intra prediction mode selected by the current block, further including the step of permitting the encoding of the syntax elements related to the second MPM. The method according to claim 20. **Claim 27** According to the method according to any one of claims 1 to 11, before sorting the MRL index list of the current block, the method further includes the step of determining the intra prediction mode selected by the current block. The method according to claim 20. **Claim 28** A code stream, wherein the code stream includes the multi-reference line index of the current block, and the multi-reference line index is used to indicate the position of the index of the reference line selected by the current block in the MRL index list, and the MRL index list is a sorted MRL index list based on the current block or a set MRL index list. Code stream. **Claim 29** The code stream is generated by the video encoding method according to any one of claims 20 to 27. The code stream according to claim 28. **Claim 30** A multi-reference line index list sorting device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it realizes the multi-reference line index list sorting method according to any one of claims 1 to 11. Multi-reference line index list sorting device. **Claim 31** A video decoding device, comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, it realizes the video decoding method according to any one of claims 12 to 19. Video decoding device. Claim 32 A video encoding device comprising a processor and a memory storing a computer program, wherein when the processor executes the computer program, the video encoding method according to any one of claims 20 to 27 is realized. Video encoding device. Claim 33 A video encoding / decoding system comprising the video encoding device according to claim 32 and the video decoding device according to claim 31. Video encoding / decoding system. Claim 34 A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the multi-reference line index list sorting method according to any one of claims 1 to 11 is realized, or the video decoding method according to any one of claims 12 to 19 is realized, or the video encoding method according to any one of claims 20 to 27 is realized. Non-transitory computer-readable storage medium.

Images