Prediction processing system and prediction processing method

Abstract

The present application provides a prediction processing system and a prediction processing method. The present application provides a prediction processing system, comprising: processing circuitry configured to perform a first inter prediction operation on a first prediction block to generate a first inter prediction result, and further configured to perform a second inter prediction operation on a second prediction block during a first time period; and a reference data buffer for buffering reference data derived from the first inter prediction result; wherein the processing circuitry is further configured to extract the reference data from the reference data buffer, and perform a non-inter prediction operation during a second time period based on at least the reference data, wherein the second time period overlaps with the first time period. The prediction processing system and the prediction processing method of the present application can enhance system performance.

Description

[Technical Field]

[0001] This invention relates to video processing, and more specifically, to a prediction processing system and related prediction processing methods that use a reference data buffer to implement parallel non-inter and inter prediction. [Background Technology]

[0002] Traditional video codec standards typically employ block-based coding techniques to leverage spatial and temporal redundancy. For example, the basic approach is to divide the entire source image into multiple blocks, perform intra / inter-frame prediction on each block, transform the residuals of each block, and then quantize and entropy encode them. Furthermore, reconstructed images are generated during the coding loop to provide reference pixel data for coding subsequent blocks. For some video codec standards, loop filters can be used to enhance the image quality of the reconstructed frames. The video decoder performs the inverse operations of the video encoding operations performed by the video encoder. For example, a video decoder may have multiple processing circuits, such as entropy decoding circuits, intra-frame prediction circuits, inter-frame prediction circuits, inverse quantization circuits, inverse transform circuits, reconstruction circuits, and one or more loop filters. For intra / inter-frame prediction, a largest coding unit (LCU) / super block (SB) can be divided into prediction blocks that may include intra-frame prediction blocks and inter-frame prediction blocks. For an intra-frame prediction block, intra-frame prediction can refer to the intra / inter-frame prediction results of adjacent blocks. In other words, there is a data dependency between an intra-frame prediction block and an inter-frame prediction block. Therefore, an innovative prediction scheme is needed that can perform parallel non-inter-frame and inter-frame predictions to enhance system performance. [Summary of the Invention]

[0003] In view of the above, the present invention provides the following technical solution:

[0004] The present invention provides a prediction processing system, comprising: a processing circuit configured to perform a first inter-frame prediction operation on a first prediction block to generate a first inter-frame prediction result, and further configured to perform a second inter-frame prediction operation on a second prediction block during a first time period; and a reference data buffer for buffering reference data derived from the first inter-frame prediction result; wherein the processing circuit is further configured to extract reference data from the reference data buffer and perform a non-inter-frame prediction operation based on the reference data at least during a second time period, wherein the second time period overlaps with the first time period.

[0005] The present invention also provides a prediction processing method, comprising: performing a first inter-frame prediction operation on a first prediction block to generate a first inter-frame prediction result; caching reference data derived from the first inter-frame prediction result in a reference data buffer; performing a second inter-frame prediction operation on a second prediction block during a first time period; extracting reference data from the reference data buffer; and performing a non-inter-frame prediction operation based on the reference data at least during a second time period, wherein the second time period overlaps with the first time period.

[0006] The predictive processing system and predictive processing method of the present invention can enhance system performance. [Attached Image Description]

[0007] Embodiments of the invention are illustrated in conjunction with the accompanying drawings, which are included and form part of this specification, and together with the description, serve to explain the principles of the invention:

[0008] Figure 1 This is a schematic diagram of the first video processing device according to an embodiment of the present invention.

[0009] Figure 2 This is a flowchart illustrating an inter-frame prediction method according to an embodiment of the present invention.

[0010] Figure 3 This is a flowchart illustrating a non-inter-frame prediction method according to an embodiment of the present invention.

[0011] Figure 4 This is a diagram illustrating a predicted block obtained by segmenting a maximum codec unit (LCU) / superblock (SB) according to an embodiment of the present invention.

[0012] Figure 5 This is a diagram illustrating the adjacent data used for normal intra-frame prediction according to an embodiment of the present invention.

[0013] Figure 6 This is an illustration based on an embodiment of the present invention. Figure 4 The diagram shows the first prediction process of multiple prediction blocks in an LCU / SB.

[0014] Figure 7 This is an illustration based on an embodiment of the present invention. Figure 4 A diagram showing the second prediction process for multiple prediction blocks in an LCU / SB.

[0015] Figure 8 This is a diagram illustrating a prediction block obtained by segmenting two consecutive LCU / SB according to an embodiment of the present invention.

[0016] Figure 9 This describes an embodiment of the present invention. Figure 8 The diagram shows the prediction processing of multiple prediction blocks in the two LCU / SBs.

[0017] Figure 10 This is a diagram illustrating non-inter-frame reference data used by CfN mode at a first LCU size according to an embodiment of the present invention.

[0018] Figure 11 This is a diagram illustrating non-inter-frame reference data used by CfN mode at a second LCU size according to an embodiment of the present invention.

[0019] Figure 12 This is a diagram illustrating non-inter-frame reference data used by CfN mode at a third LCU size according to an embodiment of the present invention.

[0020] Figure 13 This is a diagram illustrating a second video processing apparatus according to an embodiment of the present invention.

[0021] Figure 14 This is an explanation of an embodiment of the present invention. Figure 13 The diagram shows the CIIP operation performed by the non-inter-frame prediction circuit.

[0022] Figure 15 This is a diagram illustrating a third video processing apparatus according to an embodiment of the present invention.

[0023] Figure 16 This is an illustration of an embodiment of the present invention. Figure 15 The diagram shows the CIIP operation performed by the non-inter-frame prediction circuit.

Detailed Implementation Methods

[0024] Numerous specific details are set forth in the following description. However, it should be understood that embodiments of the invention can be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification. However, those skilled in the art will understand that the invention can be practiced without such specific details. Those skilled in the art with the included description will be able to implement appropriate functionality without excessive experimentation.

[0025] The following description is the best intended mode for carrying out the invention. This description is intended to illustrate the general principles of the invention and should not be construed as limiting. The scope of the invention is best determined by referring to the appended claims.

[0026] Figure 1 This is a schematic diagram of a first video processing apparatus according to an embodiment of the present invention. The video processing apparatus 100 may be part of a video decoder and may include a syntax decoder 102, an inter-frame control buffer 104, a non-inter-frame control buffer 106, a reference image memory 108, a prediction processing system 110, a reconstruction circuit 112, and a reconstruction buffer 114. It should be noted that... Figure 1 Only circuitry relevant to the invention is shown. In practice, the video processing apparatus 100 may include additional circuitry to implement the specified functions. The prediction processing system 110 may include processing circuitry 120 and a reference data buffer 126. Processing circuitry 120 may include inter-frame prediction circuitry (labeled "inter-frame prediction") 122 and non-inter-frame prediction circuitry (labeled "non-inter-frame prediction") 124, wherein non-inter-frame prediction circuitry 124 may have a local buffer 125 for buffering non-inter-frame reference data (e.g., intra-frame reference data). Non-inter-frame prediction circuitry 124 may be used to process normal intra-frame prediction (NIP) mode, and / or may be used to process codec modes similar to intra-frame prediction in terms of codec order, such as copy from neighbor (CfN) mode. Hereinafter, the term "non-inter-frame prediction" may refer to NIP mode, or may refer to codec modes processed in the same way as intra-frame prediction in terms of codec order. Processing circuitry 120 is shown as having two prediction circuits, but the invention is not limited thereto. In some embodiments, the processing circuit 120 may have an inter-frame prediction circuit, an intra-frame prediction circuit, and other non-inter-frame prediction circuits. In some embodiments, the processing circuit 120 may have control logic (not shown) for managing the control flow to implement parallel non-inter-frame and inter-frame prediction.

[0027] Syntax decoder 102 is configured to parse the encoded bitstream BS to output control syntax elements for inter-frame prediction to inter-frame control buffer 104 and control syntax elements for non-inter-frame prediction to non-inter-frame control buffer 106. Control buffer 104 can trigger inter-frame prediction of blocks based on the control syntax elements parsed from the encoded bitstream BS. Non-inter-frame control buffer 106 can trigger non-inter-frame prediction of blocks based on the control syntax elements parsed from the encoded bitstream BS. Reference picture memory 108 may be a decoded picture buffer (DPB) for storing candidate reference pictures (decoded pictures) used for inter-frame prediction. Prediction processing system 110 is used to determine the prediction result of the current block, which may be an inter-frame prediction block or a non-inter-frame prediction block. Reconstruction circuit 112 is used to combine the prediction result of the current block (e.g., an inter-frame prediction block or a non-inter-frame prediction block) with the corresponding residual data of the current block to generate a reconstruction result of the current block, and output the reconstruction result of the current block to reconstruction buffer 114. Since this invention focuses on the prediction processing system 110, for the sake of brevity, further descriptions of the syntax decoder 102, inter-frame control buffer 104, non-inter-frame control buffer 106, reference image memory 108, reconstruction circuit 112, and reconstruction buffer 114 are omitted here.

[0028] Regarding the prediction processing system 110, the inter-frame prediction circuit 122 is arranged to perform a first inter-frame prediction operation on a first prediction block to generate a first inter-frame prediction result (e.g., an inter-frame prediction block found through inter-frame prediction), and is further arranged to perform a second inter-frame prediction operation on a second prediction block during a first time period; a reference data buffer 126 is used to buffer reference data derived from the first inter-frame prediction result; and a non-inter-frame prediction circuit 124 is used to obtain reference data from the reference data buffer 126 and perform a prediction operation (e.g., NIP mode prediction or CfN mode prediction) at least based on the reference data during a second time period. In some embodiments, the second time period overlaps with the first time period, and the start time of the first inter-frame prediction operation is earlier than the start time of the non-inter-frame prediction operation. With the help of the reference data buffer 126 for buffering reference data, which is the non-inter-frame reference data required for the non-inter-frame prediction operation, parallel inter-frame and non-inter-frame prediction can be achieved (e.g., parallel processing of non-inter-frame prediction of a non-inter-frame prediction block and inter-frame prediction of an inter-frame prediction block).

[0029] Figure 2 This is a flowchart illustrating an inter-frame prediction method according to an embodiment of the present invention. The inter-frame prediction method can be derived from... Figure 1 The inter-frame prediction circuit 122 shown employs a method where the execution of steps can be managed by control logic circuitry (not shown) in processing circuitry 120. As long as the results are substantially the same, it is not necessary to follow the steps outlined in the diagram. Figure 2The steps are executed in the exact order shown. In step 202, the inter-frame prediction circuit 122 is triggered to initiate inter-frame prediction for the current inter-frame prediction block. In step 204, the inter-frame prediction circuit 122 retrieves inter-frame reference data from the reference image memory 108, wherein the inter-frame reference data may include reconstructed data of one or more reference images (decoded images). In step 206, the inter-frame prediction circuit 122 checks whether the reference data buffer 126 is full. If it is determined that the reference data buffer 126 is full, the inter-frame prediction circuit 122 waits for available free space in the reference data buffer 126. If it is determined that the reference data buffer 126 is not full or is empty, the process continues to step 208. In step 208, the inter-frame prediction circuit 122 performs inter-frame prediction on the current inter-frame prediction block based on the reference data retrieved from the reference image memory 108 and produces the inter-frame prediction result for the current inter-frame prediction block. In step 210, the inter-frame prediction circuit 122 outputs the inter-frame prediction result determined for the current inter-frame prediction block (i.e., the inter-frame prediction block found through inter-frame prediction) to the reconstruction circuit 112, so that the non-inter-frame reference data derived from the inter-frame prediction result is stored in the reference data buffer 126. Specifically, the inter-frame prediction result and the corresponding residual data are combined to obtain the reconstruction result (i.e., the reconstruction block) and stored in the reconstruction buffer 114. In this embodiment, at least a portion (i.e., part or all) of at least one reconstruction result (reconstruction block) is stored in the reference data buffer 126 to serve as non-inter-frame reference data for later use. When the non-inter-frame prediction circuit 124 adopts a first non-inter-frame prediction type (e.g., NIP), the non-inter-frame reference data (i.e., intra-frame reference data) stored in the reference data buffer 126 may only include a portion of the reconstruction result determined for one inter-frame prediction block. In another case where the non-inter-frame prediction circuit 124 employs a second non-inter-frame prediction type (e.g., CfN), the non-inter-frame reference data stored in the reference data buffer 126 may include all reconstruction results determined for an inter-frame prediction block. In step 212, the inter-frame prediction circuit 122 determines whether there is a next inter-frame prediction block that has not yet been processed. If there is a next inter-frame prediction block, the process proceeds to step 202. If all inter-frame prediction blocks have been processed, the inter-frame prediction processing ends.

[0030] Figure 3 This is a flowchart illustrating a non-inter-frame prediction method according to an embodiment of the present invention. The non-inter-frame prediction method can be derived from... Figure 1 The non-inter-frame prediction circuit 124 shown employs steps whose execution can be managed by control logic circuitry (not shown) in processing circuitry 120. As long as the results are substantially the same, it is not necessary to follow... Figure 3The steps are executed in the exact order shown. In step 302, the non-inter-frame prediction circuit 124 is triggered to initiate intra-frame prediction for the current non-inter-frame prediction block. In step 304, the non-inter-frame prediction circuit 124 checks whether the reference data buffer 126 is ready to provide the required non-inter-frame reference data (which is reconstructed data derived from one or more inter-frame prediction results of one or more previously processed inter-frame prediction blocks). If it is determined that the non-inter-frame reference data is not ready in the reference data buffer 126, the non-inter-frame prediction circuit 124 waits for the required non-inter-frame reference data to be ready in the reference data buffer 126. If it is determined that the non-inter-frame reference data is available in the reference data buffer 126, the process proceeds to step 306. In step 306, the non-inter-frame prediction circuit 124 retrieves the non-inter-frame reference data from the reference data buffer 126. In step 308, the non-inter-frame prediction circuit 124 performs non-inter-frame prediction on the current non-inter-frame prediction block based at least on the non-inter-frame reference data retrieved from the reference data buffer 126. In step 310, the non-inter-frame prediction circuit 124 outputs the non-inter-frame prediction result determined for the current non-inter-frame prediction block (i.e., the non-inter-frame prediction block) to the reconstruction circuit 112, so that the reconstruction result (i.e., the reconstruction block) obtained by combining the non-inter-frame prediction result and the corresponding residual data is stored in the reconstruction buffer 114. Furthermore, the non-inter-frame prediction circuit 124 may have a local buffer 125 for buffering at least a portion (i.e., part or all) of the reconstruction result of the current non-inter-frame prediction block as non-inter-frame reference data for later use. In step 312, the non-inter-frame prediction circuit 124 determines whether there is a next non-inter-frame prediction block that has not yet been processed. If there is a next non-inter-frame prediction block, the process proceeds to step 302. If all non-inter-frame prediction blocks have been processed, the non-inter-frame prediction processing ends.

[0031] In this embodiment, the reference data buffer 126 is used to buffer non-inter-frame reference data, which includes at least a portion (i.e., part or all) of the reconstruction result (i.e., the reconstruction block) obtained by combining inter-frame prediction results (i.e., inter-frame prediction blocks) and corresponding residual data. Since the non-inter-frame prediction of the current non-inter-frame prediction block can obtain the required non-inter-frame reference data from the reference data buffer 126 (which is obtained from the reconstruction result derived from the inter-frame prediction results determined for the previous inter-frame prediction block), the inter-frame prediction circuit 122 can begin inter-frame prediction of unprocessed inter-frame prediction blocks without waiting for the non-inter-frame prediction of the current non-inter-frame prediction block to finish. By appropriately setting the buffer size of the reference data buffer 126, parallel inter-frame and non-inter-frame prediction can be achieved. A further description of the proposed parallel inter-frame and non-inter-frame prediction scheme is provided below with reference to the accompanying drawings.

[0032] When the non-inter-frame prediction circuit 124 employs NIP, the intra-frame reference data stored in the reference data buffer 126 may include only a portion of the reconstruction result determined for one inter-frame prediction block. (Please refer to...) Figure 5 refer to Figure 4 . Figure 4 This is a diagram illustrating a predicted block obtained by segmenting a maximum codec unit (LCU) / superblock (SB) according to an embodiment of the present invention. Figure 5 This is a diagram illustrating the adjacent data used in normal intra-frame prediction according to an embodiment of the present invention. For example... Figure 4 As shown, an LCU / SB 402 can be divided into multiple prediction blocks (PBs), including inter-frame prediction blocks PB0, PB3, PB4, and PB6, and intra-frame prediction blocks PB1, PB2, PB5, and PB7. Regarding... Figure 5 An intra-prediction block 502, as shown, can reference the top-adjacent reconstructed data and / or the left-adjacent reconstructed data for intra-prediction. That is, the top or left-adjacent decoded pixels in M ​​rows can serve as intra-prediction references. For example, when the intra-prediction mode under NIP is angle prediction mode, M can be equal to 1. As another example, when the intra-prediction mode under NIP is multi-reference line (MRL) prediction mode, M can be greater than 1. Some or all of the top and left-adjacent decoded pixels may belong to inter-prediction blocks. That is, a top-adjacent block of intra-prediction block 502 can be an inter-prediction block, and / or a left-adjacent block of intra-prediction block 502 can be an inter-prediction block. Figure 5 As shown, the reference data buffer 126 can be used to buffer adjacent data of the intra-prediction block 502 obtained from the top decoded pixel and / or left decoded pixel belonging to the inter-prediction block.

[0033] Please combine Figure 6 refer to Figure 4 . Figure 6 This is an illustration based on an embodiment of the present invention. Figure 4The diagram illustrates the first prediction process for multiple prediction blocks in an LCU / SB402. It is assumed that the reference data buffer 126 can store adjacent data (i.e., intra-frame reference data) obtained only from a single inter-prediction block. When the inter-prediction circuit 122 performs inter-prediction on inter-prediction block PB0, the reconstructed data of the left neighbor of intra-prediction block PB1 and the reconstructed data of the upper neighbor of intra-prediction block PB2 are obtained through reconstruction based on the inter-prediction result of inter-prediction block PB0 and stored in the reference data buffer 126 for later use. Since the reference data buffer 126 can store adjacent data (i.e., intra-frame reference data) obtained only from a single inter-prediction block, inter-prediction of the next inter-prediction block PB3 is not triggered until the reference data buffer 126 becomes empty. After the inter-prediction of inter-prediction block PB0 is completed, the reconstructed data of the left neighbor of intra-prediction block PB1 is available in the reference data buffer 126, and the non-inter-prediction circuit 124 performs intra-prediction on intra-prediction block PB1 based on the reconstructed data of the left neighbor retrieved from the reference data buffer 126. After intra-prediction of intra-prediction block PB1 is completed, the left adjacent reconstructed data stored in reference data buffer 126 is no longer needed and can be deleted.

[0034] Furthermore, after the inter-frame prediction of inter-frame prediction block PB0 is completed, the top adjacent reconstruction data of intra-frame prediction block PB2 becomes available in reference data buffer 126. Therefore, after the intra-frame prediction of intra-frame prediction block PB1 is completed, the non-inter-frame prediction circuit 124 performs intra-frame prediction for the next intra-frame prediction block PB2 based on the top adjacent reconstruction data retrieved from reference data buffer 126. Similarly, after the intra-frame prediction of intra-frame prediction block PB2 is completed, the top adjacent reconstruction data stored in reference data buffer 126 is no longer needed and can be deleted. Since reference data buffer 126 is now empty, inter-frame prediction for the next inter-frame prediction block PB3 can be triggered, and intra-frame prediction for the next intra-frame prediction block PB5 is not triggered until the left adjacent reconstruction data of intra-frame prediction block PB5 is ready in reference data buffer 126.

[0035] Since those skilled in the art can easily understand the processing details of the subsequent inter-frame prediction blocks PB3-PB6 and intra-frame prediction blocks PB5-PB6 after reading the above paragraphs, further descriptions are omitted here for the sake of brevity.

[0036] Inter-intra sequence processing can be implemented with the maximum buffer size allowing the reference data buffer 126 to store adjacent data (i.e., intra-frame reference data) obtained only from a single inter-prediction block. However, this is for illustrative purposes only and is not intended to limit the invention. For example, inter-intra parallel processing can be implemented when the maximum buffer size is appropriately set to allow the reference data buffer 126 to store adjacent data (i.e., intra-frame reference data) obtained from multiple inter-prediction blocks.

[0037] Please combine Figure 7 refer to Figure 4 . Figure 7 This is an illustration based on an embodiment of the present invention. Figure 4 This diagram illustrates the second prediction process for multiple prediction blocks in an LCU / SB402. It is assumed that the reference data buffer 126 can store adjacent data (i.e., intra-frame reference data) obtained from all inter-prediction blocks in an LCU / SB. When the inter-prediction circuit 122 performs inter-prediction on inter-prediction block PB0, the reconstructed data of the left neighbor of intra-prediction block PB1 and the reconstructed data of the upper neighbor of intra-prediction block PB2 are obtained through reconstruction based on the inter-prediction result of inter-prediction block PB0 and stored in the reference data buffer 126 for later use. Since the reference data buffer 126 can store adjacent data (i.e., intra-frame reference data) obtained from all inter-prediction blocks in an LCU / SB, inter-prediction of inter-prediction block PB3 begins after inter-prediction of inter-prediction block PB0 is completed. Similarly, inter-prediction of inter-prediction block PB4 begins after inter-prediction of inter-prediction block PB3 is completed, and inter-prediction of inter-prediction block PB6 begins after inter-prediction of inter-prediction block PB4 is completed.

[0038] After the inter-frame prediction of inter-frame prediction block PB0 is completed, the reconstructed data of the left adjacent frame prediction block PB1 becomes available in reference data buffer 126. Since the required intra-frame reference data is ready in reference data buffer 126, non-inter-frame prediction circuit 124 is triggered to perform intra-frame prediction of intra-frame prediction block PB1 based on the reconstructed data of the left adjacent frame prediction block extracted from reference data buffer 126.

[0039] Similarly, after the inter-frame prediction of inter-frame prediction block PB0 is completed, the top adjacent reconstruction data of intra-frame prediction block PB2 is available in reference data buffer 126. Since the required intra-frame reference data is ready in reference data buffer 126 after the intra-frame prediction of intra-frame prediction block PB1 is completed, non-inter-frame prediction circuit 124 is triggered to perform intra-frame prediction of intra-frame prediction block PB2 based on the top adjacent reconstruction data extracted from reference data buffer 126.

[0040] After the inter-frame prediction of inter-frame prediction block PB4 is completed, the reconstructed data of the left adjacent frame of intra-frame prediction block PB5 becomes available in reference data buffer 126. Since the intra-frame prediction of intra-frame prediction block PB2 is completed and the required intra-frame reference data is ready in reference data buffer 126, the non-inter-frame prediction circuit 124 is triggered to perform intra-frame prediction of intra-frame prediction block PB5 based on the reconstructed data of the left adjacent frame taken from reference data buffer 126 and the reconstructed data of the top adjacent frame available in local buffer 125.

[0041] After the inter-frame prediction of inter-frame prediction block PB6 is completed, the reconstructed data of the left adjacent frame of intra-frame prediction block PB7 becomes available in reference data buffer 126. Since the intra-frame prediction of intra-frame prediction block PB5 is completed, the required intra-frame reference data is ready in reference data buffer 126, triggering non-inter-frame prediction circuit 124 to perform intra-frame prediction of intra-frame prediction block PB7 based on the reconstructed data of the left adjacent frame retrieved from reference data buffer 126 and the reconstructed data of the top adjacent frame available in local buffer 125.

[0042] like Figure 7 As shown, parallel processing between frames and within frames can be achieved. That is, the first time period during which the inter-frame prediction circuit 122 performs inter-frame prediction for an inter-frame prediction block overlaps with the second time period during which the non-inter-frame prediction circuit 124 performs intra-frame prediction for an intra-frame prediction block. This improves the overall decoding performance.

[0043] With the maximum buffer size of the reference data buffer 126 allowing it to store adjacent data (i.e., intra-frame reference data) obtained from all inter-frame prediction blocks in a single LCU / SB, inter-frame prediction for an inter-frame prediction block of the next LCU / SB will not begin until the reference data buffer 126 becomes empty at the end of intra-frame prediction for the last intra-frame prediction block of the current LCU / SB. Figure 7 As shown, when the inter-prediction of the last inter-prediction block PB6 included in the current LCU / SB is completed, the inter-prediction of the next inter-prediction block included in the next LCU / SB is not triggered. However, this is for illustrative purposes only and does not imply limitation of the invention. In an alternative design, the buffer size of the reference data buffer 126 can be large enough to allow the reference data buffer 126 to store adjacent data (i.e., intra-frame reference data) obtained from all inter-prediction blocks in N LCU / SBs, where N≥2.

[0044] Please combine Figure 9 Please refer to the above. Figure 8 . Figure 8 This is a diagram illustrating a prediction block obtained by segmenting two consecutive LCU / SB according to an embodiment of the present invention. Figure 9 This describes an embodiment of the present invention. Figure 8 The diagram shows the prediction processing for multiple prediction blocks in the two LCU / SBs. (See figure.) Figure 8 As shown, the next LCU / SB 404 can be divided into multiple inter-frame prediction blocks PB8, PB9, PB10, and PB11. Assume that the reference data buffer 126 can store adjacent data (i.e., intra-frame reference data) obtained from all inter-frame prediction blocks in the two LCU / SBs. When the inter-frame prediction circuit 122 performs inter-frame prediction on inter-frame prediction block PB0, based on the inter-frame prediction result of inter-frame prediction block PB0, it obtains the reconstructed data of the left neighbor of intra-frame prediction block PB1 and the reconstructed data of the upper neighbor of intra-frame prediction block PB2 through reconstruction, and stores them in the reference data buffer 126 for later use. Since the reference data buffer 126 can store adjacent data (i.e., intra-frame reference data) obtained from all inter-frame prediction blocks in the two LCU / SBs, after the inter-frame prediction of inter-frame prediction block PB0 included in LCU / SB 402 is completed, the inter-frame prediction of inter-frame prediction block PB3 included in LCU / SB 402 begins; after the inter-frame prediction of inter-frame prediction block PB3 included in LCU / SB 402 is completed, the inter-frame prediction of inter-frame prediction block PB4 included in LCU / SB 402 begins; after the inter-frame prediction of inter-frame prediction block PB4 included in LCU / SB 402 is completed, the inter-frame prediction of inter-frame prediction block PB6 included in LCU / SB 402 begins; after the inter-frame prediction of inter-frame prediction block PB6 included in LCU / SB 402 is completed, the inter-frame prediction of inter-frame prediction block PB8 included in LCU / SB 404 begins. After the inter-frame prediction of inter-frame prediction block PB8 included in LCU / SB 404 is completed, inter-frame prediction of inter-frame prediction block PB9 included in LCU / SB 404 begins. After the inter-frame prediction of inter-frame prediction block PB9 included in LCU / SB 404 is completed, inter-frame prediction of inter-frame prediction block PB10 included in LCU / SB 404 begins. After the inter-frame prediction of inter-frame prediction block PB10 included in LCU / SB 404 is completed, inter-frame prediction of inter-frame prediction block PB11 included in LCU / SB 404 begins. In short, if N≥2, the inter-frame prediction circuit 122 is allowed to perform inter-frame prediction on subsequent inter-frame prediction blocks included in the next LCU / SB without waiting for the intra-frame prediction of all intra-frame prediction blocks included in the current LCU / SB to be completed.

[0045] about Figure 1In the illustrated embodiment, reference data buffer 126 is used to buffer non-inter-frame reference data, which includes at least a portion (i.e., part or all) of the reconstruction result (i.e., reconstruction block) obtained by combining inter-frame prediction results (i.e., inter-frame prediction blocks) and corresponding residual data. Non-inter-frame reference data can be extracted from reference data buffer 126 for CfN mode prediction. CfN mode is a standalone mode that can be treated the same as intra-frame prediction in terms of encoding / decoding order. For example, NIP mode and CfN mode can have the same encoding / decoding order. CfN mode can be executed by a standalone module other than the inter-frame prediction module and the intra-frame prediction module, or it can be executed by reusing the intra-frame prediction module, or it can be executed by setting the current image as the reference image and reusing the inter-frame prediction module. When non-inter-frame prediction circuit 124 uses CfN mode, non-inter-frame prediction circuit 124 can be a standalone circuit designed to process CfN mode prediction, or it can be integrated into the intra-frame prediction circuit, or it can be integrated into the inter-frame prediction circuit 122.

[0046] When CFN mode is selected, the non-inter-frame reference data stored in reference data buffer 126 may include all reconstruction results determined for an inter-frame prediction block. Figure 10 This is a diagram illustrating non-inter-frame reference data used by the CfN mode at a first LCU size (e.g., 128x128) according to an embodiment of the present invention. More specifically, the CfN mode uses motion information 1002 to identify reference predictions in the current frame. For example, the CfN mode could be an intra-block copy (IBC) mode specified in the Universal Video Codec (VCC) standard or any other codec tool to specify reference predictions in the current block using motion information. Reference data buffer 126 can be used as an IBC buffer with a buffer size not less than 4x64x64 reconstructed samples, wherein reconstructed data in the shaded areas is available in the reference data buffer 126. With an LCU size of 128x128, reconstructed block BK2 is identified using the resolved motion information 1002, used as the prediction result for block BK1 in the same image. The reconstructed samples of block BK2 can be obtained by combining the inter-frame prediction results of the inter-frame prediction blocks (i.e., the inter-frame prediction blocks) and the corresponding residual data, and stored in the reference data buffer 126 to serve as non-inter-frame reference data for block BK1. Therefore, when predicting block BK1 in IBC mode, non-inter-frame reference data can be extracted from the reference data buffer 126. It should be noted that the same concept of using the reference data buffer 126 to buffer non-inter-frame reference data derived from the inter-frame prediction results of previously processed inter-frame prediction blocks and subsequently used for prediction in IBC mode can also be applied to LCUs with different LCU sizes (e.g., 64x64 or 32x32). Figure 11This is a diagram illustrating non-inter-frame reference data used by CfN mode at a second LCU size (e.g., 64x64) according to an embodiment of the present invention. Figure 12 This is a diagram illustrating non-inter-frame reference data used by CfN mode at a third LCU size (e.g., 32x32) according to an embodiment of the present invention. For an LCU size of K x K, the previously decoded Q x K x K can be reference data, such as... Figure 11 As shown. For an LCU size of (1 / 2)Kx(1 / 2)K, the previously decoded 4Qx(1 / 2)Kx(1 / 2)K can be used as reference data, such as... Figure 12 As shown. For an LCU size of 2K x 2K, the previously decoded Q x K x K can be used as reference data, such as... Figure 10 As shown. Therefore, the buffer size of reference data buffer 126 needs to be greater than Q x K x K.

[0047] It should be noted that when block BK1 is processed in IBC mode, parallel inter-frame and non-inter-frame prediction can be achieved with the help of reference data buffer 126. For example, when IBC mode prediction processes block BK1 based on non-inter-frame reference data extracted from reference data buffer 126 (e.g., reconstructed data of inter-frame block BK2), inter-frame prediction circuit 122 performs inter-frame prediction on inter-frame block BK3, wherein the start time of inter-frame prediction for inter-frame block BK2 is earlier than the start time of IBC mode prediction for block BK1, and also earlier than the start time of inter-frame prediction for inter-frame block BK3.

[0048] Figure 13 This is a diagram illustrating a second video processing apparatus according to an embodiment of the present invention. The video processing apparatus 1100 may be part of a video decoder. The main difference between video processing apparatuses 1100 and 100 is that the prediction processing system 1110 uses a reference data buffer 1126 to store the inter-frame prediction results of a prediction block as non-inter-frame reference data. Regarding the prediction processing system 1110, the processing circuit 1320 may include an inter-frame prediction circuit 1122 and a non-inter-frame prediction circuit 1124. The non-inter-frame prediction circuit 1124 may be used to process NIP modes, and / or may be used to process codec modes that are considered intra-frame predictions in terms of encoding / decoding order, such as weighted inter-intra (WII) modes. The processing circuit 1320 is shown as having two prediction circuits, but the invention is not limited thereto. In some embodiments, the processing circuit 1320 may have one inter-frame prediction circuit, one intra-frame prediction circuit, and other non-inter-frame prediction circuits. In some embodiments, the processing circuit 1320 may have control logic circuitry (not shown) for managing the control flow to implement parallel non-inter-frame and inter-frame predictions.

[0049] Inter-frame prediction circuit 1122 is arranged to perform a first inter-frame prediction operation on a first prediction block to generate a first inter-frame prediction result (e.g., an inter-frame prediction block determined by inter-frame prediction for the first prediction block), and is further arranged to perform a second inter-frame prediction operation on a second prediction block during a first time period; reference data buffer 1126 is used to buffer reference data derived from the first inter-frame prediction result (specifically, the reference data includes the first inter-frame prediction result); non-inter-frame prediction circuit 1124 is used to extract reference data from reference data buffer 1126 and perform a non-inter-frame prediction operation (e.g., WII mode prediction) based on the reference data during a second time period, wherein the second time period overlaps with the first time period, and the start time of the first inter-frame prediction operation is earlier than the start time of the non-inter-frame prediction operation. Parallel inter-frame and non-inter-frame prediction can be achieved by using reference data buffer 1126 for buffering reference data required for non-inter-frame prediction operations.

[0050] WII mode is a standalone mode and can be considered as intra-prediction in terms of encoding / decoding order. For example, NIP mode and WII mode can have the same encoding / decoding order. WII mode can be executed by a standalone module other than the inter-prediction module and the intra-prediction module, or it can be executed by reusing the intra-prediction module, or it can be executed by reusing the inter-prediction module. When the non-inter-prediction circuit 1124 adopts WII mode, the non-inter-prediction circuit 1124 can be a standalone circuit designed to handle WII mode prediction, or it can be integrated into the intra-prediction circuit, or it can be integrated into the inter-prediction circuit 1122.

[0051] In this embodiment, the non-inter-frame prediction circuit 1124 can use WII mode to process the current prediction block, wherein the non-inter-frame reference data required for the WII mode prediction of the current prediction block and stored in the reference data buffer 1126 can be the inter-frame prediction result of the same prediction block. For example, the WII mode can be the combined inter and intra prediction (CIIP) mode specified in the VVC standard.

[0052] Figure 14 This is an explanation of an embodiment of the present invention. Figure 13 The diagram shows the CIIP operation performed by the non-inter-frame prediction circuit 1124. The inter-frame prediction circuit 1122 will perform the inter-frame prediction result P of the current prediction block. inter Send to reference data buffer 1126. In the inter-frame prediction result P... inter After becoming available in the reference data buffer 1126, the non-inter-frame prediction circuit 1124 obtains the inter-frame prediction result P from the reference data buffer 1126. interObtain the intra-frame prediction result P of the current prediction block. intra And based on the intra-frame prediction result P intra Inter-frame prediction result P inter Determine the prediction result P of the current prediction block. wii For example, the prediction result P wii The calculation can be expressed by the following formula.

[0053] P wii =We*P inter +Wa*P intra , where We+Wa=1 (1)

[0054] Prediction result P wii It is provided to the reconstruction circuit 112. Therefore, the prediction result P is... wii The corresponding residual data D_RES is combined to generate the reconstruction result D_REC of the current prediction block, which is then stored in the reconstruction buffer 114. If the buffer size of the reference data buffer 1126 is sufficient to store the inter-frame prediction results of multiple prediction blocks, the inter-frame prediction circuit 1122 can perform inter-frame prediction on the next prediction block to obtain the prediction result P while the non-inter-frame prediction circuit 1124 is processing related calculations. wii And it can use the inter-frame prediction result P of the next prediction block inter _NX is stored in reference data buffer 1126 for later use. In other words, parallel inter-frame and non-inter-frame prediction can be achieved with the help of reference data buffer 1126.

[0055] Figure 15This is a diagram illustrating a third video processing apparatus according to an embodiment of the present invention. The video processing apparatus 1300 may be part of a video decoder. The main difference between the video processing apparatuses 1100 and 1300 is that the prediction processing system 1310 uses a reference data buffer 1326, which is implemented by a shared space allocated in the reconstruction buffer 1314. Regarding the prediction processing system 1310, the inter-frame prediction circuit 1322 is arranged to perform a first inter-frame prediction operation on a first prediction block to generate a first inter-frame prediction result (e.g., an inter-frame prediction block determined by inter-frame prediction for the first prediction block), and is further arranged to perform a second inter-frame prediction operation on a second prediction block during a first time period; a reference data buffer 1326 is used to buffer reference data derived from the first inter-frame prediction result (specifically, the reference data includes the first inter-frame prediction result); a non-inter-frame prediction circuit 1324 is used to extract reference data from the reference data buffer 1326 and perform a non-inter-frame prediction operation (e.g., WII mode prediction) based on the reference data during a second time period, wherein the second time period overlaps with the first time period, and the start time of the first inter-frame prediction operation is earlier than the start time of the non-inter-frame prediction operation. Parallel inter-frame and non-inter-frame prediction can be achieved by using the reference data buffer 1326 for buffering the reference data required for the non-inter-frame prediction operation. In this embodiment, the non-inter-frame prediction circuit 1324 can use WII mode to process the current prediction block, where the non-inter-frame reference data required for the WII mode prediction of the current prediction block and stored in the reference data buffer 1326 can be the inter-frame prediction result of the same prediction block. For example, the WII mode can be the CIIP mode specified in the VVC standard. Furthermore, the inter-frame prediction result and reconstruction result of the same prediction block can share the same storage space. Specifically, the inter-frame prediction result stored in the reference data buffer 1326 (which is part of the reconstruction buffer 1314) can be overwritten by the reconstruction result of the same prediction block.

[0056] Figure 16 This is an illustration of an embodiment of the present invention. Figure 15 The diagram shows the CIIP operation performed by the non-inter-frame prediction circuit 1324. The inter-frame prediction circuit 1322 will perform the inter-frame prediction result P of the current prediction block. inter Send to reference data buffer 1326. In the inter-frame prediction result P... inter After becoming available in the reference data buffer 1326, the non-inter-frame prediction circuit 1324 obtains the inter-frame prediction result P from the reference data buffer 1326. inter Obtain the intra-frame prediction result P of the current prediction block. intra And based on the intra-frame prediction result P intra Inter-frame prediction result P inter Determine the prediction result P of the current prediction block. wiiFor example, the prediction result P can be obtained using the above formula (1). wii .

[0057] Prediction result P wii It is provided to the reconstruction circuit 112. Therefore, by using the prediction result P wii The corresponding residual data D_RES is combined to generate the reconstruction result D_REC of the current prediction block, and then stored in the reference data buffer 1326 (which is part of the reconstruction buffer 1314) to overwrite the inter-frame prediction results P that are no longer needed for the prediction processing of any prediction block. inter If the buffer size of the reference data buffer 1326 is sufficient to store the inter-frame prediction results of multiple prediction blocks, the inter-frame prediction circuit 1322 can perform inter-frame prediction on the next prediction block to obtain the prediction result P while the non-inter-frame prediction circuit 1324 is processing related calculations. wii And it can use the inter-frame prediction result P of the next prediction block inter _NX is stored in reference data buffer 1326 for later use. In other words, parallel inter-frame and non-inter-frame prediction can be achieved with the help of reference data buffer 1326.

[0058] The flowchart shown is intended to illustrate examples of video encoding and decoding according to the present invention. Those skilled in the art can modify each step, rearrange the steps, split the steps, or combine the steps to implement the invention without departing from the spirit of the invention. Specific syntax and semantics have been used in this disclosure to illustrate examples of implementing embodiments of the invention. Those skilled in the art can practice the invention by substituting equivalent syntax and semantics for the existing syntax and semantics without departing from the spirit of the invention.

[0059] The above description is presented to enable those skilled in the art to practice the invention provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details have been set forth in order to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0060] The embodiments of the invention described above can be implemented in various hardware, software code, or combinations thereof. For example, one embodiment of the invention may be one or more circuits integrated into a video compression chip or program code integrated into video compression software to perform the processes described herein. Embodiments of the invention may also be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to numerous functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors may be configured to perform specific tasks according to the invention by executing machine-readable software code or firmware code that defines the specific methods embodied in the invention. The software code or firmware code may be developed in different programming languages ​​and in different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, styles, and languages ​​of the software code, as well as other ways of configuring the code to perform the tasks according to the invention, will not depart from the spirit and scope of the invention.

[0061] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All variations within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A predictive processing system, comprising: The processing circuit is configured to perform a first inter-frame prediction operation on the first prediction block to generate a first inter-frame prediction result, and is further configured to perform a second inter-frame prediction operation on the second prediction block during a first time period. as well as A reference data buffer is used to buffer reference data derived from the prediction results of this first inter-frame; The processing circuit is further configured to extract the reference data from the reference data buffer and perform non-inter-frame prediction operations based on the reference data at least during the second time period, wherein the second time period overlaps with the first time period; The block targeted by the first inter-frame prediction operation is located to the left or above the block targeted by the non-inter-frame prediction operation performed during the second time period.

2. The prediction processing system as described in claim 1, wherein, The first inter-frame prediction operation starts earlier than the non-inter-frame prediction operation.

3. The prediction processing system according to claim 1, wherein, The reference data includes at least a portion of the reconstruction results obtained by combining the first inter-frame prediction results and the corresponding residual data.

4. The prediction processing system as described in claim 3, wherein, This reference data includes only a portion of the reconstruction results.

5. The prediction processing system as described in claim 4, wherein, The processing circuit is used to perform the non-inter-frame prediction operation on the third prediction block, the first prediction block and the third prediction block are in the same image, and the first prediction block is an adjacent block of the third prediction block.

6. The prediction processing system as described in claim 3, wherein, This non-inter-frame prediction operation is performed in intra-block copy (IBC) mode.

7. The prediction processing system as described in claim 1, wherein, The reference data includes the first inter-frame prediction result.

8. The prediction processing system as described in claim 7, wherein, The processing circuit is used to perform the non-inter-frame prediction operation on the first prediction block to generate a non-inter-frame prediction result, and to determine the prediction result of the first prediction block based on the inter-frame prediction result and the non-inter-frame prediction result.

9. The predictive processing system as described in claim 8, wherein, This non-inter-frame prediction operation is performed in a combined inter-frame and intra-frame prediction (CIIP) mode.

10. The prediction processing system of claim 7, wherein, The reference data buffer is implemented by the shared space in the reconstruction buffer; the reference data buffer is also used to cache the reconstruction result obtained by combining the prediction result with the corresponding residual data, wherein the first inter-frame prediction result cached in the reference data buffer is overwritten by the reconstruction result.

11. A prediction processing method, comprising: Perform the first inter-frame prediction operation on the first prediction block to produce the first inter-frame prediction result; The reference data derived from the first inter-frame prediction result is cached in the reference data buffer; During the first time period, perform the second inter-frame prediction operation on the second prediction block; The reference data is extracted from the reference data buffer, and non-inter-frame prediction operations are performed based on the reference data at least during the second time period, wherein the second time period overlaps with the first time period; The block targeted by the first inter-frame prediction operation is located to the left or above the block targeted by the non-inter-frame prediction operation performed during the second time period.

12. The prediction processing method as described in claim 11, wherein, The first inter-frame prediction operation starts earlier than the non-inter-frame prediction operation.

13. The prediction processing method as described in claim 11, wherein, The reference data includes at least a portion of the reconstruction results obtained by combining the first inter-frame prediction results and the corresponding residual data.

14. The prediction processing method as described in claim 13, wherein, This reference data includes only a portion of the reconstruction results.

15. The prediction processing method as described in claim 14, wherein, The non-inter-frame prediction operation is performed on the third prediction block, where the first prediction block and the third prediction block are in the same image, and the first prediction block is a neighboring block of the third prediction block.

16. The prediction processing method as described in claim 13, wherein, This non-inter-frame prediction operation is performed in intra-block copy (IBC) mode.

17. The prediction processing method as described in claim 11, wherein, The reference data includes the first inter-frame prediction result.

18. The prediction processing method as described in claim 17, wherein, During the second time period, performing the non-inter-frame prediction operation based on the reference data at least includes: Perform the non-inter-frame prediction operation on the first prediction block to produce a non-inter-frame prediction result; and The prediction result of the first prediction block is determined based on the inter-frame prediction result and the non-inter-frame prediction result.

19. The prediction processing method as described in claim 18, wherein, This non-inter-frame prediction operation is performed in a combined inter-frame and intra-frame prediction (CIIP) mode.

20. The prediction processing method as described in claim 17, wherein, The reference data cache is implemented using shared space in the refactoring cache, and the prediction processing method also includes: The reconstruction result is used to overwrite the first inter-frame prediction result cached in the reference data cache, wherein the reconstruction result is obtained by combining the prediction result and the corresponding residual data.

Images