Adaptive loop filtering method and corresponding circuit

Abstract

Aspects of the present disclosure provide a method and circuit for adaptive loop filtering in a video codec system. The method may include receiving a sample block generated from a pre-filter circuit in a filter pipeline, the sample block being one of multiple blocks included in a current image, performing adaptive loop filter processing in parallel while the pre-filter circuit simultaneously processes another block in the current image, storing a first sample having a filter input area defined by a filter shape including at least one unreceived sample in a buffer, and storing a second sample included in the filter input area of ​​the first sample in the buffer. The adaptive loop filtering method and corresponding circuit of the present invention can improve decoding efficiency.

Description

[0001]

Cross-reference

[0002] This application claims priority to U.S. Provisional Application No. 62 / 340,015, filed May 23, 2016, the contents of which are incorporated herein by reference.

Technical field

[0003] The disclosed embodiments of the present invention relate to adaptive loop filtering, and more particularly, to an adaptive loop filtering method and corresponding circuit for efficient adaptive loop filtering processing for video encoding and decoding. [Background Technology]

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, the work described in this background section, and aspects that may not qualify as prior art at the time of filing are neither explicitly nor implicitly admitted as prior art to the present disclosure.

[0005] Block-based motion compensation, transforms, and quantization are widely used in video compression to improve the performance of video communication systems. However, due to coarse quantization and motion compensation, compression noise can be introduced, which causes artifacts such as blocking, ringing, and blurring in the reconstructed image. Loop filters can be used to reduce compression noise, which not only improves the quality of the output decoded image, but also provides a high-quality reference image for subsequent images to save codec bits. An adaptive loop filter is one type of such loop filter. The adaptive loop filtering process can minimize the mean square error between the original samples and the reconstructed samples by using a Wiener-based adaptive filter. [Summary of the invention]

[0006] Aspects of the present disclosure provide a method for adaptive loop filtering in a video codec system. The method may include receiving a sample block generated from a pre-stage filter circuit in a filter pipeline, the sample block being one of a plurality of blocks included in a current image, performing adaptive loop filter processing in parallel while the pre-stage filter circuit simultaneously processes another block in the current image, storing in a buffer a first sample having a filter input region defined by a filter shape including at least one unreceived sample, and storing in the buffer a second sample included in the filter input region of the first sample.

[0007] In one example, the pre-filter circuit is a deblocking filter circuit or a sample adaptive offset filter (SAO) circuit. In one embodiment, the method further includes receiving a next sample block adjacent to the sample block, reading first and second samples from a buffer, and performing ALF processing on at least a portion of the samples in a block formed by the next adjacent sample block and the first sample stored in the buffer. In one embodiment, the method further includes: starting ALF processing on at least one target sample in the received sample block before the pre-filter circuit completes processing of samples in the current coding tree unit including the received sample block.

[0008] In one embodiment of the method, the buffer includes a left buffer including a first portion of a first sample and a second sample adjacent to a first block to be processed in the same row as the sample block, and a top buffer including a second portion of a first sample and a second sample adjacent to a second block to be processed in the same row as the sample block. In one embodiment, the first and second samples are P+Q sample columns adjacent to a next block in the image, where P and Q are left and right strides of the filter shape.

[0009] In one embodiment, ALF processing is performed in parallel on multiple target samples in a sample block, including receiving samples in the sample block, performing a first ALF processing on a first target sample in the sample block to generate a first filtered sample based on the received sample in a first filter input region of the first target sample defined by a filter shape, and performing a second ALF processing on a second target pixel in the sample block adjacent to the first target pixel to generate a second filtered sample based on the received sample in a second filter input region of the second target sample defined by the filter shape, wherein the received sample used for the first ALF processing of the first target sample is reused for the second ALF processing of the second target sample.

[0010] In one example, receiving the sample block includes receiving the samples in the block row-by-row as input to a multi-stage pipeline filter circuit, and sequentially generating first and second filtered samples from the MSPF circuit. In another example, receiving the sample block includes receiving the samples in the block row-by-row as input to a first MSPF circuit and a second MSPF circuit operating in parallel with the first MSPF circuit, and generating the first and second filtered samples at the first and second MSPF circuits, respectively, based on the samples received in the block.

[0011] In one embodiment, receiving the samples in the block includes shifting the samples in the block row by row into an array of shift registers having rows of shift registers, each row of shift registers storing a row of shifted samples. Thus, performing the first and second ALF processes includes computing a first filtered sample based on the samples currently stored in the shift register array via a multiply-add circuit coupled to the shift register array to receive the samples from the shift register array, shifting the next row of samples in the first block into the shift register array, and computing a second filtered sample based on the samples currently stored in the shift register array of the multiply-add circuit.

[0012] In another embodiment, receiving the block of samples includes shifting the samples in the block row by row into an array of shift registers having rows of shift registers, each row of shift registers storing a row of samples. Thus, performing the first and second ALF processes includes calculating first filtered samples based on a first set of samples currently stored in the shift register array by a first adding circuit coupled to the shift register array to receive the samples from the shift register array, and calculating second filtered samples based on a second set of samples currently stored in the shift register array by a second multiply-add circuit coupled to the shift register array to receive the second set of samples from the shift register array in parallel with the calculation of the first filtered samples, the second set of samples including a portion of the first set of samples.

[0013] The embodiment of the method further comprises dividing the sample block into a first sub-block and a second sub-block, and then performing a first ALF process on the first target sample in the first sub-block and a second ALF process on the second target sample in the second sub-block in parallel.

[0014] Aspects of the present disclosure provide an adaptive loop filter (ALF) circuit in a video codec system. The ALF circuit may include a working buffer configured to store a sample block generated by a front-stage filter circuit in a filter pipeline, the sample block being one of a plurality of blocks included in a current image, the filter circuit configured to perform adaptive loop filter processing on a plurality of target samples in the sample block in parallel while the front-stage filter circuit simultaneously processes another block in the current image, and a side buffer configured to store a plurality of first samples, wherein each of the plurality of first samples has a filter input region defined by a filter shape including at least one unreceived sample, and to store a plurality of second samples included in the plurality of filter input regions of the plurality of first samples.

[0015] The adaptive loop filtering method and corresponding circuit of the present invention can improve the quality of output decoded images.

Brief Description of the Drawings

[0016] Various embodiments of the present disclosure, presented as examples, will be described in detail with reference to the following drawings, in which like reference numerals represent like elements, and in which:

[0017] Figure 1 An encoder according to an embodiment of the present disclosure is shown;

[0018] Figure 2 A decoder according to an embodiment of the present disclosure is shown;

[0019] Figure 3A The traditional image-level filter pipeline is shown;

[0020] Figure 3B shows an example timing diagram of an image-level filter pipeline operating on an image-by-image basis;

[0021] Figure 4A shows a block-level filter pipeline according to an embodiment of the present disclosure;

[0022] Figure 4B shows an example timing diagram of a block-level filter pipeline according to an embodiment of the present disclosure;

[0023] Figure 5 An example of a deblocking filter and sample adaptive offset filter (DF / SAO) ready area according to an embodiment of the present disclosure is shown;

[0024] Figure 6 shows a filter shape of a finite impulse response (FIR) filter according to an embodiment of the present disclosure;

[0025] Figures 7A-7B A first block-level processing technique according to an embodiment of the present disclosure is shown;

[0026] Figure 8 shows an adaptive loop filtering process according to an embodiment of the present disclosure;

[0027] Figures 9A-9B A second block-level processing technique according to an embodiment of the present disclosure is shown;

[0028] Figure 10A shows a set of blocks according to an embodiment of the present disclosure;

[0029] Figure 10B shows an example timing diagram of a sub-block level filtering process according to an embodiment of the present disclosure;

[0030] Figures 11A-11B A third block-level processing technique according to an embodiment of the present disclosure is shown;

[0031] Figure 12 An adaptive loop filter (ALF) according to an embodiment of the present disclosure is shown;

[0032] Figure 13 shows a first ALF according to an embodiment of the present disclosure;

[0033] Figure 14 shows a second ALF according to an embodiment of the present disclosure;

[0034] Figure 15 shows a third ALF according to an embodiment of the present disclosure;

[0035] Figure 16 shows a fourth ALF according to an embodiment of the present disclosure;

[0036] Figure 17 A first loop filter circuit according to an embodiment of the present disclosure is shown;

[0037] Figure 18 A second loop filter circuit according to an embodiment of the present disclosure is shown; and

[0038] Figure 19 An adaptive loop filtering process 1900 in a video coding system according to an embodiment of the present disclosure is shown. [Specific implementation method]

[0039] Certain words are used in the specification and claims to refer to specific components. It should be understood by those skilled in the art that manufacturers may use different terms to refer to the same component. This specification and claims do not distinguish components based on differences in name, but rather on differences in the functions of the components. The word "including" mentioned throughout the specification and claims is an open-ended term and should be interpreted as "including but not limited to". In addition, the word "coupled" includes any direct and indirect electrical connection means. Therefore, if the text describes a first device coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means.

[0040] Figure 1 The encoder 100 according to an embodiment of the present disclosure is shown. The encoder 100 may include a decoded image buffer 110, an inter-frame intra-frame prediction module 112, a first adder 114, a residual encoder 116, an entropy encoder 118, a residual decoder 120, a second adder 122, a deblocking filter (DF) 130, a sample adaptive offset (SAO) filter 132, and an adaptive loop filter (ALF) 134. These components may be as follows: Figure 1 shown coupled together.

[0041] The encoder 100 receives input video data 101 and performs a video compression process to generate a bitstream 102 as output. The input video data 101 may include a series of images. Each image may include one or more color components, such as a luminance component or a chrominance component. The bitstream 102 may have a format that conforms to a video codec standard, such as the Advanced Video Codec (AVC) standard, the High Efficiency Video Codec (HEVC) standard, or the like.

[0042] According to one aspect of the present disclosure, the ALF 134 can employ block-level processing techniques to process reconstructed video data block by block. A conventional ALF is a picture-level codec that requires a buffer for the entire picture. In contrast, a block-level ALF can have a smaller working buffer to store a sample block. The block-based ALF 134 enables, for example, a block-level filter pipeline 136 formed by the DF 130, SAO 132, and ALF 134, which can reduce processing latency and buffer size compared to a conventional picture-level filter pipeline. Furthermore, the ALF 134 can employ data reuse techniques to reduce data access time for ALF processing performed on reconstructed samples. For example, sample data acquired in a single data access operation can be used for ALF processing on multiple pixels. Furthermore, in one example, two filter pipelines can be employed to operate in parallel. Each of the two filter pipelines can include an ALF similar to the ALF 134.

[0043] like Figure 1 As shown, the decoded image buffer 110 stores reference images used for motion estimation and motion compensation performed by the intra prediction module 112. The inter-intra prediction module 112 performs inter-image prediction or intra-image prediction to determine blocks of a current image in the video compression process. The current image refers to the image in the input video data 101 being processed by the intra prediction module 112. The current image can be divided into a plurality of blocks of the same or different sizes for inter or intra prediction operations.

[0044] In one example, the inter-frame prediction module 112 processes a block using either inter-frame coding or intra-frame coding. Thus, a block using inter-frame coding is referred to as an inter-frame coding block, while a block using intra-frame coding is referred to as an intra-frame coding block. Inter-frame coding uses a reference image to obtain a prediction of the block currently being processed (referred to as the current block). For example, when encoding the current block using inter-frame coding, motion estimation can be performed to search for a matching area in the reference image. The matching area is used as a prediction for the current block. In contrast, intra-frame coding uses neighboring blocks of the current block to generate a prediction for the current block. The neighboring blocks and the current block are within the same image. The prediction of the block is provided to the first and second adders 114 and 122.

[0045] The first adder 114 receives the prediction of the block from the inter-intra prediction module 112 and the original samples of the block from the input video data 101. The adder 114 then subtracts the prediction from the original sample values ​​of the block to obtain a residual of the block. The residual of the block is sent to the residual encoder 116.

[0046] The residual encoder 116 receives the residual of the block and compresses the residual to produce a compressed residual. For example, the residual encoder 116 may first apply a transform such as a discrete cosine transform (DCT), a wavelet transform, etc. to the received residual corresponding to the transform block and generate transform coefficients for the transform block. The division of the image into transform blocks may be the same as or different from the division of the image into prediction blocks for intra-frame and inter-frame prediction processing. Subsequently, the residual encoder 116 may quantize the coefficients to compress the residual. The compressed residual (quantized transform coefficients) is sent to the residual decoder 120 and the entropy encoder 118.

[0047] Residual decoder 120 receives the compressed residual and performs the inverse of the quantization and transform operations performed at residual encoder 116 to reconstruct the residual of the transformed block. Due to the quantization operation, the reconstructed residual is similar to the original residual produced from adder 114, but is generally different from the original version.

[0048] The second adder 122 receives the prediction of the block from the inter-intra prediction module 112 and the reconstructed residual of the transformed block from the residual decoder 120. The second adder 122 then combines the reconstructed residual with the received prediction corresponding to the same region of the image to generate reconstructed video data. The reconstructed video data can then be passed to the filter pipeline 136.

[0049] In one example, the filter pipeline 136 includes DF 130, SAO 132, and ALF 134, and performs block-based filtering. For example, the image can be divided into multiple blocks. Therefore, reconstructed video data can be generated from the second adder 122 block by block. The filter pipeline 136 receives the reconstructed video data and processes the reconstructed data blocks block by block. In one example, dividing the image into blocks is consistent with dividing the image into coding tree units (CTUs) defined in the HEVC standard. As defined, the image can be divided into a sequence of CTUs. Each CTU can be further divided into smaller codec units (CUs). The CUs in the CTU can be processed independently by the encoder through the steps of motion estimation and compensation, transformation, quantization, and reconstruction. Each CTU or CU can include sampling blocks corresponding to different color components.

[0050] DF 130 applies a set of low-pass filters to block boundaries to reduce block artifacts. The filters can be applied based on the characteristics of the reconstructed samples on both sides of the block boundary in the reconstructed image and the prediction parameters (coding mode or motion vector) determined at the inter-frame-intra prediction module 112. The deblocked reconstructed samples can then be provided to SAO 132. SAO 132 receives the deblocked reconstructed samples and categorizes the pixels in the reconstructed video data into groups. SAO 132 can then determine an intensity offset (offset value) for each group to compensate for the intensity offset of each group. The shifted reconstructed video data can then be provided from SAO 132 to ALF 134. As an example, DF and SAO are defined in the HEVC standard.

[0051] In one example, the ALF 134 receives the reconstructed sample block from the SAO 132 and performs an adaptive loop filtering process. During the adaptive loop filtering process, the ALF process is performed on each target pixel (or target sample) in the block by applying a finite impulse response (FIR) filter. In one example, the FIR filter can be represented by the following expression:

[0052]

[0053] Wherein St represents the filtered sample of the target pixel (or target sample), i is an index indicating the pixel to be filtered in the area (also called the filter input area) surrounding the target pixel, N represents the number of samples to be filtered contained in the area to be filtered of the target pixel, Ci represents the filter parameter corresponding to the i-th pixel to be filtered, Si represents the filtered sample corresponding to the i-th pixel, and is called the tap of the FIR filter. As shown in the figure, the FIR filter can be defined by a sequence of filter coefficients, and the filtered sample of the target pixel can be calculated by applying the FIR filter to the samples adjacent to the target pixel. Applying the FIR filter to the samples adjacent to the target pixel to obtain the filtered samples is called ALF processing of the target pixel.

[0054] In one example of adaptive loop filtering processing, filter parameters for an FIR filter are first derived upon receiving a block of reconstructed samples. For example, expression (1) can be used to represent the filtered samples of a target pixel in a block. Thus, the sum of square errors (SSE) between the original samples received from the input video data and the filtered samples can be formulated. By minimizing the SSE, the Wiener-Hopf equation can be derived. The filter parameters can then be derived by solving the Weiner-Hopf equation. After the filter parameter deviation, the FIR filter can be determined and applied to the target pixel to obtain a filtered block. At the final stage of the adaptive loop filtering process, the filtered block can be stored in a decoded image buffer 110 to form a reference image, and the filter parameters 103 including the derived coefficients can be provided to an entropy encoder 118 and then transmitted to a decoder.

[0055] In an alternative example of adaptive loop filtering, an FIR filter to be applied to a target pixel may be selected from a set of pre-configured FIR filters, for example, based on characteristics of the block of reconstructed samples. Furthermore, an on / off control flag indicating whether to apply the FIR filter to the block may be determined for the block, for example, based on performance criteria used to evaluate the effectiveness of the adaptive loop filtering process. The on / off control flag and index indicating the pre-configured FIR filter may also be included in the filter parameters 103 and transmitted to the decoder.

[0056] The entropy encoder 118 receives the compressed residual from the residual encoder 116 and the filter parameters 103 from the ALF 134. The entropy encoder 118 may also receive other parameters and / or control information, such as intra-prediction mode information, motion vectors, quantization parameters, parameters or control flags from the DF 130 and the SAO 132, etc. The entropy encoder 118 encodes the received filter parameters or other information to form a bitstream 102. The bitstream 102 may be sent to a decoder via a communication network, or to a storage device that may store the video data carried by the bitstream 102.

[0057] Figure 2 2 shows a decoder 200 according to an embodiment of the present disclosure. The decoder 200 includes an entropy decoder 218, a residual decoder 220, a decoded picture buffer 210, an inter-intra prediction module 212, an adder 222, a DF 230, a SAO 232, and an ALF 234. These elements are as follows: Figure 2 DF 230 , SAO 232 , and ALF 234 may form a filter pipeline 236 , as shown coupled together.

[0058] In one example, decoder 200 receives a bitstream 201 from an encoder (e.g., bitstream 102 from encoder 100) and performs a decompression process to generate output video data 202. Output video data 202 may include a series of images that may be displayed on a display device such as a monitor, touch screen, or the like.

[0059] The entropy decoder 218 receives the bitstream 201 and performs a decoding process, which is performed by Figure 1 The decoded image buffer 210 is a reverse process of the encoding process performed by the entropy encoder 118 in the decoded image buffer 210. As a result, a compressed residual, prediction parameters, filter parameters 203, etc. are obtained. The compressed residual is provided to the residual decoder 220, and the prediction parameters are provided to the inter-frame-intra prediction module 212. The inter-frame-intra prediction module 212 generates a prediction of the image block based on the received prediction parameters and provides the prediction to the adder 222. The decoded image buffer 210 stores a reference image useful for motion compensation performed at the inter-frame-intra prediction module 212. The reference image can be formed by a filter block received from the ALF 134, for example. In addition, the reference image is obtained from the decoded image buffer 210 and included in the image video data 202 for display on the display device.

[0060] In terms of function and structure, the residual decoder 220, the adder 222, the DF 230, and the SAO 232 are similar to the residual decoder 120, the second adder 122, the DF 130, and the SAO 132. Description of these components is omitted.

[0061] Similar to Figure 1 For example, the filter pipeline 236 can process the reconstructed samples received from the adder 222 on a block-by-block basis. Figure 1 In the example, ALF 134 and ALF 234 can operate on a block-by-block basis using block-level processing techniques and employ data reuse techniques to reduce data access time for ALF processing performed on target pixels. Furthermore, in alternative examples, more than one filter pipeline similar to pipeline 236 can be employed. Unlike ALF 134, ALF 234 receives filter parameters from entropy decoder 218 to perform adaptive loop filtering processing. For example, the filter parameters may include filter parameters derived at encoder 100 or a filter index determined at encoder 100. ALF 234 can then perform ALF processing on the target pixel in the corresponding block using the received coefficients or an FIR filter indicated by the filter index.

[0062] Using block-level processing techniques in ALFs 134 and 234 enables block-based pipeline processing in filter pipelines 136 and 236. Block-based pipeline processing not only reduces the working buffer size but also reduces processing delays in the filter sequence. In addition, using data reuse techniques can increase the operating speed of ALFs 134 and 234, thereby further improving the performance of encoder 100 and decoder 200.

[0063] although Figure 1 and Figure 2 The examples show a series of filters 130-134 or 230-234 included in the encoder 100 or decoder 200, but it should be understood that in other embodiments, fewer such filters may be included in the encoder or decoder. In addition, although the ALF 134 or 234 is generally arranged at the last stage of the filter sequence, other positions of the ALF 134 or 234 within the filter sequence are also possible. Those positions may be different from Figure 1 or Figure 2 The location shown in the example.

[0064] In various embodiments, the ALF 134 or 234 may be implemented using hardware, software, or a combination thereof. For example, the ALF 134 or 234 may be implemented using one or more integrated circuits (ICs), such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and the like. For another example, the ALF 134 or 234 may be implemented as software or firmware including instructions stored in a computer-readable non-volatile storage medium. When executed by a processing circuit, the instructions cause the processing circuit to perform the functions of the ALF 134 or 234.

[0065] Note that the ALF 134 or 234 implementing the block level processing technique and data reuse technique disclosed herein may be included in a Figure 1 or Figure 2 Furthermore, in various examples, the encoder 100 and the decoder 200 may be included in the same device, or may comprise separate devices.

[0066] Figure 3A 3. A conventional image-level filter pipeline 300A is shown. The filter pipeline 300A includes a first frame buffer 310, a DF 311, a second frame buffer 312, an SAO 313, a third frame buffer 314, and an ALF 315. The DDF 311, SAO 313, and ALF 315 may have similar functions and structures to the DF 130, SAO 132, and ALF 134. However, the DF 311, SAO 313, and ALF 315 operate on a per-image basis.

[0067] Figure 3B An exemplary timing diagram 300B illustrates an image-level filter pipeline 300A on a per-image basis. As shown, operations 331, 341, and 351 corresponding to each stage of pipeline 300A (DF 311, SAO 313, and ALF 315) are sequentially performed. Specifically, during time period T31, reconstruction operations 321-324 corresponding to different blocks a through d (denoted as BLK A through BLK D) are sequentially performed. The different blocks a through d may be segmented from the image, for example, each corresponding to a CTU. Reconstructed samples for the four blocks are stored in a first frame buffer 310. During time period T32, after receiving the entire image of reconstructed video data, DF 311 may begin operation, and deblocking filtered samples may be stored in a second frame buffer 312. During time period T33, SAO 313 may begin operation, and SAO-filtered samples may be stored in a third frame buffer 314. During the time period T34, the ALF 315 may start running and adaptive loop filtering samples are generated. In the above process, the frame buffers 310, 312 and 314 each store the entire image of the reconstructed samples.

[0068] Figure 4A FIG4 shows a block-level filter pipeline 400A according to an embodiment of the present disclosure. The filter pipeline 400A may be the filter pipeline 136 or 236, however, more details are included. The filter pipeline 400A includes a first block buffer 410, a DF 411, a second block buffer 412, an SAO 413, a third block buffer 414, and an ALF 415. The DDF 411, the SAO 413, and the ALF 415 have the following characteristics: Figure 1 The DF 130, SAO 132 and ALF 134 in the example have similar structures or functions or have similar Figure 2 DF 230, SAO 232 and ALF 234 in the example have similar structures or functions. The filter pipeline 400A has three stages, each corresponding to one of the three filters 411, 413 and 415. In addition, DF 411, SAO 413 and ALF 415 operate on a block-by-block basis. Each of the buffers 410, 412 and 414 stores samples of one block, rather than Figure 3A The entire image is sampled in this example. Although Figure 4A In the example of , the buffers 410 , 412 and 414 are shown as separate components from the filters 411 , 413 and 415 , in other examples, the buffers 410 , 412 or 414 may be integrated into the filters 411 , 413 or 415 .

[0069] Figure 4B An example timing diagram 400B of a block-level filter pipeline 400A is shown in accordance with an embodiment of the present disclosure. Figure 4B In the example shown in FIG4 , the entire image or region of the image is divided into four blocks: Block 0-Block 3. Each of the four blocks traverses the filter pipeline 400A and is processed sequentially by DF 411, SAO 413, and ALF 415. Specifically, during time period T41, reconstructed samples for Block 0 are generated and subsequently stored in Block Buffer 410. During time period T42, DF 411 processes Block 0 while Block 1 is being reconstructed. The filtered Block 0 is then stored in Block Buffer 412. During time period T43, SAO 413 processes Block 0. The filtered Block 0 is then stored in Block Buffer 414. Simultaneously, DF 411 processes Block 1 while Block 2 is being reconstructed.

[0070] During time period T44, ALF 415 begins processing block 0. Simultaneously, DF 411 and SAO 413 process blocks 2 and 1, while block 3 is being reconstructed. Similarly, during time period T45-T47, blocks 1-3 may pass through each stage of filter pipeline 400A, resulting in a series of processed blocks. As shown, during time period T42-T46, one or more stages (filters) in filter pipeline 400A operate in parallel. Conversely, during Figure 3BIn the example shown in FIG4 , only one stage (filter) in the filter pipeline 300A is running at any given time. Therefore, the processing latency of the block-level pipeline 400A has been reduced compared to the image-level pipeline 300A. Furthermore, the block buffers 410 , 412 , and 414 in the block-level pipeline 400A can have a smaller buffer size (reduced from the entire image to blocks) than the frame buffers 310 , 312 , and 314 in the image-level pipeline 400B.

[0071] Figure 5 An example of a DF / SAO ready area 510 according to an embodiment of the present disclosure is shown. In one example, the DF / SAO ready area 510, also referred to as a DF / SAO ready block, is Figure 1 The output of SAO 132 in this example includes filtered samples processed successively by DF 130 and SAO 132. Figure 5 A plurality of blocks (such as CTU blocks) 501-509 partitioned from an image are shown. The blocks partitioned from the image including the plurality of blocks 501-509 are processed one by one in a raster scan order, for example, row by row, from left to right in the encoder 100. The reconstructed sample blocks are generated from the second adder 122 and received one by one in the same order at the filter pipeline 136.

[0072] In one example, blocks 501-505 have already been processed by SAO 132, while blocks 506-509 will be processed by SAO 132. DF / SAO-ready area 510 is the output of SAO 132 and will be provided to ALF 134 for further processing. As shown, due to the deblocking and SAO processing, DF / SAO-ready area 510 may be shifted several rows of pixels upward and left relative to block 505. For example, when processing block 505 of reconstructed samples, DF 130 requires reconstructed samples from block 508 in order to perform deblocking filtering on the bottom horizontal block boundary 531. However, the reconstructed samples from block 508 to the bottom of block 505 are not available before block 508 is reconstructed. Therefore, several rows of reconstructed samples near boundary 531 in shaded area 520 cannot be processed. Similarly, since reconstructed samples near the right vertical block boundary 532 in block 506 are not yet available, several columns of reconstructed samples near the block boundary 532 in the shadow area 520 cannot be processed by DF 130. Likewise, since SAO filtering is applied after deblocking, SAO filtering cannot be performed on samples in the shadow area 520 when SAO 132 processes block 505.

[0073] The samples in the shaded area 520 may be stored in a buffer of the DF 130 and processed later when samples in blocks 506 or 508 are available. As a result of the above processing, the output area of ​​the SAO 132 may be as follows: Figure 5 The offset region 510 is shown. Generally, the DF / SAO ready region can be configured to have the same size as the blocks 501-509 so that the processing time at each stage of the filter pipeline can be balanced.

[0074] Although the DF / SAO ready region or block in the above example is generated from a filter pipeline including DF and SAO, it should be understood that in other examples, the DF / SAO ready region may refer to a block generated from a filter pipeline including either DF or SAO (or including filters other than DF or SAO). Such a block is used as input to the ALF and processed by the ALF using the various techniques described herein.

[0075] Figure 6 Filter shape 600 of an FIR filter according to an embodiment of the present disclosure is shown. An FIR filter having filter shape 600 can be used at ALF 134 or 234 to generate filtered samples for a target pixel. Filter shape 600 has 17 squares. Each square is indexed by a number i. Each square i corresponds to a sample Si to be filtered corresponding to pixel Pi. Each square i is associated with a filter parameter Ci. Thus, as defined by filter shape 600, pixel P8 ( Figure 6 The shaded square in ( ) is the target pixel (also called target sample), and the filtered sample of the target pixel P8 represented by St(P8) can be calculated using the following expression,

[0076]

[0077] The pixels P0-P16 or samples S0-S16 to be filtered are Figure 6 The area to be filtered (also called the filter input area) is formed in . The area to be filtered includes rows or columns of samples. The number of columns between the target pixel P8 and the right boundary of the area to be filtered is called the right span of the filter shape 600 or the filter having the filter shape 600. Figure 6 In the figure, the right span is denoted as Q. Alternatively, the right span can also be defined as the number of pixels to the right of the target pixel in the area to be filtered. Similarly, the left span P, the lower span S, and the upper span R can also be defined. The left span refers to the number of columns between the target pixel P8 and the left boundary of the area to be filtered; the lower span refers to the number of rows between the target pixel P8 and the bottom boundary of the area to be filtered; and the upper span refers to the number of rows between the target pixel P8 and the upper boundary of the area to be filtered. The left, lower, and upper spans P, S, and R are also in Figure 6 Shown in.

[0078] Although filter shape 600 is shown as a 7×7 cross overlapping a 3×3 square, it should be understood that in various examples, the FIR filter used in ALF 134 or 234 can have various shapes and sizes. For example, the filter shape can be square, diamond, etc., and can have different sizes.

[0079] Figures 7A-7B A first block-level processing technique according to an embodiment of the present disclosure is shown. Figure 7A Two DF / SAO ready blocks 710-720 are shown. The two blocks 710-720 are similar to Figure 5 The DF / SAO ready block 510 in the example can be generated sequentially from the SAO 413 in the filter pipeline 400A. In one example, the block 710 is first generated and stored in the first buffer in the block buffer 414. The ALF 415 processes the block 710 using the FIR filter defined by the filter shape 600. Figure 6 As shown, the ALF processing of the target pixel P8 requires samples from the right Q column and the left P column to be available. Figure 7A As shown, the rightmost Q column of samples in block 711 in block 710 cannot be processed as target pixels using the FIR filter until the next block 720 is available. To address this issue, in one example, ALF 415 is configured to store the rightmost Q column of samples in block 711 in a second buffer (also referred to as a side buffer), for example, in block buffer 414. Furthermore, ALF 415 may also store the P column of samples to the left of block 711 in the second buffer in block buffer 414. This P column of samples will be needed when the samples in block 711 are being processed. As a result, all P+Q columns of samples in block 712 are stored in the side buffer.

[0080] When the next DF / SAO ready block 720 is generated and received at the first buffer of the block buffer 414, the block 712 is combined with the block 720 to form the area to be filtered. Figure 7B , when processing the area to be filtered, pixels within the shaded area 731 (including block 711) can be processed as target pixels. However, pixels in the rightmost Q column cannot be processed as target pixels until the next DF / SAO ready block to the right of block 720 is available.

[0081] exist Figures 7A-7B In the above example, the first block-level processing technique for processing samples near block boundaries is not applied to samples near the upper boundary 701 and the lower boundary 702 in blocks 710 and 720, as shown in FIG. Figure 7AAs shown. In contrast, in one example, when processing samples near the upper boundary 701 and the lower boundary 702, a modified filter shape can be used to avoid filtering samples in rows of blocks above or below blocks 710-720. In this way, the buffer required to store samples from multiple blocks in the row above blocks 710-720 can be avoided. This modification of the filter shape is called a conditional transformation of the filter. For example, when processing samples near the upper boundary 701, Figure 6 The filter shape 600 in FIG. 6 can be modified in such a way that samples 0-4 in the upper portion of the filter shape 600 can be ignored, or sample 2 can be replaced by sample 7; samples 0, 1, and 3 are replaced by sample 8; and sample 4 is replaced by sample 9. As a result, the ALF with the modified filter shape for target sample 8 does not need to process samples above the target sample 8.

[0082] Figure 8 FIG. 8 shows an adaptive loop filtering process 800 according to an embodiment of the present disclosure. The process 800 uses Figures 7A-7B The first block level processing technique described in the example. Process 800 can be performed at ALF 134, 234 or 415. ALF 415 in filter pipeline 400A is used as a reference Figures 7A-7B The process 800 starts from S801 and proceeds to S810.

[0083] At S810, the current sampling block 720 (at Figure 7B The current sample block 720 is received at the ALF 415 and stored in the first buffer of the block buffer 414. The current sample block 720 may be one of a series of blocks in the image. The sequence of blocks in the image may be processed in a horizontal scan order or a vertical scan order. The sequence of blocks may be processed sequentially by the filter pipeline 400A. The current sample block 720 may be a DF / SAO-ready block generated from the SAO 413.

[0084] In S820, based on Figure 7B The ALF process is performed on the target pixel by sampling the current block 720 and the samples in the block 712. The sampled block 712 may be stored in the second buffer (side buffer) of the block buffer 414. The block 712 may include Q+P pixel columns adjacent to the current block 720 in the previous block 710. The target pixel in the process is a pixel in the block 731. Figure 7B As shown, block 731 includes samples of pixels in blocks 720 and 712 except for the rightmost Q column and the leftmost P column within blocks 720 and 712 .

[0085] At S830, the Q+P columns of samples adjacent to the next block (not shown) within the current block 720 are stored in the side buffer. Figure 7BIn the example of , the next block is to the right of block 720. However, in other examples, the position of the previous block or the next block relative to the current block 720 may be different from Figure 7B For example, it depends on the processing order of the blocks in the image. For example, the previous block may be to the right of the current block 720, and the next block may be to the left of the current block 720.

[0086] In other examples, the current block 720 may be the last block in a scan row or scan column. Therefore, in S830, the samples of the rightmost Q+P column are not stored in the side buffer. Instead, the modified filter shape can be used to process the rightmost Q column. In other examples, the block 720 may be the first block in a scan row or scan column. Therefore, no block 712 is available. Similarly, the modified filter shape can be used to process the target samples near the left edge of the current block 720.

[0087] Figures 9A-9B A second block-level processing technique according to an embodiment of the present disclosure is shown. Figure 9A Two adjacent DF / SAO ready blocks 910-920 are shown that are generated in a manner similar to the DF / SAO block 510. In one example, the DF / SAO ready blocks 910-920 are sequentially generated by the SAO 413 in the filter pipeline 400A. However, unlike Figures 7A-7B In the example of , SAO 413 does not output DF / SAO ready block 920 as its output region. Instead, SAO 413 outputs extended block 901 as its output region. Extended block 901 includes DF / SAO ready region 920 and a portion of the previous DF / SAO ready block 910 in block 912. Block 912 includes at least the P+Q column samples of the sample column of the extension region 901 (or DF / SAO block 920) closest to the DF / SAO block 910 in block 911. Block 911 and block 920 may form a similar Figure 7B The extended region 901 may then be stored in the block buffer 414 and processed by the ALF 415.

[0088] Figure 9B An extended region 901 and a second extended region 902 adjacent to the first extended region 901 are shown. The second extended region 902 may be output from the SAO 413 in the filter pipeline 400A after the first extended region is output. The two extended regions 901 and 902 may overlap with each other. The overlapping region may include the P+Q columns of samples within the first extended region that are closest to the second extended region 902.

[0089] Figure 10A Shown as Figure 5The same group of blocks 501-509 are shown. However, Figure 5 Unlike the example, after each block 501-509 is processed in the encoder 100 to generate reconstructed video data for the block 501-509, each block 501-509 is divided into a plurality of sub-blocks for filtering processing on the filter pipeline 136, 236 or 400A. Figure 10A As shown, block 505 is divided into four sub-blocks 0-3 for subsequent filtering processing. In one example, the group of blocks 501-509 is a group of blocks corresponding to a CTU partition, and each block 501-509 corresponds to one CTU.

[0090] Figure 10B An example timing diagram 1000B of sub-block level filtering processing according to an embodiment of the present disclosure is shown. The timing diagram 1000B corresponds to Figure 10A , DF 411 processes the deblocking filter processing sub-blocks in a sub-block partitioning manner. As shown in the figure, the encoder 100 processes a sequence of blocks (blocks 505-506), and sequentially generates blocks of reconstructed samples. For example, after receiving the reconstructed video data of block 505 at the block buffer 410 of the filter pipeline 400A, the DF 411 processes the deblocking filter processing sub-blocks in sub-blocks. As shown in the figure, sub-blocks 0-3 are processed sequentially. Each sub-block is provided to the SAO 413 for processing, for example, once the corresponding deblocking processing is completed. Therefore, the SAO413 performs SAO processing sub-block by sub-block to generate DF / SAO ready sub-blocks as shown in the figure. Subsequently, the ALF 415 processes the DF / SAO ready sub-blocks one by one. With Figure 4B Compared to the block-level pipeline processing in the example, the sub-block-level pipeline processing can further reduce the processing delay and reduce the working buffer size from the sample block in the filter pipeline to the sample sub-block. Figure 10B As shown, ALF 415 begins processing samples in block 505 before DF 411 or SAO 413 completes its processing of samples in block 505. Thus, ALF 415 processes sub-blocks 0-1, DF 411 processes sub-blocks 2-3 in parallel, and SAO 413 processes sub-blocks 1-2 in parallel.

[0091] In other examples, the order in which sub-blocks are processed at DF 411 and SAO 413 may differ from the order at ALF 415. For example, at DF 411 and SAO 413, sub-blocks 0-3 may be processed in the order 0-2-1-3, while at ALF 415, the same sub-blocks 0-3 may be processed in another order 0-1-2-3. Furthermore, in some examples, the partitioning of sub-blocks at DF 411 and SAO 413 may differ from the partitioning of sub-blocks at ALF 415. For example, for DF 411 and SAO 413, sub-blocks 0-1 may be processed as one block, while sub-blocks 2-3 may be processed as another block. In contrast, for ALF 415, sub-blocks 0-3 may be processed separately.

[0092] Figures 11A-11B 10A-10B . Filter pipeline 400A is used as an example to illustrate the third block-level processing technique. Figure 11A A sequence of DF / SAO-ready sub-blocks 1101-1116 is shown, each having its own processing order, generated from the SAO 413 at the filter pipeline 400A and subsequently processed as sub-blocks at the ALF 415. The sequence of DF / SAO-ready sub-blocks 1101-1116 can each correspond to one of the sub-block sequences partitioned from a block, such as one of the blocks 501-509. As shown, the sequence of sub-blocks 1101-1116 is arranged into four rows 1121-1124 and four columns 1131-1134 and processed in a horizontal scan order, such as from left to right and from top to bottom.

[0093] To process sub-blocks 1101-1116, block buffer 414 may be configured to include a first buffer, a second buffer 1141 (referred to as a top buffer), and a third buffer 1142 (referred to as a left buffer). The first buffer may be used to store one of sub-blocks 1101-1116 received from SAO 413. Top buffer 1141 may be used to store samples of the bottom R+S rows of a sub-block in one of rows 1121-1124. Left buffer 1142 may be used to store samples of the rightmost P+Q columns of a sub-block in one of rows 1121-1124.

[0094] For example, the ALF 415 processes the sub-blocks 1101-1104 one by one during the adaptive filtering process. The ALF 415 may first process the sub-block 1101. As described above, assuming that Figure 6For an FIR filter, the rightmost Q columns of samples cannot be processed due to the unavailability of the next sub-block 1102, and the bottom S columns of samples cannot be processed due to the unavailability of the next row of sub-blocks 1105. Therefore, after sub-block 1101 is processed, ALF 415 may store the bottom R+S rows of samples in sub-block 1101 in top buffer 1141 and store the rightmost P+Q columns of samples in sub-block 1101 in left buffer 1142.

[0095] ALF 415 can then continue processing sub-block 1102. The rightmost P+Q columns of samples from sub-block 1101 stored in left buffer 1142 can then be combined with sub-block 1102 for corresponding ALF processing. After sub-block 1102 is processed, ALF 415 can store the bottom R+S rows of samples from sub-block 1102 in top buffer 1141 and store the rightmost P+Q columns of samples from sub-block 1102 into left buffer 1142, replacing the previously stored P+Q columns of samples from sub-block 1102. Sub-blocks 1103-1104 can be processed in a similar manner. As a result, top buffer 1141 can now store the bottom R+S rows of samples from sub-blocks 1101-1104, while left buffer 1142 can store the rightmost P+Q columns of samples from 1104.

[0096] In a similar manner, sub-blocks 1105-1116 can then be processed by the ALF 415. During this processing, the bottom R+S row samples stored in a row of the top buffer can be used to process the sub-block in the next adjacent row of the sub-block. Simultaneously, the bottom R+S row samples in the next adjacent row can replace the bottommost R+S row samples in the row. In one example, the storage of the bottom R+Q row samples is not performed for the last row 1124. Furthermore, after performing ALF processing on all sub-blocks 1101-1116, the rightmost P+Q column samples of sub-blocks 1104, 1108, 1112, and 1116 in the rightmost column 1134 are stored in the left buffer 1142. The samples in the left buffer 1142 can then be reserved for processing the next set of DF / SAO-ready sub-blocks to the right of the current DF / SAO-ready sub-block 1101-1116.

[0097] Figure 11B Shown as Figure 11A The same sequence of DF / SAO ready sub-blocks 1101-1116 is shown, however, they are processed in vertical scan order, e.g., from top to bottom, from left to right. Figure 11BIn the example of FIG, a top buffer 1151 and a left buffer 1152 may be employed. However, the size of the top buffer 1151 is further reduced compared to the top buffer 1141. Specifically, during the adaptive filtering process of processing sub-blocks 1101-1116 in vertical scan order, the top buffer 1151 may be used to store the bottom R+S row samples of the sub-block of one of the columns 1131-1134. The left buffer 1152 may be used in a manner similar to the left buffer 1142 to store samples of the rightmost P+Q columns of each sub-block. At the end of the adaptive filtering process, the samples in the left buffer 1142 may similarly be retained for processing the next set of DF / SAO-ready sub-blocks to the right of the current DF / SAO-ready sub-block 1101-1106.

[0098] In other examples, the order in which the sub-blocks are processed may be different from Figures 11A-11B For example, the sub-blocks may be processed in a zig-zag order. However, Figures 11A-11B Described method.

[0099] Figure 12 An ALF 1200 according to an embodiment of the present disclosure is shown. The ALF 1200 may employ one-dimensional (1D) or two-dimensional (2D) data reuse technology to reduce data access time. The ALF 1200 may include a block buffer 1210 and an ALF circuit 1230. The block buffer 1210 is configured to store pre-ALF data. The pre-ALF data may include a DF / SAO ready block currently being processed, and one or more side buffers (e.g., a top buffer, a left buffer, etc.) for storing P+Q column or R+S row samples from a previous DF / SAO ready block as described above. As an example, the block buffer 1210 includes a block 1220 of pre-ALF data. The ALF circuit 1230 is configured to receive samples from the block buffer 1210 and perform ALF processing on a target pixel in the block 1220. Assume that the ALF circuit 1230 uses a DF / SAO ready block having Figure 6 Therefore, during ALF processing of a target pixel, an FIR filter is applied to pixel samples within the area to be filtered surrounding the target pixel, and the filtered samples can be calculated based on Expression (2).

[0100] In one example, the ALF 1230 uses a 1D data reuse technique to calculate filtered samples for two adjacent pixels P1 and P2, such as Figure 12. The two pixels are distributed along the horizontal dimension. As shown, the adjacent samples for the ALF processing of pixel P1 are within area 1221 surrounded by the bold solid line, while the adjacent samples for the ALF processing of pixel P2 are within the shaded area 1222. In the first scenario, the samples in block 1220 are read row by row from row buffer 1210 (row scan), and the samples in columns C1-C8 in rows R1-R7 are received at the ALF circuit. Based on the received samples, ALF circuit 1230 can calculate two filtered samples corresponding to samples P1 and P2. Assuming that reading one or several columns of samples from block buffer 1210 requires one clock cycle, seven clock cycles are required to access the adjacent samples of two pixels P1 and P2. In the second scenario, the samples in block 1220 can be read column by column from block buffer 1210 (column scan). Therefore, accessing the samples in rows R1-R7 and columns C1-C8 may require eight clock cycles.

[0101] In another example, the ALF 1230 uses a 2D data reuse technique to calculate filtered samples of three adjacent pixels P1, P2, and P3, as shown in FIG. Figure 12 The three target pixels are distributed in two dimensions: horizontal and vertical. Specifically, samples of columns C1-C8 and rows R1-R8 can be read from block buffer 1210 via row scanning or column scanning. ALF circuit 1230 can calculate three filtered samples for target pixels P1, P2, and P3 based on the obtained samples. Therefore, eight clock cycles are required to obtain ALF-processed samples for the three target pixels.

[0102] Figure 13 FIG. 1 shows a first ALF 1300 according to an embodiment of the present disclosure. The ALF 1300 implements a 1D data reuse technique. The ALF 1300 includes a block buffer 1310 and a multiple stage pipeline filter (MSPF) circuit 1330. The block buffer 1310 is configured to store blocks 1320 of pre-ALF data. The MSPF circuit 1330 implements a 1D data reuse technique. Figure 6 The MSPF 1330 has a filter shape with left, right, top, and bottom spans of P, Q, R, and S. Specifically, the MSPF 1330 has seven stages 1361-1367 (R+S+1). Each stage includes a multiplication-addition (MA) circuit 1351-1357 and a delay element 1341-1347. Each delay element 1341-1347 can be a shift register or other type of circuit, and when triggered by a control clock signal 1373, can store the output of the MA circuit 1351-1357 and output the stored value to the next stage.

[0103] As shown, each MA circuit 1351-1357 may take one or more samples from input line 1371 as a first input and one or more filter parameters as a second input, and accordingly calculate the sum of products of the samples and the filter parameters. In addition, in each MA circuit of stages 1362-1367, the sum calculated from the previous stage may be taken as a third input and added to the sum of products. For example, the MA circuits 1351-1357 of each stage 1361-1367 may calculate according to the following expression,

[0104] Level 1361: Sd x C0;

[0105] Level 1362: Sd x C1 + Sum 1;

[0106] Level 1363: Sc x C2+Sd x C3+Se x C4+Sum 2;

[0107] Level 1364:Sa x C5+Sb x C6+Sc x C7+Sd x C8+Se x C9+Sf x C10+Sg xC11+Sum3;

[0108] Level 1365: Sc x C12+Sd x C13+Se x C14+Sum 4;

[0109] Level 1366: Sd x C15 + Sum 5;

[0110] Level 1367: Sd x C16 + Sum 6.

[0111] In the above expression, C1-C16 corresponds to Figure 6 Sa-Sg represent the filter parameters of the filter shape. Sa-Sg represent the samples in columns ag of one row of L1-L9 read from the pre-ALF ALF data block 1320. Sum1-Sum6 correspond to the outputs of the preceding MA circuit.

[0112] In operation, samples are read row by row from block 1320 in synchronization with clock signal 1373. During each read operation, P+Q+1 samples comprising rows Sa-Sg are provided to input line 1371. Thus, each MA circuit 1351-1357 calculates a sum based on two (first stage 1361) or three (stages 1362-1367) inputs. This sum is provided as input to each corresponding delay element 1341-1347. Then, triggered by clock signal 1373, the calculated sum for each stage is shifted to the next adjacent stage as input to the MA circuit at the next stage. Next, the next row of samples Sa-Sg can be provided to input line 1371. Similarly, the sum of the current sample on input line 1371 and the sum output from the previous stage can be obtained at each stage. Subsequently, when triggered by clock signal 1373, the newly calculated sum can be shifted to the next stage.

[0113] Therefore, when samples in the lines L1-L7 are received row by row along input line 1371, filtered samples corresponding to target pixel P1 can be obtained at output terminal 1372 of MSPF circuit 1330. When samples of another row (L8) are provided, filtered samples corresponding to target pixel P2 can be obtained at output terminal 1372. When samples are continuously input row by row along the vertical direction in block 1320, target pixels (e.g., P3) below pixel P2 in column d can be continuously obtained.

[0114] Figure 14 A second ALF 1400 according to an embodiment of the present disclosure is shown. ALF 1400 implements 2D data reuse techniques. Second ALF 1400 includes a block buffer 1410 and a filter circuit 1430. Block buffer 1410 is similar to block buffer 1310 and is configured to store pre-ALF data. As shown, a block 1420 of pre-ALF samples is stored in block buffer 1410. Filter circuit 1430 is configured to receive samples from block buffer 1410 and perform ALF processing on target pixels in block 1420. Figure 6 The FIR filter is used for ALF processing. The filter circuit 1430 includes two groups of MSPF circuits 1431-1432. Each MSPF circuit 1431-1432 can be similar in function and structure. Figure 13 MSPF circuit 1330 in the example.

[0115] In operation, samples are read from block 1420 row by row and applied to input line 1470 coupled to filter circuit 1430. However, each row of samples includes P+Q+2 samples in columns ah. Furthermore, samples Sa-Sg are received at input line 1471 to first MSPF circuit 1431 and provided to first MSPF circuit 1431. Conversely, samples Sb-Sh are received at input line 1473 to second MSPF circuit 1432 and provided to second MSPF circuit 1432. Thus, since samples Sa-Sh are received continuously row by row at filter circuit 1430, a first column of target pixels, such as pixels P1-P3, and a second column of target pixels, such as pixels Pa-Pc, can be obtained in parallel from outputs 1472 and 1474, respectively.

[0116] although Figure 14 In the example, two MSPF circuits are shown to implement 2D data reuse. It is understood that more than two MSPF circuits can be used in other examples. For example, N MSPF circuits operating in parallel can be included in the filter circuit 1430, and each row of samples read from the block 1420 can include P+Q+N samples. Therefore, as a result of the above-mentioned adaptive loop filtering process, Figure 14 The filtered samples of N columns of target pixels adjacent to each other in the horizontal direction in the example.

[0117] Figure 15 FIG. 1 shows a third ALF 1500 according to an embodiment of the present disclosure. The ALF 1500 implements a 1D data reuse technique. The ALF 1500 may include a block buffer 1510 and a filter circuit 1530. The block buffer 1510 is similar to the block buffer 1310 or 1410 and is configured to store pre-ALF data. As shown, a sample block 1520 is stored in the block buffer 1510. The filter circuit 1530 is configured to receive samples from the block buffer 1510 and perform ALF processing on target pixels in the block 1520. In one example, the ALF processing is performed using Figure 6 The filter circuit 1530 may include a register array 1540 and a multiplication-addition (MA) circuit 1550 .

[0118] Register array 1540 is configured to store rows of samples received from block buffer 1510. Specifically, register array 1540 may include R+S+1 rows of shift registers labeled R1-R7. In one example, each of the first S+1=4 rows R1-R4 includes P+Q+1=7 registers, while rows R5-R7 each include a number of registers corresponding to the number of adjacent samples in the corresponding row in the upper portion of filter shape 600. Controlled by a clock signal, the samples stored in each row of registers can be shifted to the next row of shift registers. In this way, samples can be received row by row at the input of the shift register in row R1 and pushed down row by row to the registers below.

[0119] MA circuit 1550 is configured to receive samples from register array 1540 and calculate filtered samples accordingly. Figure 6 For different filters adopted by ALF 1500, MA circuit 1550 may include different circuits for calculation.

[0120] In operation, controlled by the clock signal, the samples Sa-Sg in the rows L1-L7 in the block 1520 can be received and stored row by row at the register array 1540. As a result, for example, the sample Sd in the row L1 in the block 1520 can be stored at the shift register at the row R7 in the register array 1540, while the samples Sa-Sg in the row L7 can be stored at the shift register at the row R1. Thereafter, using the samples stored in the shift register in the region 1541 in the register array 1540 as input, the MA circuit 1550 performs a shift operation on the shift register. Figure 15 The first calculation and first filtered sample are performed for the first target pixel P1 shown. Subsequently, a row of samples in row L8 can be read and shifted into register array 1540. The MA circuit 1550 can then perform a second calculation to obtain a second filtered sample for the second target pixel P2. In this manner, as samples are received and stored row by row into register array 1540, a sequence of filtered samples for the target pixel below P2 (e.g., P3) in column d in block 1520 can be continuously obtained.

[0121] In one example, register array 1540 includes Figure 15The number of R+S+1 rows shown is greater than the number of R+S+N rows. Therefore, filter circuit 1530 includes N sets of MA circuits 1550. In operation of this configuration, in an initial phase, R+S+N sample lines can first be stored in register array 1540, and then the N sets of MA circuits 1550 can perform calculations in parallel based on adjacent samples of the corresponding target pixel. Then, in a second phase, N more new sample lines can be retrieved and stored in the register array. Thereafter, another set of N filtered samples can be calculated in parallel on the N sets of MA circuits 1550.

[0122] Figure 16 FIG. 1 shows a fourth ALF 1600 according to an embodiment of the present disclosure. The ALF 1600 implements a 2D data reuse technique. The ALF 1600 includes a block buffer 1610 and a filter circuit 1630. The block buffer 1610 is similar to the block buffer 1510 and is configured to store pre-ALF data. As shown, samples of block 1620 are stored in the block buffer 1610. The filter circuit 1630 is configured to receive samples from the block buffer 1610 and calculate filtered samples accordingly. Assuming the use of Figure 6 The filter circuit 1630 may include a register array 1640 and two MA circuits 1651-1652. The register array 1640 is similar to the register array 1540 in terms of function and structure. However, compared with the register array 1540, each row of the register array 1640 includes one more shift register. The two MA circuits 1651-1652 are similar to the register array 1540 in terms of function and structure. Figure 15 MA circuit 1650 in the example.

[0123] In operation, the samples in block 1620 may be read out row by row, and each row may include P+Q+2 samples. Figure 15 Compared to the example, one more sample Sh is read per row. In the initial stage, R+S+1=7 rows of samples can be received and stored in the register array 1640. Thereafter, the two groups of MA circuits 1651-1652 can calculate the filtered samples P1 or Pa of each target pixel in parallel, such as Figure 16 The calculation can be based on the adjacent samples of each corresponding target pixel P1 or Pa, as shown in Figure 6 The example defined is stored in register array 1640. During the second phase, a row of samples in row L8 in block 1620 can be received and stored in register array 1640. Then, two MA circuits 1651-1652 can calculate two filtered samples of target pixels P2 and Pb, as shown in FIG. Figure 16Subsequently, sample rows can be read continuously from block 1620 and stored row by row in register array 1640. For each row of samples, two filtered samples of two adjacent target pixels in columns d and e in block 1620 can be obtained, such as pixels P3 and Pc.

[0124] although Figure 16 In the example shown, two MA circuits 1651-1652 are used to implement 2D data reuse. It is understood that more than two MA circuits may be used in other examples. For example, N MA circuits may be included in the filter circuit 1630 operating in parallel, and each row of samples read from block 1420 may include P+Q+N samples. Therefore, Figure 14 In this example, filtered samples of N columns of target pixels adjacent to each other in the horizontal direction may be obtained in parallel based on the samples currently stored in the register array 1640 .

[0125] Figure 17 FIG1 shows a first loop filter circuit 1700 according to an embodiment of the present disclosure. The filter circuit 1700 includes a filter pipeline 1740. The filter pipeline 1740 may perform functions similar to those of the filter pipelines 136, 236, or 400A. In one example, the filter pipeline 1740 includes a DF 1710, a SAO 1720, and an ALF 1730. In terms of function and structure, the DF 1710 and the SAO 1720 may be similar to the DF 411 and the SAO 413. The ALF 1730 may be an ALF that implements a one-dimensional or two-dimensional data reuse technique. For example, the ALF 1730 may be a Figure 13-16 One of the ALFs described in .

[0126] In one example, the filter pipeline 1740 operates at a block level. For example, a block 1750 of reconstructed video data may be generated and received as input to the filter pipeline 1740. The block 1750 may be one of a sequence of blocks segmented from an image. In one example, the partitioning of the sequence of blocks may coincide with the partitioning of CTUs. The filter pipeline 1740 may then operate in a manner similar to Figures 4A-4B Block 1750 and other blocks are processed in the manner of block-level pipeline processing described in the example. Alternatively, reference Figures 7A-7B 、 Figure 8 and Figures 9A-9B The block-level processing technique described is used to process each sample block.

[0127] In another example, the filter pipeline 1740 operates at a sub-block level. For example, the sampled block 1750 is further divided into sub-blocks 1751 to be processed at the filter pipeline. For example, the sub-blocks 1751 can be processed at the same level as Figures 10A-10B The sub-block level pipeline processing is similar to that described in . In addition, refer to Figures 11A-11BThe block processing technique described may be used to process each sub-block of samples.

[0128] Furthermore, in an alternative example, ALF 1730 may include two or more sets of ALF circuits capable of 1D or 2D data reuse. For sample blocks or sub-blocks received from SAO 1720, a scheduler may be used to distribute the received blocks to two or more sets of ALF circuits, allowing for parallel processing of two or more sample blocks or sub-blocks. When the throughput of the ALF circuits is significantly lower than that of the pre-filter circuits, scheduling can balance pipeline processing time. Otherwise, if the throughput of the ALF circuits and pre-filter circuits is similar, the received blocks or sub-blocks may be further divided into more sub-blocks by the scheduler to balance pipeline processing time.

[0129] Figure 18 FIG. 1 shows a second loop filter circuit 1800 according to an embodiment of the present disclosure. The filter circuit 1800 may include a first filter pipeline 1811 and a second filter pipeline 1821. Each of the first and second filter pipelines 1811-1821 may be similar to Figure 17 1821. In the example, the filter pipeline 1740 is shown. Therefore, the description of the first and second filter pipelines 1811-1821 is omitted. In operation, the two filter pipelines 1811-1821 can be operated in parallel to further improve the processing speed of the loop filtering process. For example, reconstructed video data blocks can be generated sequentially in the encoder 100 or the decoder 200. When the reconstructed video data block 1850 is received, the block 1850 can be further divided into two parts. Each part is then processed in parallel by the two filter pipelines 1811-1821. For filtering each part of the sample, Figures 10A-10B Sub-block level pipeline processing in Figures 11A-11B The corresponding block-level processing techniques in can be applied to each filter pipeline 1810-1820.

[0130] Furthermore, in one example, ALF 1810 in filter pipeline 1811 may include two or more sets of ALF circuits capable of 1D or 2D data reuse. Thus, upon receiving data from a preceding filter circuit in ALF 1810, one of the two portions processed in filter pipeline 1811 may be further divided into two or more sub-blocks. The two or more sub-blocks may then be processed in parallel by two or more sets of ALF circuits capable of 1D or 2D data reuse. Similarly, ALF 1820 in filter pipeline 1821 may also include two or more sets of ALF circuits capable of one-dimensional or two-dimensional data reuse, which may facilitate parallel ALF processing of target samples from a portion of samples received from a preceding filter circuit.

[0131] Figure 19FIG1 shows an adaptive loop filtering process 1900 in a video coding system according to an embodiment of the present disclosure. The process 1900 may be performed in the ALF 134, 234, 415, 1730, 1810, or 1820 in various embodiments. The process 1900 starts from S1901 and proceeds to S1910.

[0132] At S1910, a sample block is received at an ALF. The sample block may be generated in a filter pipeline from a pre-filter circuit (e.g., DF or SAO). The sample block is one of multiple blocks included in the current image. For example, the received block may be a CTU block or a sub-block of a CTU block.

[0133] At S1920, ALF processing of multiple target samples in the sample block can be performed in parallel. ALF processing can be performed while the pre-stage filter circuit is simultaneously processing another block in the current image. For example, the sample block can be divided into two or more sub-blocks, and the ALF processing can be performed using Figure 17-18 For example, the ALF 1730, 1810, or 1820 may include two or more sets of filter circuits capable of 1D or 2D data reuse, which may operate simultaneously so that two or more target samples may be processed in parallel.

[0134] If necessary, at S1930, the first samples are stored in a buffer. Each first sample has a filter input region defined by a filter shape, the filter input region including at least one unreceived sample. At S1940, a second sample included in the filter input region of the first sample is stored in the buffer. Figure 7A In the example of FIG, the samples in block 711 correspond to the first samples, and the samples in block 712 other than block 711 correspond to the second samples.

[0135] At S1950, a next sample block adjacent to a sample block in the current image is received. The next sample block may also be generated from a pre-stage filter circuit. At S1960, the plurality of first samples and the plurality of second samples are read from a buffer as necessary. At S1970, an ALF process is performed on a portion of samples in a block formed by the next adjacent block and the first samples. Figure 7B In the example of , block 720 is the next block, and the rightmost Q sample columns cannot be processed until another next block is received. The process proceeds to S1999 and terminates at S1999.

[0136] Although various aspects of the present disclosure have been described in conjunction with specific embodiments provided as examples, substitutions, modifications, and variations of the examples may be described. Therefore, the embodiments set forth herein are intended to be illustrative rather than restrictive. Changes may be made without departing from the scope of the claims set forth below.

Claims

1. An adaptive loop filtering method for a video encoding and decoding system, characterized in that: include: storing a current block of samples generated from a preceding filter circuit in the filter pipeline into a block buffer; Processing samples in a region to be filtered using an adaptive loop filter, the adaptive loop filter having left, right, top, and bottom spans of P, Q, R, and S samples in length, respectively, the region to be filtered being part of a first combined region including the samples in the current block and P+Q columns of samples of a first block previously received from the pre-filter circuit in the filter pipeline, the P+Q columns of samples of the first block being stored in a side buffer and adjacent to the current block in a current image, wherein the processing comprises: reading a set of adjacent samples in the area to be filtered from the block buffer or the side buffer into an adaptive loop filter circuit, and The adaptive loop filter circuit performs adaptive loop filtering on two adjacent target samples in the set of adjacent samples to generate two adjacent filtered samples based on the set of adjacent samples; as well as P+Q column samples of the current block are stored in the side buffer, the P+Q column samples of the current block being adjacent to a second block that has not yet been received from the pre-stage filter circuit in the filter pipeline.

2. The adaptive loop filtering method according to claim 1, wherein: The front-stage filter circuit is a deblocking filter circuit or a sample adaptive offset filter circuit.

3. The adaptive loop filtering method according to claim 1, wherein: Also includes: receiving a second sampling block adjacent to the current block; as well as The samples that are a subset of samples in a second combined region including the samples in the second block and the P+Q column samples of the current block stored in the side buffer are processed using adaptive loop filtering.

4. The adaptive loop filtering method according to claim 1, wherein: The current block is one of a plurality of blocks included in the current image, and the plurality of blocks are coding tree unit blocks or sub-blocks of the coding tree unit blocks.

5. The adaptive loop filtering method according to claim 1, wherein: The side buffer includes a first buffer and a second buffer for storing the P+Q column samples of the current block, and the adaptive loop filtering method further includes: R+S column samples of the current block are stored in the second buffer, the R+S column samples being adjacent to a third block that has not been received from the pre-stage filter circuit in the filter pipeline.

6. The adaptive loop filtering method according to claim 1, wherein: The reading operation includes: reading the group of adjacent samples row by row as input to a multi-stage pipeline filter circuit, and The two adjacent filtered samples are generated continuously from the multi-stage pipeline filter circuit.

7. The adaptive loop filtering method according to claim 1, wherein: The reading operation includes: reading the group of adjacent samples row by row as input to a first multi-stage pipeline filter circuit and a second multi-stage pipeline filter circuit operating in parallel with the first multi-stage pipeline filter circuit, and The two adjacent filtered samples are generated at the first multi-stage pipeline filter circuit and the second multi-stage pipeline filter circuit, respectively, based on the set of adjacent samples.

8. The adaptive loop filtering method according to claim 1, wherein: The reading operation includes shifting the set of adjacent samples row by row into a shift register array having a plurality of shift register rows, each row of the plurality of shift registers storing a row of shifted samples, and Performing adaptive loop filtering processing in the adaptive loop filter circuit includes, calculating a first filtered sample of the two adjacent filtered samples based on a plurality of samples currently stored in the shift register array by a multiplication-addition circuit coupled to the shift register array to receive the plurality of samples from the shift register array, shifting the next row of samples in the group of adjacent samples into the shift register array, and A second filtered sample of the two adjacent filtered samples is calculated in the multiplication and addition circuit based on the plurality of samples currently stored in the shift register array.

9. The adaptive loop filtering method according to claim 1, wherein: The reading operation includes shifting the group of adjacent samples row by row into a shift register array having a plurality of shift register rows, each shift register row storing a row of samples, and Performing adaptive loop filtering processing in the adaptive loop filter circuit includes, calculating a first filtered sample of the two adjacent filtered samples based on a first set of samples currently stored in the shift register array by a first multiplication-addition circuit coupled to the shift register array to receive the first set of samples from the shift register array, and Based on a second group of samples currently stored in the shift register array, second filtered samples of the two adjacent filtered samples are calculated in parallel with the calculation of the first filtered samples by a second multiplication-addition circuit coupled to the shift register array to receive the second group of samples from the shift register array, the second group of samples including a portion of the first group of samples.

10. The adaptive loop filtering method according to claim 1, wherein: Also includes: Divide the area to be filtered into a first sub-block and a second sub-block, wherein said processing comprises, performing a first adaptive loop filtering process on a first target sample in the first sub-block samples; as well as A second adaptive loop filtering process is performed on a second target sample in the second sub-block samples, wherein the first adaptive loop filtering process and the second adaptive loop filtering process are performed in parallel.

11. An adaptive loop filter circuit for a video encoding and decoding system, characterized in that: include: a block buffer configured to store a current block of samples generated from a preceding filter circuit in the filter pipeline; a filter circuit configured to process samples in the region to be filtered using an adaptive loop filter having left, right, upper, and lower spans of P, Q, R, and S samples in length, respectively; as well as a side buffer configured to store P+Q column samples of a first block previously received from the pre-stage filter circuit in the filter pipeline, the P+Q column samples of the first block being adjacent to the current sample block in a current image, in: The filter circuit further reads a group of adjacent samples in the to-be-filtered area from the block buffer or the side buffer into an adaptive loop filter circuit, and performs adaptive loop filtering on two adjacent target samples in the group of adjacent samples to generate two adjacent filtered samples based on the group of adjacent samples. wherein the area to be filtered is part of a first combined area, the first combined area including the samples in the current sample block and the P+Q column samples of the first block, and After the samples in the area to be filtered are processed, P+Q columns of samples of the current sample block are stored in the side buffer, the P+Q columns of samples of the current sample block being adjacent to a second block that has not yet been received from the pre-stage filter circuit in the filter pipeline. 12 . The adaptive loop filter circuit according to claim 11 , wherein the front-stage filter circuit is a deblocking filter circuit or a sample adaptive offset filter circuit.

13. The adaptive loop filter circuit according to claim 11, wherein: The block buffer is configured to receive and store a second sample block adjacent to the current sample block, and The filter circuit is configured to process samples as a subset of samples in a second combined region including samples in the second block and the P+Q column samples of the current sample block stored in the side buffer using adaptive loop filtering.

14. The adaptive loop filter circuit according to claim 11, wherein: The current sampling block is one of a plurality of blocks included in the current image, and the plurality of blocks are coding tree unit blocks or subblocks of the coding tree unit blocks.

15. The adaptive loop filter circuit according to claim 11, wherein: The side buffer includes a first buffer storing the P+Q column samples of the current sample block and a second buffer storing R+S column samples of the current sample block, wherein the R+S column samples are adjacent to a third block, wherein the third block is not received from the previous filter circuit in the filter pipeline.

16. The adaptive loop filter circuit according to claim 11, wherein: The filter circuit includes a multi-stage pipeline filter circuit configured to: receiving the set of adjacent samples from the block buffer or the side buffer row by row; as well as The two adjacent filtered samples are generated continuously.

17. The adaptive loop filter circuit according to claim 11, wherein: The filter circuit includes a first multi-stage pipeline filter circuit and a second multi-stage pipeline filter circuit operating in parallel with each other, and the first multi-stage pipeline filter circuit and the second multi-stage pipeline filter circuit are configured to: receiving as input the set of adjacent samples row by row from the block buffer or the side buffer, and The two adjacent filtered samples are generated in parallel at the first multi-stage pipeline filter circuit and the second multi-stage pipeline filter circuit, respectively, based on the received set of adjacent samples.

18. The adaptive loop filter circuit according to claim 11, wherein: The filter circuit comprises: a shift register array configured to receive the set of adjacent samples from the block buffer or the side buffer on a row-by-row basis, wherein each row in the shift register array stores a row of samples, and each row register is updated with samples stored in an adjacent row register as a new row of samples is shifted into the first row of the shift register array; a multiplication and addition circuit connected to the shift register array and configured to calculate a first filtered sample of the two adjacent filtered samples based on a plurality of first samples stored in the shift register array, and then calculate a second filtered sample of the two adjacent filtered samples based on a plurality of second samples stored in the shift register array, wherein the plurality of second samples are obtained by shifting a new sample line to the first row of the shift register array.

19. The adaptive loop filter circuit according to claim 11, wherein: The filter circuit comprises: a shift register array configured to receive the set of adjacent samples from the block buffer or the side buffer on a row-by-row basis, wherein each row in the shift register array stores a row of samples, and each row register is updated with a plurality of samples stored in an adjacent row register when a new line of samples is shifted into the first row of the shift register array; a first multiply-add circuit coupled to the shift register array and configured to receive a first set of samples currently stored in the shift register array; and a second multiply-add circuit coupled to the shift register array and configured to receive a second set of samples currently stored in the shift register array, the second set of samples including a portion of the first set of samples, The first multiplication-addition circuit and the second multiplication-addition circuit are further configured to calculate the two adjacent filtered samples in parallel based on the corresponding first group of samples and the second group of samples currently stored in the shift register array.

20. The adaptive loop filter circuit according to claim 11, wherein: The filter circuit is further configured to: performing a first adaptive loop filtering process on a first target sample in a first sub-block divided from the area to be filtered; and A second adaptive loop filtering process is performed on a second target sample in a second sub-block divided from the to-be-filtered area, wherein the first adaptive loop filtering process and the second adaptive loop filtering process are performed in parallel.

Images