Inter prediction with filtering

Abstract

Decoding the current block using inter prediction with filtering includes identifying an intermediate predicted block for the current block using a motion vector and a reference frame. Filter coefficients are obtained for the filter. The filter coefficients are obtained using reconstructed pixels and second reconstructed pixels. The reconstructed pixels are in the vicinity of the current block. The second reconstructed pixels are in the vicinity of the intermediate predicted block. The filter is applied to the intermediate predicted block to obtain a final predicted block. The current block is reconstructed using the final predicted block. Encoding the current block includes obtaining an intermediate motion vector for the current block. The filter coefficients are obtained by minimizing an error metric between the predicted block corresponding to the intermediate motion vector and the current block. A motion vector is obtained for the current block by refining the intermediate motion vector using the filter coefficients.

Description

[Background technology]

[0001] A digital video stream may display video using a series of frames or still images. Digital video can be used for a variety of applications, such as video conferencing, high-definition video entertainment, video advertising, or sharing user-generated videos. Digital video streams can contain large amounts of data, consuming significant computing or communication resources of computing devices to process, transmit, or store the video data. Various approaches, such as compression and other encoding techniques, have been proposed to reduce the amount of data in video streams.

[0002] Motion estimation and compensation based coding may be performed by dividing a frame or image into blocks that are predicted based on one or more predictive blocks of a reference frame. The differences between the blocks and the predictive blocks (i.e., residual errors) are compressed and coded in a bitstream. A decoder uses the differences and the reference frames to reconstruct the frame or image. Summary of the Invention

[0003] Disclosed herein are aspects, features, elements, and implementations for encoding and decoding blocks using segmentation-based parameterized motion models.

[0004] One or more computer systems may be configured to perform particular operations or actions by having software, firmware, hardware, or a combination thereof installed on the system that, when operated, causes the system to perform the actions. One or more computer programs may be configured to perform particular operations or actions by containing instructions that, when executed by a data processing device, cause the device to perform the actions.

[0005] One general aspect includes a method for decoding a current block using inter prediction with filtering. The method also includes identifying an intermediate predicted block for the current block using a motion vector and a reference frame. The method also includes obtaining filter coefficients for a filter, the filter coefficients being obtained using first reconstructed pixels and second reconstructed pixels, the first reconstructed pixels being in a neighborhood of the current block and the second reconstructed pixels being in a neighborhood of the intermediate predicted block. The method also includes applying the filter to the intermediate predicted block to obtain a final predicted block. The method also includes reconstructing the current block using the final predicted block. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. Implementations may include one or more of the following features.

[0006] A method in which the filter contains more than two coefficients.

[0007] The method may include decoding an inter-prediction mode that indicates applying a filter.

[0008] The method may include applying a filter to an intermediate predicted block in response to determining that a block other than the current block is being reconstructed using an inter-prediction mode that indicates applying filtering.

[0009] The method may include decoding, from the compressed bitstream, a radix of filter coefficients to obtain for the filter. The radix of the filter coefficients may be greater than two.

[0010] Obtaining filter coefficients for the filter may include obtaining prediction filter coefficients for filter coefficients among the filter coefficients, decoding coefficient refinement values ​​from the compressed bitstream, and adjusting the prediction filter coefficients using the coefficient refinement values ​​to obtain the filter coefficients. The coefficient refinement values ​​may be used for intermediate predicted pixels to which the filter is applied. The coefficient refinement values ​​may be used to refine coefficients corresponding to nonlinear terms of the filter.

[0011] The filter coefficients may be obtained by minimizing an error metric between the first reconstructed pixel and the second reconstructed pixel. The error metric may be a sum-of-squares error. The filter coefficients may be applied to at least a subset of pixels within a 3×3 neighborhood of the intermediate predicted pixel to obtain predicted pixels of the final predicted block. The at least a subset of pixels within the 3×3 neighborhood of the intermediate predicted pixel may include the intermediate predicted pixel, a pixel above the intermediate predicted pixel, a pixel to the right of the intermediate predicted pixel, a pixel below the intermediate predicted pixel, and a pixel to the left of the intermediate predicted pixel.

[0012] The filter may further include a constant component. The filter may include at least one non-linear component.

[0013] The current block may be a luminance block, and a chroma prediction block for a chroma block corresponding to the current block may be derived from the final prediction block.

[0014] A first filter shape may be used if the current block is a luma block, and a second filter shape different from the first filter shape may be used if the current block is a chroma block.

[0015] One general aspect includes a method used to encode a current block. The method also includes obtaining an intermediate motion vector for the current block. The method also includes obtaining filter coefficients by minimizing an error metric between a predictive block corresponding to the intermediate motion vector and the current block. The method also includes obtaining a motion vector for the current block by refining the intermediate motion vector using the filter coefficients. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. Implementations may include one or more of the following features.

[0016] The filter coefficients may include a first coefficient (a), a second coefficient (b), and a third coefficient (c), and refining the intermediate motion vector using the filter coefficients includes making a first adjustment to a first component of the intermediate motion vector by 2. * b / a and the second adjustment to the second component of the intermediate motion vector is 2 * c / a.

[0017] The method may include encoding at least one of the first adjustment or the second adjustment into a compressed bitstream.

[0018] Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0019] It will be understood that aspects may be implemented in any convenient form. For example, aspects may be implemented by a suitable computer program, which may be embodied on a suitable carrier medium, which may be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). Aspects may also be implemented using a suitable apparatus, which may take the form of a programmable computer executing a computer program configured to implement the methods and / or techniques disclosed herein. For example, a non-transitory computer-readable storage medium may include executable instructions that, when executed by a processor, facilitate the performance of acts operable to cause the processor to perform any of the methods described herein. Aspects may be combined such that functionality described in the context of one aspect can be implemented in other aspects.

[0020] Variations and further details of these methods, techniques, and apparatus can be found in the figures, description, and claims that follow.

[0021] This description makes reference to the accompanying drawings, which are listed below, in which like reference numerals refer to like parts throughout the several views. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a schematic diagram of a video encoding and decoding system; [Figure 2] FIG. 2 is a block diagram illustrating an example of a computing device that can implement a sending station or a receiving station. [Figure 3] FIG. 1 illustrates a video stream that is encoded and then decoded. [Figure 4] FIG. 2 is a block diagram of an encoder according to an embodiment of the present disclosure. [Figure 5] FIG. 2 is a block diagram of a decoder according to an embodiment of the present disclosure. [Figure 6]FIG. 10 is a flowchart of a technique for decoding a current block using inter prediction with filtering. [Figure 7] 1 shows an example of a template of reconstructed pixels. [Figure 8] 1 illustrates an example of a neighborhood of intermediate pixels of an intermediate predicted block. [Figure 9] 10 shows an example of the positions of the left and top samples and the current block samples involved in the cross-component filtering mode. [Figure 10] FIG. 1 is a flowchart diagram of a technique used to encode a current block. DETAILED DESCRIPTION OF THE INVENTION

[0023] As described above, compression schemes for coding video streams may include dividing an image into blocks and generating a digital video output bitstream (i.e., an encoded bitstream) using one or more techniques to limit the information included in the output bitstream. The received bitstream may be decoded to recreate the blocks and source image from the limited information. Encoding a video stream or a portion thereof, e.g., a frame or block, may include using temporal similarities within the video stream to improve coding efficiency. For example, a current block of a video stream may be coded based on identifying differences (residuals) between previously coded pixel values ​​and pixel values ​​in the current block, or between a combination of previously coded pixel values ​​and pixel values ​​in the current block.

[0024] Coding using temporal similarity is known as inter-prediction. Inter-prediction attempts to predict pixel values ​​of a block using possibly displaced block(s) from temporally nearby frames (i.e., reference frames). Temporally nearby frames are frames that appear earlier or later in the video stream than the frame of the block being coded. Inter-prediction can be performed using motion vectors that represent translational motion, i.e., pixel shifts in the x and y axes of a prediction block in a reference frame compared to the block being predicted.

[0025] If the source data is noisy, if there are lighting differences between the current block being predicted and the reference frame used to obtain the predicted block (i.e., also known as the reference block), or if the source data contains motion blur, prediction accuracy may be reduced, and as a result compression performance may also be reduced.

[0026] Embodiments of the present disclosure improve this situation by obtaining an intermediate predicted block for the current block and further filtering pixels of the intermediate predicted block to obtain a (final) predicted block for the current block. The intermediate predicted block may be a reference block in a reference frame. Residual data (i.e., the residual block) may be obtained as a difference (i.e., pixel-by-pixel difference) between the current block and the final predicted block. The residual data may be encoded in a compressed bitstream as described herein. In one example, upon decoding the current block, the decoder similarly applies a filter to the intermediate prediction to obtain the final predicted block, decodes the residual block from the compressed bitstream, and combines the final predicted block and the residual block to reconstruct the current block.

[0027] Given an intermediate pixel at position (x, y) of the intermediate predicted block, a filter is used to obtain the corresponding (i.e., co-located) pixel in the predicted block. The filter may be a weighted combination of intermediate pixels in the neighborhood of the intermediate predicted pixel. Different neighborhoods may be used. The weighted combination may be a linear combination or a non-linear combination (i.e., may include at least one non-linear term). As is well known, the filter uses filter coefficients as weights for different intermediate pixels in the neighborhood. The encoder and decoder derive the filter coefficients using first reconstructed pixels around the current block and second reconstructed pixels around the intermediate predicted block.

[0028] Further details of inter-prediction techniques with current block filtering will be described herein, first with reference to systems in which they may be implemented. Figure 1 is a schematic diagram of a video encoding and decoding system 100. The transmitting station 102 can be, for example, a computer having a hardware internal configuration as described in Figure 2. However, other suitable implementations of the transmitting station 102 are also possible. For example, the processing of the transmitting station 102 can be distributed across multiple devices.

[0029] A network 104 may connect the sending station 102 and the receiving station 106 for encoding and decoding of the video stream. Specifically, the video stream may be encoded at the sending station 102, and the encoded video stream may be decoded at the receiving station 106. The network 104 may be, for example, the Internet. The network 104 may also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular network, or other means of transferring the video stream from the sending station 102 to the receiving station 106, as in this example.

[0030] Receiving station 106 may, in one example, be a computer having the internal hardware configuration illustrated in Figure 2. However, other suitable implementations of receiving station 106 are possible. For example, the processing of receiving station 106 may be distributed across multiple devices.

[0031] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation could omit the network 104. In other implementations, the video stream may be encoded and stored for later transmission to the receiving station 106 or other device having memory. In one implementation, the receiving station 106 receives the encoded video stream (e.g., via the network 104, a computer bus, and / or some other communication path) and stores the video stream for later decoding. In one example implementation, the Real-Time Transport Protocol (RTP) is used to transmit the encoded video over the network 104. In other implementations, a transport protocol other than RTP, such as the HyperText Transfer Protocol (HTTP) video streaming protocol, may be used.

[0032] For example, when used in a videoconferencing system, sending station 102 and / or receiving station 106 may include the functionality to encode and decode video streams as described below. For example, receiving station 106 may be a videoconference participant that receives, decodes, and displays an encoded video bitstream from a videoconference server (e.g., sending station 102), and also encodes and sends its own video bitstream to the videoconference server for decoding and display by other participants.

[0033] Figure 2 is a block diagram illustrating an example of a computing device 200 that may implement a transmitting station or a receiving station. For example, computing device 200 may implement one or both of transmitting station 102 and receiving station 106 of Figure 1. Computing device 200 may be in the form of a computing system that includes multiple computing devices, or in the form of a single computing device, such as, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.

[0034] CPU 202 in computing device 200 may be a conventional central processing unit. Alternatively, CPU 202 may be any other type of device or devices now existing or later developed that are capable of manipulating or processing information. While the disclosed embodiments may be implemented with one processor (e.g., CPU 202) as shown, advantages of speed and efficiency may be achieved by using multiple processors.

[0035] In an embodiment, the memory 204 in the computing device 200 may be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as the memory 204. The memory 204 may include code and data 206 accessed by the CPU 202 using a bus 212. The memory 204 may further include an operating system 208 and application programs 210, which may include at least one program that enables the CPU 202 to perform the methods described herein. For example, the application programs 210 may include applications 1 through N, and further include a video coding application that implements the techniques described herein, e.g., techniques for performing inter-prediction of a current block with filtering. The computing device 200 may also include secondary storage 214, which may be, for example, a memory card used in a mobile computing device. Because a video communication session may contain a large amount of information, all or a portion of it may be stored in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0036] Computing device 200 may also include one or more output devices, such as a display 218. Display 218, in one example, may be a touch-sensitive display that combines a display with touch-sensing elements operable to sense touch input. Display 218 may be connected to CPU 202 via bus 212. In addition to, or instead of, display 218, other output devices may be provided that enable a user to program or otherwise use computing device 200. If the output device is or includes a display, the display may be implemented in various ways, to include a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0037] Computing device 200 may also include or communicate with image sensing device 220, such as a camera or any other existing or later developed image sensing device 220 that can sense images, such as an image of a user operating computing device 200. Image sensing device 220 may be positioned to point toward a user operating computing device 200. In one example, the position and optical axis of image sensing device 220 may be configured to be directly adjacent to display 218 such that its field of view includes the area from which display 218 is viewable.

[0038] Computing device 200 may also include or communicate with sound sensing device 222, such as a microphone or any other existing or later developed sound sensing device that can sense sounds near computing device 200. Sound sensing device 222 may be positioned to face toward a user operating computing device 200 and may be configured to receive sounds, such as speech or other utterances, made by the user while the user is operating computing device 200.

[0039] While FIG. 2 depicts the CPU 202 and memory 204 of computing device 200 as integrated into a single unit, other configurations are possible. The operations of CPU 202 may be distributed across multiple machines (each machine may have one or more processors) that may be coupled directly or via a local area network or other network. Memory 204 may be distributed across multiple machines, such as network-based memory or memory within multiple machines that perform the operations of computing device 200. While depicted here as a single bus, bus 212 of computing device 200 may be comprised of multiple buses. Furthermore, secondary storage 214 may be directly connected to other components of computing device 200 or may be accessible over a network and may include an integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, computing device 200 may be implemented in a wide variety of configurations.

[0040] FIG. 3 illustrates an example of a video stream 300 that may be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At a next level, the video sequence 302 includes several adjacent frames 304. While three adjacent frames 304 are shown, the video sequence 302 may include any number of adjacent frames 304. The adjacent frames 304 may be further subdivided into individual frames, such as frame 306. At a next level, the frame 306 may be divided into a series of planes or segments 308. A segment 308 may be a subset of a frame, for example, to enable parallel processing. A segment 308 may be a subset of a frame that allows for separation of the video data into individual colors. For example, a frame 306 of color video data may include a luminance plane and two chrominance planes. The segments 308 may be sampled at different resolutions.

[0041] Regardless of whether a frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which may contain data corresponding to, for example, 16x16 pixels within the frame 306. The blocks 310 may also be arranged to contain data from one or more segments 308 of pixel data. The blocks 310 may also be any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise specified, the terms block and macroblock are used interchangeably herein.

[0042] 4 is a block diagram of an encoder 400 according to an embodiment of the present disclosure. The encoder 400 may be implemented in the sending station 102, such as by providing a computer software program stored in a memory, such as memory 204, as described above. The computer software program may include machine instructions that, when executed by a processor, such as CPU 202, cause the sending station 102 to encode video data in the manner described in FIG. 4. The encoder 400 may also be implemented as dedicated hardware included, for example, in the sending station 102. In one particularly preferred embodiment, the encoder 400 is a hardware encoder.

[0043] The encoder 400 has the following stages to perform various functions in a forward path (indicated by solid connecting lines): an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy coding stage 408, to use the video stream 300 as input to generate an encoded or compressed bitstream 420. The encoder 400 may also include a reconstruction path (indicated by dotted connecting lines) to reconstruct frames for encoding future blocks. In FIG. 4, the encoder 400 has the following stages to perform various functions in the reconstruction path: an inverse quantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filtering stage 416. Other structural variations of the encoder 400 may be used to encode the video stream 300.

[0044] When the video stream 300 is submitted for encoding, each neighboring frame 304, such as frame 306, may be processed block-by-block. In the intra / inter prediction stage 402, each block may be coded using intra-frame prediction (also called intra-prediction) or inter-frame prediction (also called inter-prediction). In either case, a predictive block may be formed. In the case of intra-prediction, the predictive block may be formed from previously coded and reconstructed samples of the current frame. In the case of inter-prediction, the predictive block may be formed from samples in one or more previously constructed reference frames. Implementations for forming the predictive block are described below with reference to Figures 6, 7, and 8, e.g., using a parameterized motion model identified for coding the current block of the video frame.

[0045] Next, with continued reference to FIG. 4 , the prediction block may be subtracted from the current block in the intra / inter prediction stage 402 to generate a residual block (also referred to as a residual). The transform stage 404 converts the residual into transform coefficients, e.g., in the frequency domain, using a block-based transform. The quantization stage 406 converts the transform coefficients into discrete quantum values, called quantized transform coefficients, using a quantization value or quantization level. For example, the transform coefficients may be divided by the quantization value and truncated. The quantized transform coefficients are entropy coded by the entropy coding stage 408. The entropy coded coefficients, along with other information used to decode the block (which may include, for example, the type of prediction used, the transform type, motion vectors, and the quantization value), are output to a compressed bitstream 420. The compressed bitstream 420 may be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 may also be referred to as a coded video stream or coded video bitstream, and these terms are used interchangeably herein.

[0046] The reconstruction path (indicated by the dotted connecting line) in Figure 4 may be used to ensure that the encoder 400 and decoder 500 (described below) use the same reference frame to decode the compressed bitstream 420. The reconstruction path performs functions similar to those performed during the decoding process (described below), including inverse quantizing the quantized transform coefficients in an inverse quantization stage 410 and inverse transforming the inverse quantized transform coefficients in an inverse transform stage 412 to generate a derived residual block (also referred to as a derived residual). In the reconstruction stage 414, a prediction block predicted in the intra / inter prediction stage 402 may be added to the derived residual to create a reconstructed block. A loop filtering stage 416 may be applied to the reconstructed block to reduce distortions such as blocking artifacts.

[0047] Other variations of the encoder 400 may be used to encode the compressed bitstream 420. For example, a non-transform-based encoder may directly quantize the residual signal for a particular block or frame without using the transform stage 404. In other implementations, the encoder may have the quantization stage 406 and the inverse quantization stage 410 combined into a common stage.

[0048] Figure 5 is a block diagram of a decoder 500 according to an embodiment of the present disclosure. The decoder 500 may be implemented in the receiving station 106, for example, by providing a computer software program stored in memory 204. The computer software program may include machine instructions that, when executed by a processor such as CPU 202, cause the receiving station 106 to decode video data in the manner described in Figure 5. The decoder 500 may also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.

[0049] The decoder 500, similar to the reconstruction path of the encoder 400 described above, in one example includes the following stages that perform various functions to generate an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, an inverse quantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a post-filtering stage 514. Other structural variations of the decoder 500 may be used to decode the compressed bitstream 420.

[0050] When the compressed bitstream 420 is presented for decoding, data elements in the compressed bitstream 420 may be decoded by an entropy decoding stage 502 to generate a set of quantized transform coefficients. An inverse quantization stage 504 inverse quantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantization value), and an inverse transform stage 506 inverse transforms the inverse quantized transform coefficients to generate derived residuals that may be identical to those created by the inverse transform stage 412 of the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 may use an intra / inter prediction stage 508 to create predictive blocks identical to those created by the encoder 400, e.g., the intra / inter prediction stage 402. In a reconstruction stage 510, a predictive block may be added to the derived residuals to create a reconstructed block. A loop filtering stage 512 may be applied to the reconstructed block to reduce blocking artifacts.

[0051] Other filtering may be applied to the reconstructed blocks. In this example, a post-filtering stage 514 is applied to the reconstructed blocks to reduce blocking artifacts or perform other post-processing on the frame, and the result is output as the output video stream 516. The output video stream 516 may also be referred to as a decoded video stream, and these terms are used interchangeably herein. Other variations of the decoder 500 may be used to decode the compressed bitstream 420. For example, the decoder 500 can generate the output video stream 516 without the post-filtering stage 514.

[0052] FIG. 6 is a flowchart diagram of a technique 600 for decoding a current block using inter prediction with filtering. The technique 600 performs inter prediction with filtering. The technique 600 may be implemented in a decoder such as the decoder 500 of FIG. 5 or in the reconstruction path of FIG. 4. The technique 600 may be implemented as a software program that can be executed by a computing device such as the transmitting station 102 or the receiving station 106 of FIG. 1, for example. The software program may be stored in a memory such as the memory 204 or the secondary storage 214 and may include machine-readable instructions (e.g., executable instructions) that can be executed by a processor such as the CPU 202 to cause the computing device to perform the technique 600. In at least some implementations, the technique 600 may be performed in whole or in part by the intra / inter prediction stage 508 of the decoder 500 of FIG. 5.

[0053] Technique 600 may be implemented using dedicated hardware or firmware. Some computing devices may have multiple memories, multiple processors, or both. The steps or operations of technique 600 may be distributed using different processors, memories, or both. Use of the terms "processor" or "memory" in the singular encompasses computing devices having one processor or one memory, as well as devices having multiple processors or multiple memories that may be used to perform some or all of the recited steps.

[0054] At 602, an intermediate predicted block is identified for the current block. The current block may be a luminance block (e.g., a Y block) or a chrominance block (e.g., a Cb block, a Cr block, a U block, or a V block). The intermediate predicted block may also be referred to as a reference block. In one example, a motion vector and a reference frame may be identified for the current block. In one example, the reference block and motion vector may be identified using data obtained from a compressed bitstream, such as compressed bitstream 420 of FIG. 5. The motion vector and reference may be identified as described with respect to FIG. 5.

[0055] In one example, data obtained from a compressed bitstream may indicate that a motion vector and / or reference of another block, which may be a temporal or spatial neighboring block of the current block, should be used for the current block. In such a situation, the current block may be considered merged with the neighboring block. An intermediate prediction block is a block in a reference frame specified by a motion vector. As is well known, the intermediate prediction block (i.e., the reference block) may be at an integer pixel position or a sub-pixel position. If the intermediate prediction block is at a sub-pixel position, interpolation filtering may be performed to obtain values ​​at the sub-pixels, as is well known.

[0056] At 604, filter coefficients are obtained for the filter. The filter coefficients are obtained using first reconstructed pixels and second reconstructed pixels. The first reconstructed pixels are located around the current block, and the second reconstructed pixels are located around the intermediate predicted block. The set of reconstructed pixels may be referred to as a template. Thus, the first reconstructed pixels may also be referred to as a current block template, and the second reconstructed pixels may also be referred to as a reference block template. Regardless of whether the intermediate predicted block is located at an integer position or a sub-pixel position, the second reconstructed pixels (i.e., the reference block template) are located at an integer position.

[0057] 7 shows an example of a reconstructed pixel template 700. The example 700 shows pixels filled with different patterns. The example 700 is used to represent a current block template and also to represent a reference block template.

[0058] When used to describe a current block template, block 708 indicates the current block (i.e., the block being decoded). Although block 708 is shown as being 8x4 in size, this disclosure is not so limited. Block 708 may be any other size. Pixels filled with pattern 702 are pixels of the current block of the current frame. Pixels filled with pattern 706 (or a subset thereof, as described further herein) indicate reconstructed pixels of the current frame. Pixels filled with pattern 704 are unavailable pixels and may include (i.e., be set to) padding values. Depending on the neighborhood used in the filter, one or more pixels used by the filter may be unavailable (e.g., because they are outside the frame boundary or outside the largest coding unit containing block 708). Thus, padding values ​​may be used (e.g., assumed) for such pixels.

[0059] When used to describe a reference block template, block 708 indicates a reference block within a reference frame. Again, block 708 is shown as being 8x4 in size, but the disclosure is not so limited. Block 708 may be of a size corresponding to the size of the current block. Thus, pixels filled with pattern 702 are pixels of the reference block of the reference frame. Pixels filled with pattern 706 (or a subset thereof, as described further herein) indicate reconstructed pixels of the reference frame. Pixels filled with pattern 704 are unavailable pixels and may contain padding values.

[0060] The template may include a top region 710 that may include rows 1 through N (where N>1) of pixels. The template may include a top right region 712 that includes rows 1 through N. The template may include a left region 714 that includes columns 1 through M (where M>1) of pixels. The template may include a bottom left region 716 that includes columns 1 through M (where M>1) of pixels.

[0061] In one example, N=M. In one example, if the current block is a luma block, the template may be four samples wide. If the current block is a chroma block, the template (i.e., the chroma template) may be based on the chroma color format. For example, for 4:4:4 content, the chroma template may also be four samples wide, and for 4:2:0 or 4:2:2 color format, the chroma template may be two samples wide. In one example, if the top right region 712 is available, only the top right 4x4 luma block is included in the template. Similarly, if the bottom left region 716 is available, only the bottom right 4x4 luma block is included in the template. The chroma template may be adjusted accordingly based on the chroma color format. In another example, the top template may always be one sample wide for both luma and chroma, while the left template may be four samples wide for luma.

[0062] Referring again to FIG. 6, in one example, the filter coefficients include at least two coefficients. In one example, the filter coefficients include two or more coefficients for at least one of the color components (i.e., at least one of the luma or chroma components). In one example, the number of filter coefficients (i.e., the base) may be decoded from the compressed bitstream. For example, an indicator of the number of filter coefficients may be decoded from the compressed bitstream. For example, in response to the indicator of the number of filter coefficients being a first value (e.g., 0), technique 600 may not be performed on the current block. That is, when the indicator of the number of coefficients is the first value, no filtering is performed on the predictive block. When the indicator of the number of filter coefficients is a second value (e.g., 1), two filter coefficients are derived, and when the indicator of the number of filter coefficients is a third value (e.g., 2), more than two filter coefficients are derived.

[0063] When the number of filter coefficients is a second value (e.g., when the indicator for the number of filter coefficients is 2), the two filter coefficients may be obtained using a technique known as local illumination compensation (LIC). LIC is described in U.S. Patent Publication No. 2021 / 0352309, which is incorporated herein by reference. Briefly, LIC is an inter-prediction technique that models local illumination variations between a current block and its predicted block as a function of illumination between the current block template and the reference block template. The parameters of the function may be represented by a scale α and an offset β, thereby forming a linear equation α×p[x]+β for compensating for illumination changes, where p[x] is the reference sample pointed to by the motion vector (MV) at position x in the reference frame. Because α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for them. That is, the encoder does not need to encode the values ​​of the α and β parameters, and the decoder does not need to decode them.

[0064] In one example, the filter may be a convolution filter. The filter coefficients may be obtained by minimizing an error metric between a first reconstructed pixel and a second reconstructed pixel. The error metric may be a mean square error (MSE) between pixel values ​​of the respective reconstructed pixels. The error may be a sum of absolute differences (SAD) error between pixel values ​​of the reconstructed pixels. Any other suitable error metric may be used.

[0065] In one example, the number of coefficients obtained depends on which pixels in the neighborhood of the intermediate predicted pixel to which the filter is applied are used for filtering. The pixels in the neighborhood of the intermediate predicted pixel used for filtering are referred to herein as at least a subset of the pixels in the neighborhood. FIG. 8 shows an example neighborhood 800 of an intermediate pixel 802 of an intermediate predicted block. The example 800 shows a 3×3 neighborhood. However, the neighborhood may be larger, smaller, rectangular, or some other shape (e.g., diamond-shaped). The example 800 shows that pixels 804-810 (i.e., pixels north, east, south, and west of the intermediate pixel 802, respectively) are used for filtering. Thus, the filter coefficients include at least five coefficients, i.e., one coefficient used for each of the pixels 802-810.

[0066] Therefore, the filter is a 5-tap filter, and the predicted pixel corresponding to the intermediate pixel 802 can be obtained using equation (1), where c i (i=0,...,4) are the filter coefficients, and pred is the filter pixel of the final predicted block. It is shown that equation (1) further includes a constant term (i.e., c), which may also be derived and used in some implementations.

number

[0067] In one example, one or more, but not all, filter coefficients may be further refined after being derived. Thus, the obtained filter coefficients may be considered to be predicted filter coefficients. Differences between the predicted filter coefficients and the actual values ​​of the filter coefficients (i.e., coefficient refinement values) may be signaled in the compressed bitstream. Thus, obtaining filter coefficients for a filter may include obtaining predicted filter coefficients for filter coefficients among the filter coefficients, decoding the coefficient refinement value from the compressed bitstream, and adjusting the predicted filter coefficients using the coefficient refinement value to obtain the filter coefficients.

[0068] In one example, the coefficient refinement value corresponds to (i.e., is used for) the intermediate predicted pixel itself. That is, for example, the coefficient refinement value may be used to refine a filter coefficient obtained for the intermediate pixel 802 in FIG. 8 . Thus, the coefficient refinement value is used for the intermediate predicted pixel to which the filter is applied. In one example, the coefficient refinement value may be used to refine a coefficient corresponding to a nonlinear term of the filter. For example, the filter may include a filter coefficient corresponding to the intermediate predicted pixel, one nonlinear term, and a constant value. The nonlinear term (i.e., the nonlinear component) may be a square term of the intermediate predicted pixel. Thus, the filter may be a filter coefficient corresponding to a×p[x] 2 +b×p[x]+c, where a and b are filter coefficients, c is a constant component, and p[x] is the value of the intermediate predicted pixel at position x.

[0069] In one example, as described with respect to FIG. 8, the filter coefficients may be applied to at least a subset of pixels within a 3×3 neighborhood of the intermediate predicted pixel to obtain predicted pixels of the final predicted block. The 3×3 neighborhood may be used regardless of whether the current block is a luma block or a chroma block. The subset of pixels may form a shape (e.g., may be any shape). In one example, the subset of pixels within the 3×3 neighborhood may be pixels forming a cross shape as shown in FIG. 8. That is, the subset of pixels may be pixels 802-810. That is, at least the subset of pixels within the 3×3 neighborhood of the intermediate predicted pixel may be or include the intermediate predicted pixel, the pixel above the intermediate predicted pixel, the pixel to the right of the intermediate predicted pixel, the pixel below the intermediate predicted pixel, and the pixel to the left of the intermediate predicted pixel.

[0070] In one example, the filter may use at least a subset of pixels in a 3x3 neighborhood and may further include a constant term (also referred to as a DC value). In one example, one shape (i.e., a first filter shape) may be used for at least a subset of pixels in the neighborhood for a current block that is a luma block, and then a second filter shape may be used for the current block that is a chroma block. Thus, the first filter shape may be used when the current block is a luma block, and a second filter shape, different from the first filter shape, may be used when the current block is a chroma block.

[0071] At 606, a filter is applied to the intermediate predicted block to obtain a final predicted block. At 608, the current block is reconstructed using the final predicted block. In one example, a residual block may be decoded from the compressed bitstream and added to the final predicted block to obtain the current block.

[0072] In one example, cross-component filtering may be applied. That is, a prediction obtained for a luma block may be used to obtain a prediction for a chroma block. In other words, if the current block is a luma block, a chroma prediction block for a chroma block corresponding to the current block is derived from the final prediction block. Thus, if the current block is a luma block, technique 600 may further include obtaining a chroma prediction block from the final prediction block. In one example, a 1×1 chroma filter and DC values ​​may be used in addition to a 3×3 luma filter. Alternatively, a 3×3 chroma filter and DC values ​​may be used in addition to a 3×3 luma filter.

[0073] The cross-component filtering may be similar to the cross-component filtering described in U.S. Patent Publication No. 2022 / 0272351, which is incorporated herein by reference. In summary, in cross-component filtering, chroma samples are predicted based on reconstructed luma samples of the same coding unit (sometimes called a maximum coring unit, macroblock, or other such nomenclature) of the current block by using a linear model according to the following equation (2):

number

[0074] In equation (2), pred c (i,j) represents the chroma sample prediction, and rec ’ l (i,j) represents the downsampled reconstructed luma prediction of the current luma block. Downsampling is performed when the chroma samples and luma samples do not have the same resolution. For example, downsampling may be performed when a 4:2:2 or 4:2:0 format is used. Downsampling aligns the resolution of the luma block and the chroma block.

[0075] The cross-component parameters (α and β) may be derived using up to four neighboring chroma samples and their corresponding downsampled luma samples. Figure 9 shows an example 900 of the locations of the left and top samples and samples of the current block involved in the cross-component filtering mode. The division operation to calculate the parameter α may be implemented using a lookup table. For luma block 902, the locations of the left and top samples are shown as filled circles, such as filled circle 904. For chroma block 906, the locations of the left and top samples are shown as filled circles, such as filled circle 908.

[0076] In another example, a 7-tap convolution filter may be used to obtain a chroma prediction block from a luma prediction block. The convolution filter may include a 5-tap plus a sign-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of a center (C) luma sample co-located with the chroma sample to be predicted and its above / north (N), below / south (S), left / west (W), and right / east (E) neighbors, as described above with respect to Equation (1). Thus, a prediction pixel may be obtained using the following Equation (3):

number

[0077] In equation (3), P is a nonlinear term. In one example, the nonlinear term P is expressed as a power of two of the central luma sample C (i.e., the intermediate pixel 802), and the content sample value range P=(C 2 +midVal)≫bitDepth. For example, assuming 10-bit content, P can be scaled as P=(C 2 +512)≫10. Other nonlinear terms are possible. The bias term B, if used, may represent a scalar offset between the input and output. The coefficient c i can be obtained in a similar manner as described above with respect to equation (1). For example, the coefficient ci can be obtained by minimizing the MSE between the predicted chroma samples in the reference domain and the reconstructed chroma samples. In equation (3), C, N, S, E, and W correspond to the values ​​of the luma prediction value as shown in FIG.

[0078] 6, technique 600 may be determined to perform in response to decoding one or more syntax elements from a compressed bitstream that indicate that technique 600 should be performed. Thus, in one example, technique 600 may include decoding inter-prediction in a filtering mode (i.e., a mode that instructs a decoder to apply filtering to (inter) prediction blocks obtained using inter-prediction). Inter-prediction with filtering mode may be decoded from a compressed bitstream.

[0079] In another example, technique 600 is performed on a current block when the block is merged with a block that uses inter prediction with filtering. Thus, in one example, inter prediction with filtering may be performed on the current block in response to determining that a block other than the current block has been reconstructed using inter prediction with filtering mode. In other words, a filter is applied to an intermediate predicted block in response to determining that a block other than the current block has been reconstructed using an inter prediction mode that indicates that filtering is applied. More generally, technique 600 is performed on a current block in response to determining that one or more of the spatial and / or temporal neighbors of the current block have been predicted using inter prediction with filtering mode.

[0080] In one example, an indicator may be signaled (e.g., encoded) in the compressed bitstream indicating that inter prediction with filtering is allowed at the block level. If the indicator indicates that inter prediction with filtering is not allowed at the block level, technique 600 is performed for the current block. The indicator may be signaled for a group of blocks. That is, the indicator may be signaled within a header corresponding to the group of blocks. The group of blocks may be a group of frames, a frame, a segment of blocks, a tile of blocks, or a superblock. More generally, the group of blocks may be any structure used to packetize data and that provides identification information for the data contained therein. In one example, the indicator may be signaled at the sequence level within a sequence parameter set (SPS).

[0081] 10 is a flowchart diagram of a technique 1000 used to encode a current block. The technique 1000 may be implemented in an encoder such as the encoder 400 of FIG. 4. The technique 1000 may be used to obtain (e.g., find, identify, etc.) a motion vector for weighted inter prediction (i.e., inter prediction with filtering) as described above. The technique 1000 may refine the motion vector obtained for the current block.

[0082] Technique 1000 may be implemented as a software program that may be executed by a computing device such as, for example, transmitting station 102. The software program may be stored in a memory such as memory 204 or secondary storage 214 and may include machine-readable instructions (e.g., executable instructions) that may be executed by a processor such as CPU 202 to cause the computing device to perform technique 1000. In at least some implementations, technique 1000 may be performed in whole or in part by intra / inter prediction stage 402 of encoder 400 of FIG.

[0083] Technique 1000 may be implemented using dedicated hardware or firmware. Some computing devices may have multiple memories, multiple processors, or both. The steps or operations of technique 1000 may be distributed using different processors, memories, or both. The use of the terms "processor" or "memory" in the singular encompasses computing devices having one processor or one memory, as well as devices having multiple processors or multiple memories that may be used to perform some or all of the recited steps.

[0084] At 1002, an intermediate motion vector may be obtained for the current block. The intermediate motion vector may be a motion vector obtained using any technique for identifying a motion vector for the current block, such as those described with respect to the intra / inter prediction stage 402 of FIG. 4. For example, as is well known, a motion vector may be identified by performing a motion-compensated search within a search area of ​​one or more reference frames to identify the closest matching reference block in one of the reference frames. However, other methods of identifying the intermediate motion vector are also possible.

[0085] At 1004, filter coefficients are obtained by minimizing an error metric between a prediction block (i.e., a reference block) corresponding to (i.e., referenced or pointed to by) the intermediate motion vector and the current block (i.e., a source block). The error metric may be a sum of squares error (SSE). Four independent filter coefficients may be obtained. Using example 800, a first coefficient (denoted as a) may be derived for the center pixel (i.e., middle pixel 802), a second coefficient (denoted as b) may be derived for the upper pixel (i.e., pixel 804), the complement of the second coefficient (i.e., −b) may be used for the lower pixel (i.e., pixel 808), a third coefficient (denoted as c) may be derived for the left pixel (i.e., pixel 810), and the complement of the fourth coefficient (i.e., −c) may be used for the right pixel (i.e., pixel 806). The fourth coefficient is simply a DC constant value (denoted as d).

[0086] At 1006, a motion vector is obtained for the current block by refining the intermediate motion vector using the filter coefficient. That is, the motion vector refinement for the current MV is derived based on the filter coefficient. In one example, the motion vector refinement can be obtained using the following equation (4):

number

[0087] Technique 1000 may further include encoding a motion vector in a compressed bitstream, such as compressed bitstream 420 of FIG. 4. Any technique for encoding a motion vector may be used. In one example, a prediction of the motion vector is obtained. In such a case, encoding the motion vector in the compressed bitstream includes encoding a difference between the motion vector and the prediction of the motion vector in the compressed bitstream. Illustratively, the intermediate motion vector may be (MV x ,MV y ), and the motion vector refinement can be (dMV x ,dMV y), so the motion vectors coded into the compressed bitstream are (MV x +dMV x ,MV y +dMV y )

[0088] For ease of explanation, techniques 600 and 1000 are each shown and described as a series of steps or acts. However, steps or acts according to the present disclosure may be performed in various orders and / or simultaneously. In addition, other steps or acts not shown and described herein may be used. Furthermore, not all illustrated steps or acts may be required to implement a methodology in accordance with the disclosed subject matter.

[0089] The encoding and decoding aspects described above illustrate some examples of encoding and decoding techniques, although it should be understood that encoding and decoding, as the terms are used in the claims, may mean compression, decompression, conversion, or other data processing or modification.

[0090] The word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "exemplary" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clearly dictated otherwise by context, a statement such as "X includes A or B" is intended to refer to its natural inclusive permutations. That is, if X includes A, if X includes B, or if X includes both A and B, then "X includes A or B" is satisfied in each of the foregoing cases. Additionally, the articles "a" and "an," as used in this application and the appended claims, should generally be construed to mean "one or more" unless otherwise specified or unless the context clearly indicates that the singular form is intended. Furthermore, use of the term "embodiment" or "one embodiment" throughout this disclosure is not intended to refer to the same embodiment or implementation unless described as such.

[0091] Implementations of the transmitting station 102 and / or receiving station 106 (and the algorithms, methods, instructions, etc. stored therein and / or executed by, including those executed by the encoder 400 and decoder 500) may be realized in hardware, software, or any combination thereof. Hardware may include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the foregoing hardware, alone or in combination. The terms "signal" and "data" are used interchangeably. Furthermore, portions of the transmitting station 102 and receiving station 106 need not necessarily be implemented in the same way.

[0092] Furthermore, in one aspect, for example, the transmitting station 102 or the receiving station 106 may be implemented using a general-purpose computer or processor with a computer program that, when executed, performs any of the respective methods, algorithms, and / or instructions described herein. Additionally or alternatively, a special-purpose computer / processor may be utilized that may include other hardware for performing any of the methods, algorithms, or instructions described herein, for example.

[0093] The sending station 102 and the receiving station 106 can be implemented on computers, for example, in a videoconferencing system. Alternatively, the sending station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communication device. In this case, the sending station 102 can encode content into an encoded video signal using the encoder 400 and transmit the encoded video signal to the communication device. The communication device can then decode the encoded video signal using the decoder 500. Alternatively, the communication device can decode content stored locally on the communication device, e.g., content not transmitted by the sending station 102. Other suitable sending and receiving implementations are also possible. For example, the receiving station 106 can be a generally stationary personal computer rather than a portable communication device, and / or the device including the encoder 400 can include the decoder 500.

[0094] Furthermore, all or part of the embodiments of the present disclosure may take the form of a computer program product, for example, accessible from a computer-usable or computer-readable medium. The computer-usable or computer-readable medium may be, for example, any device that can tangibly store, store, communicate, or transfer a program for use by or in connection with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media may also be used.

[0095] The above-described embodiments, implementations, and aspects have been described to facilitate easy understanding of the present disclosure, and are not intended to limit the present disclosure. Rather, the present disclosure is intended to cover various modifications and equivalent arrangements that fall within the scope of the appended claims, the scope of which should be accorded the broadest interpretation permitted under law so as to encompass all such modifications and equivalent arrangements.

Claims

1. 1. A method for decoding a current block using inter prediction with filtering, the method comprising: identifying an intermediate prediction block for the current block using the motion vector and a reference frame; obtaining filter coefficients for a filter, the filter coefficients being obtained using the first reconstructed pixel and the second reconstructed pixel; the first reconstructed pixels are in the periphery of the current block; The second reconstructed pixels are in the periphery of the intermediate prediction block, and the method further comprises: applying the filter to the intermediate predicted block to obtain a final predicted block; and reconstructing the current block using the final predicted block.

2. The method of claim 1 , wherein the filter comprises more than two coefficients.

3. The method of claim 1 , comprising decoding an inter-prediction mode that indicates applying the filter.

4. 4. The method of claim 1, further comprising applying the filter to the intermediate predicted block in response to determining that a block other than the current block has been reconstructed using an inter prediction mode that indicates applying filtering.

5. A method according to any one of claims 1 to 3, comprising decoding, from a compressed bitstream, the radix of the filter coefficients to be obtained for the filter.

6. The method of claim 5 , wherein the radix of the filter coefficients is greater than two.

7. Obtaining the filter coefficients for the filter includes: obtaining a prediction filter coefficient for one of the filter coefficients; decoding coefficient refinement values ​​from the compressed bitstream; and adjusting the prediction filter coefficients using the coefficient refinement value to obtain the one filter coefficient.

8. The method of claim 7 , wherein the coefficient refinement values ​​are used for intermediate predicted pixels to which the filter is applied.

9. The method of claim 7 , wherein the coefficient refinement values ​​are used to refine coefficients corresponding to non-linear terms of the filter.

10. The method of any one of claims 1 to 3, wherein the filter coefficients are obtained by minimizing an error metric between the first reconstructed pixel and the second reconstructed pixel.

11. The method of claim 10 , wherein the error metric is the sum of squares error.

12. The method of any one of claims 1 to 3, wherein the filter coefficients are applied to at least a subset of pixels within a 3x3 neighborhood of an intermediate predicted pixel to obtain predicted pixels of the final predicted block.

13. 13. The method of claim 12, wherein the at least the subset of pixels within a 3x3 neighborhood of the intermediate predicted pixel includes the intermediate predicted pixel, a pixel above the intermediate predicted pixel, a pixel to the right of the intermediate predicted pixel, a pixel below the intermediate predicted pixel, and a pixel to the left of the intermediate predicted pixel.

14. The method of any one of claims 1 to 3, wherein the filter further comprises a constant component.

15. The method according to any one of claims 1 to 3, wherein the filter comprises at least one non-linear component.

16. the current block is a luminance block, The method according to any one of claims 1 to 3, wherein a chroma prediction block for a chroma block corresponding to the current block is derived from the final prediction block.

17. 4. The method of claim 1, wherein a first filter shape is used if the current block is a luma block, and a second filter shape different from the first filter shape is used if the current block is a chroma block.

18. A method used to encode a current block, said method comprising: obtaining an intermediate motion vector for the current block; obtaining filter coefficients by minimizing an error metric between a prediction block corresponding to the intermediate motion vector and the current block; and obtaining a motion vector for the current block by refining the intermediate motion vector using the filter coefficients.

19. the filter coefficients include a first coefficient (a), a second coefficient (b), and a third coefficient (c); Refining the intermediate motion vector using the filter coefficients A first adjustment of a first component of the intermediate motion vector is made by 2 ＊ b / a and A second adjustment of the second component of the intermediate motion vector is 2 ＊ and obtaining the signal as c / a.

20. 20. The method of claim 19, comprising encoding at least one of the first adjustment or the second adjustment into a compressed bitstream.

21. A device comprising a processor configured to perform the method of any one of claims 1 to 20.

22. A device, the device comprising: Memory and A device comprising: a processor; and said memory storing instructions operable to cause said processor to perform the method of any one of claims 1 to 20.

23. A non-transitory computer-readable storage medium comprising executable instructions that facilitate performance of operations that, when executed by a processor, are operable to cause the processor to perform the method of any one of claims 1 to 20.

Images