Spatially adaptive quantization-aware deblocking filter

Abstract

Spatially adaptive quantization-aware deblocking filters are disclosed. The spatially adaptive quantization-aware deblocking filters are used to encode or decode video or image frames. The deblocking filters receive a reconstructed frame generated based on dequantized and inverse transformed coefficients of the video or image frames. The reconstructed frame is filtered according to adaptive quantization field data of the video or image frames. The adaptive quantization field data represents weights applied to quantization values used at different regions of the video or image frames. The number of blocking artifacts remaining in the generated filtered frame is determined. The adaptive quantization field data is then adjusted based on the number of blocking artifacts. The filtered frame is then filtered according to the adjusted adaptive quantization field data. The generated re-filtered frame is then output to an output source, such as for transmission, display, storage, or further processing.

Description

[0001] Divisional Statement

[0002] This application is a divisional application of the application with the application number 201980007525.9 and the filing date of 09 / 04 / 2019, and the title of “A method and apparatus for generating a filtered reconstructed frame”. TECHNICAL FIELD

[0003] The present application relates to spatially adaptive quantization-aware deblocking filter. BACKGROUND

[0004] Digital video streams can represent video using a series of frames or still images. Digital video can be used for a variety of applications, including for example, video conferencing, high definition video entertainment, video advertising, or sharing of user-generated videos. Digital video streams can contain a large amount of data and consume a large amount of computational or communication resources of a computing device used to process, transmit, or store the video data. Various methods have been proposed, including encoding or decoding techniques, to reduce the amount of data in a video stream.

[0005] Methods for reducing the amount of data in a video stream can also be used to reduce the amount of data in an image. Image content represents a large amount of online content. Web pages can include multiple images, and a large portion of the time and resources spent rendering a web page is dedicated to rendering those images for display. The amount of time and resources required to receive and render an image for display depends in part on the way the image is compressed. As such, using encoding and decoding techniques to reduce the overall data size of an image can result in faster rendering of the image and, therefore, the web page that includes the image. SUMMARY

[0006] Disclosed herein, among other things, are systems and techniques for video or image coding using a spatially adaptive quantization-aware deblocking filter.

[0007] A method according to an embodiment of the disclosure includes applying first filter parameters to a reconstructed frame to produce a first filtered frame, wherein the first filter parameters are defined based on adaptive quantization field data associated with the reconstructed frame, adjusting the first filter parameters to produce second filter parameters, wherein the adjusting includes increasing or decreasing at least one value of the adaptive quantization field data, applying the second filter parameters to the first filtered frame to produce a second filtered frame, and outputting or storing the second filtered frame. For example, the second filtered frame can be output for display.

[0008] Another method according to embodiments of the disclosure includes decoding syntax data associated with an encoded frame from a bitstream. The syntax data includes quantized transform coefficients of a coding block of the encoded frame and adaptive quantization field data representing weights applied to quantization values used to encode the coding block. The method further includes dequantizing and inverse transforming the quantized transform coefficients of the coding block to produce a decoded block. The method also includes reconstructing the decoded block into a reconstructed frame. The method also includes applying a deblocking filter to the reconstructed frame according to the adaptive quantization field data to produce a first filtered frame. The method also includes determining a number of blocking artifacts within the first filtered frame. The method also includes adjusting at least some of the adaptive quantization field data based on the number of blocking artifacts to produce adjusted adaptive quantization field data. The method also includes applying the deblocking filter to the first filtered frame according to the adjusted adaptive quantization field data to produce a second filtered frame. The method also includes outputting the second filtered frame for display.

[0009] An apparatus according to embodiments of the disclosure includes a processor. For example, the processor can execute instructions stored in a non-transitory storage medium. The processor is configured to decode a coding block of an encoded frame to produce a decoded block, and reconstruct the decoded block into a reconstructed frame. The processor is also configured to filter the reconstructed frame according to adaptive quantization field data associated with the encoded frame to produce a first filtered frame, and subsequently adjust at least some of the adaptive quantization field data. For example, the adjustment can use a psychovisual model. The processor is configured to filter the first filtered frame according to the adaptive quantization field data after the adaptive quantization field data is adjusted to produce a second filtered frame, and output, store, or display the second filtered frame.

[0010] A method according to embodiments of the disclosure includes applying first filter parameters to a reconstructed frame to produce a first filtered frame, where the first filter parameters are defined based on adaptive quantization field data associated with the reconstructed frame, adjusting the first filter parameters to produce second filter parameters, where the adjusting includes increasing or decreasing at least one value of the adaptive quantization field data, applying the second filter parameters to the first filtered frame to produce a second filtered frame, and outputting or storing the second filtered frame. For example, the second filtered frame can be output for display.

[0011] These and other aspects of the disclosure are disclosed in the following detailed description of embodiments, appended claims, and figures. BRIEF DESCRIPTION OF DRAWINGS

[0012] Reference is made herein to the accompanying drawings, which are described below, wherein like numerals refer to like parts throughout the several views.

[0013] Figure 1 is a schematic diagram of an example of an encoding and decoding system.

[0014] Figure 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station of an encoding and decoding system.

[0015] Figure 3A is a diagram of an example of a video stream to be encoded and subsequently decoded.

[0016] Figure 3B is a diagram of an example of an image to be encoded and subsequently decoded.

[0017] Figure 4 is a block diagram of an example of an encoder.

[0018] Figure 5 is a block diagram of an example of a decoder.

[0019] Figure 6 is a block diagram of an example of a spatially adaptive quantization perceptual deblocking filter for encoding or decoding a video frame or image.

[0020] Figure 7 is a block diagram representing a portion of a video frame or image.

[0021] Figure 8 is a flowchart of an example of a technique for encoding or decoding a video frame or image frame using a spatially adaptive quantization perceptual deblocking filter to produce a filtered reconstructed frame.

[0022] Figure 9 is a flowchart of an example of a technique for iteratively filtering a video frame or image frame from adaptive quantization field data.

[0023] Figure 10 is a diagram of an example of a rendition of an original video frame or image using different filtering or no filter techniques. DETAILED DESCRIPTION

[0024] Lossy encoding involves reducing the amount of data in an image or video to be encoded, such as using quantization. In exchange for reducing the bit cost of the resulting encoded image or video, the image suffers some loss of quality. The degree of quality loss depends largely on the way the image data or video data is quantized during encoding. In particular, quantization of the image data or video data can result in discontinuities along block boundaries, such as block artifacts. Quantization error resulting from lossy encoding generally indicates the number of block artifacts resulting from the encoding. As such, the greater the quantization error, the more block artifacts, and thus the greater the loss of quality.

[0025] Block artifacts can be reduced by applying a filter, such as a deblocking filter, to the coefficients of a video block or image block. The deblocking filter can be applied to a reconstructed frame or a portion of a reconstructed frame at the end of a reconstruction stage in the encoding process or at the end of the decoding process. The deblocking filter removes block artifacts from a frame, reproducing the frame in its pre-encoding form. However, a typical deblocking filter does not have visibility into the amount of quantization used to encode a given frame or particular block therein. As such, and particularly in cases where different quantization levels are used for different blocks within a single frame, the deblocking filter can use suboptimal or inappropriate filter strength or filter radius size.

[0026] Embodiments of the present disclosure use a deblocking filter controlled based on adaptive quantization field data to address such issues. The adaptive quantization field data is used to control the strength and spatial size of the deblocking filter. As such, the deblocking filter uses a greater strength and / or spatial size for blocks that the adaptive quantization field data indicates were encoded with a greater quantization level, and a lesser strength and / or spatial size for blocks that the adaptive quantization field data indicates were encoded with a lesser quantization level. By controlling the strength and spatial size of the filter based on the adaptive quantization field data, fine details in the frame can be preserved. A psychovisual model is used in conjunction with the filtering to determine whether adjustments should be made to the adaptive quantization field data. For example, if the amount of block artifacts remaining in a region of the frame to be encoded or decoded is too great, the psychovisual model can indicate that the value of the adaptive quantization field data for that region should be increased.

[0027] More details of techniques for video or image coding using a spatially adaptive quantization perceptual deblocking filter are described herein first with reference to a system in which they can be implemented. Figure 1 is a schematic diagram of an example of an encoding and decoding system 100. The encoding and decoding system 100 includes a sending station 102, a receiving station 104, and a network 106.

[0028] The sending station 102 is a computing device that encodes and sends video or images. Alternatively, the sending station 102 can include two or more distributed computing devices for encoding and sending video or images. The receiving station 104 is a computing device that receives and decodes encoded video or encoded images. Alternatively, the receiving station 104 can include two or more distributed computing devices for receiving and decoding encoded video or encoded images. Examples of computing devices for implementing one or both of the sending station 102 or the receiving station 104 are described below with reference to Figure 2 Examples of computing devices for implementing one or both of the sending station 102 or the receiving station 104 are described below with reference to

[0029] The network 106 connects the sending station 102 and the receiving station 104 for encoding, sending, receiving, and decoding video or images. The network 106 can be, for example, the Internet. The network 106 can also be a local area network, a wide area network, a virtual private network, a cellular telephone network, or other means of transmitting video or images from the sending station 102 to the receiving station 104.

[0030] Embodiments of the encoding and decoding system 100 can differ from those shown and described with respect to Figure 1 In some embodiments, the encoding and decoding system 100 can omit the network 106. In some embodiments, a video or an image can be encoded and then stored for sending to the receiving station 104 or another device having storage at a later time. In some embodiments, the receiving station 104 can receive an encoded video or an encoded image (e.g., via the network 106, a computer bus, and / or some communication pathway) and store the encoded video or the encoded image for later decoding.

[0031] In some embodiments, the functions of the sending station 102 and the receiving station 104 can change based on the particular operation being performed. For example, during an operation to encode a video or an image, the sending station 102 can be a computing device for uploading a video or an image to be encoded to a server, and the receiving station 104 can be a server that receives the video or the image from the sending station 102 and encodes the video or the image for later use (e.g., for storing a bitstream, rendering a webpage, etc.).

[0032] In another example, during an operation to decode an encoded video or an encoded image, the sending station 102 can be a server that decodes an encoded video or an encoded image, and the receiving station 104 can be a computing device that receives the decoded video or the decoded image from the sending station 102 and outputs or renders the decoded video or the decoded image (e.g., as part of a webpage, etc. or to an output video stream).

[0033] In some embodiments, the encoding and decoding system 100 can omit the network 106. In some embodiments, a transfer protocol can be used to transfer a video or an image, or an encoded video or an encoded image, over the network 106. For example, the transfer protocol can be a real-time transfer protocol, a hypertext transfer protocol, or other image or video streaming protocol.

[0034] In some implementations, each of the sending station 102 and the receiving station 104 can include functionality for encoding and decoding video or images. For example, the encoding and decoding system 100 can be implemented using a video conferencing system. The receiving station 104 can be a computing device of a video conference participant. The receiving station 104 can receive an encoded video bitstream from a video conference server (e.g., the sending station 102) for decoding and viewing. The receiving station 104 can further encode another video bitstream and transmit it to the video conference server, such as for decoding and viewing by computing devices of other video conference participants.

[0035] Figure 2 The encoding and decoding system 100 shown in FIG. 1 can be implemented in a variety of devices, such as Figure 1 A block diagram of an example of a computing device 200 of a sending station or a receiving station of an encoding and decoding system of the encoding and decoding system 100 shown in FIG. 1. For example, the computing device 200 can implement one or both of the sending station 102 or the receiving station 104 shown in FIG. 1. The computing device 200 can be in the form of a computing system including multiple computing devices or in the form of one computing device. For example, the computing device 200 can be one of a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, a server computer, a game console, a wearable device, etc. Figure 1

[0036] The processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be one or more devices enabling manipulation or processing of information, existing or hereafter developed. For example, although the disclosed implementations can be implemented with one processor (e.g., the processor 202) as shown, advantages in speed and efficiency can be realized through the use of more than one processor.

[0037] The memory 204 in the computing device 200 can be a read-only memory device or a random access memory device in implementations. However, other suitable types of storage devices can be used as the memory 204. The memory 204 can include code and data 206 that are accessed by the processor 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, including at least one program that permits the processor 202 to perform the techniques described herein. For example, the application programs 210 can include applications 1 through N, which further include video or image coding software that performs some or all of the techniques described herein. The computing device 200 can also include secondary storage 214, which can be, for example, a memory card used with a mobile computing device. For example, images can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing. ​

[0038] The computing device 200 may also include one or more output devices, such as a display 218. In one example, the display 218 may be a touch-sensitive display, which combines a display with a touch-sensitive element capable of operating a sensor that senses touch input. The display 218 may be coupled to the processor 202 via a bus 212. In addition to or as an alternative to the display 218, other output devices may be provided that allow the user to program the computing device 200 or otherwise use the computing device 200. When the output device is a display or includes a display, the display may be implemented in various ways, including as a liquid crystal display, a cathode ray tube display, or a light-emitting diode display (such as an organic light-emitting diode display).

[0039] The computing device 200 may also include or communicate with the following devices: an image sensing device 220 (e.g., a camera) or other existing or subsequently developed image sensing devices capable of sensing images (such as images of a user operating the computing device 200). The image sensing device 220 may be positioned such that it is pointed toward a user operating the computing device 200. For example, the position and optical axis of the image sensing device 220 may be configured such that the field of view includes an area directly adjacent to and visible from the display 218.

[0040] The computing device 200 may also include or communicate with the following devices: a sound sensing device 222 (e.g., a microphone) or other existing or subsequently developed sound sensing devices capable of sensing sounds in the vicinity of the computing device 200. The sound sensing device 222 may be positioned such that it is directed toward a user operating the computing device 200 and may be configured to receive sounds, such as speech or other words, made by the user while the user is operating the computing device 200.

[0041] The implementation of computing device 200 may differ from that regarding Figure 2 The embodiments shown and described. In some embodiments, the operation of processor 202 may be distributed across multiple machines (where each machine may have one or more processors) that may be directly coupled or across a local area network or other network. In some embodiments, memory 204 may be distributed across multiple machines (such as network-based memory or memory in multiple machines performing the operation of computing device 200). In some embodiments, bus 212 of computing device 200 may consist of multiple buses. In some embodiments, secondary storage 214 may be directly coupled to other components of computing device 200 or may be accessed via a network, and may include integrated units such as memory cards or multiple units such as multiple memory cards.

[0042] Figure 3Ais a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 represents a typical video stream, which can be encoded, for example, using the transmitting station 102 shown in Figure 1 is encoded into a compressed bitstream and subsequently decoded into an output video stream, for example, using the receiving station 104 shown in Figure 1

[0043] The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of neighboring frames 304. While three frames are depicted as neighboring frames 304, the video sequence 302 can include other numbers of neighboring frames 304. The neighboring frames 304 can then be further subdivided into individual frames, for example, frame 306. At the next level, the frame 306 can be divided into a series of planes or slices 308. For example, the slices 308 can be subsets of the frame that are permitted to be processed in parallel, for example. The slices 308 can also be subsets of the frame that separate the video data into individual colors. For example, a frame 306 of color video data can include a luminance plane and two chrominance planes. The slices 308 can be sampled at different resolutions.

[0044] Whether or not the frame 306 is divided into slices 308, the frame 306 can be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more slices 308 of pixel data. The blocks 310 can also be any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. The terms block and macroblock can be used interchangeably herein, unless otherwise indicated.

[0045] Figure 3B is a diagram of an example of an image frame 312 to be encoded and subsequently decoded. The image frame 312 represents a typical image that can be encoded into a compressed bitstream or stored and subsequently decoded for rendering on a display, for example, using the transmitting station 102 shown in Figure 1 Figure 1 Figure 3A The image frame 312 can have the same form as the frame 306 shown in

[0046] Figure 4 is a block diagram of an encoder 400. The encoder 400 can be used to encode a video stream, such as the video stream 300 shown in Figure 2 for example, by providing a computer software program stored in a memory (e.g., the memory 204 shown in Figure 1 ​​​The encoder 400 is implemented in the transmitting station 102 as shown in FIG. 1. A computer software program can include machine instructions that, when executed by the processor of the processor 202 in the transmitting station 102, cause the transmitting station 102 to encode the video data in the manner described in FIG. 1. Figure 2 The computer software program can include machine instructions that, when executed by the processor of the processor 202 in the transmitting station 102, cause the transmitting station 102 to encode the video data in the manner described in FIG. 1. Figure 4 The video data is encoded in the manner described in FIG. 1. The encoder 400 can also be implemented as special-purpose hardware (e.g., integrated circuits) included in, for example, the transmitting station 102. In some implementations, the encoder 400 is a hardware encoder.

[0047] The encoder 400 has the following stages in a forward path (shown by solid connection lines) that perform various functions to produce an encoded bitstream or compressed bitstream 420 using the video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy coding stage 408. The encoder 400 can also include a reconstruction path (shown by dashed connection lines) to reconstruct frames for encoding future blocks. In the reconstruction path, the encoder 400 has the following stages that perform various functions: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a deblocking filter stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300. Figure 4

[0048] When the video stream 300 is presented for encoding, individual neighboring frames 304, such as the frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, individual blocks can be encoded using intra prediction (also referred to as intra prediction) or inter prediction (also referred to as inter prediction). In either case, a prediction block can be formed. In the case of intra prediction, the prediction block can be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter prediction, the prediction block can be formed from samples in one or more previously constructed reference frames.

[0049] Next, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also referred to as a residual or prediction residual). The transform stage 404 uses a block-based transform to transform the residual into transform coefficients in, for example, the frequency domain. The quantization stage 406 uses a quantizer value or quantization level to transform the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients. For example, the transform coefficients can be divided by the quantizer value and truncated.

[0050] ​The quantized transform coefficients are then entropy encoded by an entropy encoding stage 408. The entropy encoded coefficients, along with other information used to decode the block (which can include, for example, syntax elements such as those used to indicate the type of prediction used, the type of transform, motion vectors, quantizer values, etc.) are then output to a compressed bitstream 420. Various techniques can be used to format the compressed bitstream 420, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or an encoded video bitstream, and the terms will be used herein in an interchangeable manner.

[0051] The reconstruction path (shown by the dashed connection lines) can be used to ensure that the encoder 400 and the decoder 500 (described below with reference to Figure 5 perform the same reference frames to decode the compressed bitstream 420. The reconstruction path performs similar functions to those performed during the decoding process (described below with reference to Figure 5 perform the same reference frames to decode the compressed bitstream 420. The reconstruction path performs similar functions to those performed during the decoding process (described below with reference to

[0052] At the reconstruction stage 414, the predicted block predicted at the intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. A deblocking filter stage 416 can be applied to the reconstructed block to reduce distortions such as block artifacts. Embodiments and examples of deblocking filters used at the deblocking filter stage 416 are described below with reference to Figure 6

[0053] Embodiments of the encoder 400 can differ from the embodiments shown and described with reference to Figure 4 In particular, the encoder 400 as shown in Figure 4 is an example of an encoder for encoding video data, such as a video frame or a video block. However, in other embodiments, the encoder 400 can be an example of an encoder for encoding image data, such as an image frame or an image block.

[0054] ​In such implementations, the encoder 400 omits the intra / inter prediction stage 402. For example, the input image data is first processed at the transform stage 404 and then at the quantization stage 406 before entering the reconstruction path of stages 410, 412, 414, and 416. The output of the deblocking filter stage 416 can be sent to the transform stage 404 for further processing. If the reconstruction path is not needed, such as because the error level of the image frame meets a threshold, the reconstruction path can not be followed and the quantized image data can continue to the entropy encoding stage 408 and then output to the compressed bitstream 420. In some implementations of encoding image data using the encoder 400, the encoder 400 can omit the entropy encoding stage 408.

[0055] In some implementations, the encoder 400 can be a non-transform based encoder for image or video coding. In such implementations, the encoder 400 can directly quantize the residual signal without the transform stage 404 for certain blocks or frames. In some implementations, the quantization stage 406 and the dequantization stage 410 can be combined into a common stage.

[0056] Figure 5 is a block diagram of a decoder 500. As described above, the decoder 500 can be implemented in the receiving station 104 shown in FIG. 1, such as by providing a computer software program stored in a memory (e.g., the memory 204 shown in FIG. 2) that is executed by a processor (e.g., the processor 202 shown in FIG. 2). The computer software program can include machine instructions that, when executed by a processor, cause the receiving station 104 to decode video data in the manner described in FIG. 5. Figure 1 Figure 2 The computer software program can include machine instructions that, when executed by a processor, cause the receiving station 104 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in dedicated hardware (e.g., an integrated circuit) included in the receiving station 104, for example. Figure 2 Figure 5 The computer software program can include machine instructions that, when executed by a processor, cause the receiving station 104 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in dedicated hardware (e.g., an integrated circuit) included in the receiving station 104, for example.

[0057] Similar to the reconstruction path of the encoder 400 described above, the decoder 500 in one example includes the following stages that perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a deblocking filter stage 512, and an optional post-filter stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0058] ​​When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients, for example, by multiplying the quantized transform coefficients by the quantizer value, and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to the derivative residual created by the inverse transform stage 412 in the encoder 400. Using the header information decoded from the compressed bitstream 420, the decoder 500 can use the intra / inter prediction stage 508 to create the same prediction block that was created in the encoder 400, for example, at the intra / inter prediction stage 402.

[0059] At the reconstruction stage 510, the prediction block can be added to the derivative residual to create a reconstructed block. A deblocking filter stage 512 can be applied to the reconstructed block to reduce blocking artifacts (e.g., using deblocking filtering, sample adaptive offset filtering, other in-loop filter functionality, or combinations thereof). Additional details regarding the deblocking filter stage 512 are described below with reference to Figure 6 Embodiments and examples of deblocking filters used at the deblocking filter stage 512 are described. Other filtering can be applied to the reconstructed block. In this example, a post-filter stage 514 is applied to the reconstructed block to reduce blocking distortion, and the results are output as an output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the terms will be used interchangeably herein.

[0060] Embodiments of the decoder 500 can differ from the embodiments shown and described with reference to Figure 5 In some embodiments, the decoder 500 can use the post-filter stage 514 to omit or otherwise not process data. Further, while the decoder 500 as shown in Figure 5 is an example of a decoder for decoding encoded video data, such as an encoded video frame or an encoded video block, in other embodiments, the decoder 500 can be an example of a decoder for decoding encoded image data, such as an encoded image frame or an encoded image block.

[0061] In such embodiments, the decoder 500 omits the intra / inter prediction stage 508. For example, the output of the deblocking filter stage 512 can be sent to the dequantization stage 504 for further processing. In another example, the output of the deblocking filter stage 512 can be sent to the post-filter stage 514 and / or as output to a display for rendering in the case that the error level of the image frame meets a threshold. In some embodiments in which the decoder 500 is used to decode encoded image data, the decoder 500 can omit the entropy decoding stage 502.

[0062] Figure 6is a block diagram of an example of a spatially adaptive quantization perceptual deblocking filter 600 (hereafter deblocking filter 600) for encoding or decoding a frame, such as a video frame or an image frame. The deblocking filter 600 can be, for example, a deblocking filter used in the deblocking filter stage 416 shown in FIG. 4 in Figure 4 FIG. 4, a deblocking filter used in the deblocking filter stage 512 shown in FIG. 5 in Figure 5 FIG. 5, or both.

[0063] The deblocking filter 600 receives a reconstructed frame and adaptive quantization field data 602 and uses them to produce a filtered frame 604. The reconstructed frame includes data produced by reconstructing a number of decoded blocks that represent quantized and inverse transformed coefficients. For example, the reconstructed frame can be produced using the reconstruction stage 414 shown in FIG. 4 or the reconstruction stage 510 shown in FIG. 5. The adaptive quantization field data represents weights applied to quantization values used to encode those blocks. The adaptive quantization field data includes values that indicate quantization levels used to encode different regions of the frame. The values of the adaptive quantization field data are determined by analyzing the entire frame, such as determining regions to which a higher or lower weight is applied for quantization. Figure 4 Figure 5 The deblocking filter 600 includes an artifact removal stage 606, a psycho- visual modeling stage 608, and a parameter adjustment stage 610. The artifact removal stage 606 receives the reconstructed frame and the adaptive quantization field data 602. The artifact removal stage 606 filters the reconstructed frame to remove a number of block artifacts from the reconstructed frame. The adaptive quantization field data is used to control the filtering of the reconstructed frame by the artifact removal stage 606.

[0064] The artifact removal stage 606 compares the reconstructed frame to the original (e.g., pre-encoded) frame to determine differences between those frames as a result of encoding. In regions where those differences reflect that the reconstructed frame does not accurately represent the original frame, the artifact removal stage 606 applies a filter to reduce the number of block artifacts. The artifact removal stage 606 modulates the transition from small differences to large differences based on the adaptive quantization field data. For example, the artifact removal stage 606 can change the neighborhood size of the deblocking filter 600 based on the adaptive quantization field data. In another example, the artifact removal stage 606 can use a greater weight to values within the original frame when performing filtering, such as erring on the side of preserving more of the original frame data.

[0065] The artifact removal stage 606 compares the reconstructed frame to the original (e.g., pre-encoded) frame to determine differences between those frames as a result of encoding. In regions where those differences reflect that the reconstructed frame does not accurately represent the original frame, the artifact removal stage 606 applies a filter to reduce the number of block artifacts. The artifact removal stage 606 modulates the transition from small differences to large differences based on the adaptive quantization field data. For example, the artifact removal stage 606 can change the neighborhood size of the deblocking filter 600 based on the adaptive quantization field data. In another example, the artifact removal stage 606 can use a greater weight to values within the original frame when performing filtering, such as erring on the side of preserving more of the original frame data.

[0066] ​The deblocking filter 600 can be a directional filter such that the blocking artifacts removed by the artifact removal stage 606 can be filtered based on a filtering direction. In particular, the deblocking filter 600 (e.g., at the artifact removal stage 606 or at another stage (not shown) prior to the artifact removal stage 606) can determine a maximum invariant direction within a given region of the frame and filter along that maximum invariant direction. The maximum invariant direction refers to a line of direction of pixels for which the variation in their color, light, or other intensity is minimal. The filtering can include calculating the average of the pixels on each side of the line of direction and replacing the blocking artifacts with those averages.

[0067] The artifact removal stage 606 modulates the parameters of the deblocking filter 600 along the determined filtering direction by using adaptive quantization field data for one or more respective regions of the frame. The directional filter 600 can have a plurality of parameters for filtering the frame data, including, for example, a non-linearity selection parameter, a filter size parameter, or a direction sensitivity parameter. The non-linearity selection parameter reflects data or types of data to be preserved within the frame (e.g., not removed by filtering). The filter size parameter reflects the number of pixels to which the filter is applied in a given operation. The direction sensitivity parameter reflects the sensitivity of the filtering direction selection.

[0068] The adaptive quantization field data is used to modulate some or all of the parameters of the deblocking filter 600 based on the quantization levels used to encode different regions of the frame. Thus, during the artifact removal stage 606, the adaptive quantization field data indicates the quantization levels used to encode a given region of the frame. The artifact removal stage 606 then uses this quantization level information to control the removal of blocking artifacts within that given region. For example, the artifact removal stage 606 can modulate the non-linearity selection parameter used to filter a given region of the frame after determining that fine details (e.g., scratches on a painted surface) are to be preserved within the frame.

[0069] Modulating the non-linearity selection parameter includes changing a threshold value that indicates the difference between the original frame and the reconstructed frame. The threshold value can reflect, for example, the maximum acceptable difference for the given region of the frame based on the value of the adaptive quantization field data associated with that given region. For example, a larger difference between the original frame and the reconstructed frame within a given region of the frame can be acceptable according to the non-linearity selection parameter that applies a larger quantization weight in that given region.

[0070] The psychovisual modeling stage 608 uses software rules for processing the frame based on the visually perceptible quality of the frame after filtering at the artifact removal stage 606. The software rules of the psychovisual modeling stage 608 focus on three properties of vision: first, gamma correction should not be applied individually to each RGB channel; second, high frequency variations in blue data can not be accurately encoded; and third, regions within the frame that include a greater amount of visual noise can be less accurately encoded.

[0071] A first property of vision is driven by the overlap of the sensitivity spectra of the human eye's cone cells. For example, because there is some relationship between the amount of yellow light seen and the sensitivity to blue light, the variations in the blue data near the yellow data can be less accurately compressed. The YUV color space is defined as a linear transformation of gamma-compressed RGB, and thus is not powerful enough to model such phenomena. A second property of vision is driven by the color receptors of the human eye's retina. In particular, the human eye has a lower spatial resolution in blue than in red and green, and the retina has almost no blue receptors in the high-resolution area. A third property of vision is defined based on the relationship between visibility and near-end visual activity. That is, the visibility of fine structure in an image region can depend on the amount of visual activity near that region.

[0072] Although the software rules of the psycho-visual modeling stage 608 reference three properties of vision to describe, other numbers of visual properties, other rules related to video or image encoding or perceptibility, or combinations thereof can be used to define or otherwise configure the software rules of the psycho-visual modeling stage 608.

[0073] Accordingly, the software rules of the psycho-visual modeling stage 608 can be used to determine a number of block artifacts that are preserved within some or all of the regions of the frame. The psycho-visual modeling stage 608 next compares the number of block artifacts to a psycho-visual model threshold that reflects a maximum number of visually perceptible artifacts to include in the frame. The value of the psycho-visual model threshold can be configured by default or empirically, such as by iterating the deblocking filter 600 over N frames. If the psycho-visual model threshold is exceeded such that the number of block artifacts preserved within the frame after filtering at the artifact removal stage 606 is too large, the psycho-visual modeling stage 608 sends the filtered frame and associated data to the parameter adjustment stage 610. Otherwise, the filtered frame 604 is output.

[0074] The psycho-visual model threshold can also or instead reflect a maximum acceptable difference between the original frame and the frame after processing at the artifact removal stage 606. For example, the psycho-visual modeling stage 608 can use the psycho-visual model threshold as a basis for comparing coefficient or pixel values within particular regions of the original frame and the frame after processing at the artifact removal stage 606. If this comparison indicates that the coefficient or pixel value differences between these frames are less than the psycho-visual model threshold, the differences are preserved. However, if this comparison indicates that those differences are not less than the psycho-visual model threshold, the psycho-visual modeling stage 608 can cause the frame to undergo further filtering at the artifact removal stage 606.

[0075] The parameter adjustment stage 610 adjusts some of the parameters used by the artifact removal stage 606 before returning the frame to the artifact removal stage 606 for further filtering. The parameter adjustment stage 610 can, for example, adjust one or more values of the adaptive quantization field data. For example, based on the results of the psycho-visual modeling stage 608, the parameter adjustment stage 610 can increase or decrease an adaptive quantization field data value for a given region of the frame, such as to correspondingly increase or decrease the amount of quantization within the given region.

[0076] In another example, the parameter adjustment stage 610 can adjust one or more parameters of the deblocking filter 600 itself. For example, based on a visually perceptible quality of the filtered frame, such as can be determined as a result of the psycho-visual modeling stage 608, the parameter adjustment stage 610 can adjust a non-linearity selection parameter of the deblocking filter 600 for certain regions of the frame.

[0077] Adjustments to the adaptive quantization field data can be limited by an error level definition that represents a maximum quantization error for the frame. For example, in the event that an adjustment to the adaptive quantization field data results in a total quantization error for the frame that exceeds the error level definition, the adjustment is either discarded or offset by a corresponding adjustment to other regions of the frame. For example, in the event that a determination is made to adjust a first region of the frame by increasing a quantization weight for the region by X, and in the event that the increase results in a total quantization error that exceeds the error level definition, a corresponding determination is also made to adjust a second region of the frame by decreasing a quantization weight for the second region by X.

[0078] Adjustments determined or otherwise made at the parameter adjustment stage 610 are looped back to the artifact removal stage 606 for further filtering. The deblocking filter 600 can iterate through some or all of the artifact removal stage 606, the psycho-visual modeling stage 608, or the parameter adjustment stage 610 multiple times before outputting the final filtered frame 604. Iterating filtering of a frame using the deblocking filter 600 can include developing a model of the deblocking filter 600. For example, the deblocking filter 600 can include functionality to learn a type of filtering to apply to a given frame and a type of adjustment to make for a given frame. When another frame is received for filtering, the deblocking filter 600 can use the learned model to make multiple filtering applications or adjustments at a time, such as to improve processing speed.

[0079] Implementations of the deblocking filter 600 can be used in conjunction with the techniques described with reference to FIGS. 1-4, and vice versa. For example, the deblocking filter 600 can be used in conjunction with the techniques described with reference to FIGS. 1-4 to filter a frame of video data. Figure 6The illustrated and described embodiments differ. In some embodiments, the psychovisual modeling stage 608 and the parameter adjustment stage 610 can be external to the deblocking filter 600. For example, the deblocking filter 600 can only include functionality to perform the operations of the artifact removal stage 606. The deblocking filter 600 can thus output the filtered frame 604 produced using the artifact removal stage 606 to the psychovisual modeling stage 608, which can then process the filtered frame 604 to determine whether to re-filter the filtered frame 604 using the deblocking filter 600.

[0080] In some such embodiments, the psychovisual modeling stage 608 can process the reconstructed frame and adaptive quantization field data 602 prior to the artifact removal stage 606. For example, the psychovisual modeling stage 608 can process the reconstructed frame and adaptive quantization field data 602 to determine the number of blocking artifacts to remove from the reconstructed frame and the locations of those blocking artifacts within the frame. The artifact removal stage 606 then removes those blocking artifacts and outputs the filtered frame 604. The parameter adjustment stage 610 can then receive the filtered frame 604 output from the deblocking filter 600 and process the filtered frame 604 to determine whether to adjust the adaptive quantization field data and re-filter the filtered frame 604 using the adjusted adaptive quantization field data.

[0081] In the case of a determination to adjust the adaptive quantization field data and re-filter the filtered frame 604, the parameter adjustment stage 610 adjusts the respective values of the adaptive quantization field data accordingly. The parameter adjustment stage 610 then sends the filtered frame and the adjusted adaptive quantization field data back to the deblocking filter 600 for reprocessing by the artifact removal stage 606. In the case of a determination not to adjust the adaptive quantization field data and not to re-filter the filtered frame 604, the parameter adjustment stage 610 causes the filtered frame 604 to be output to, for example, a compressed bitstream for storage, for further processing, for rendering at a display, etc.

[0082] In some embodiments, the performance of the psychovisual modeling stage 608 can differ depending on the type of data that the deblocking filter 600 is processing. For example, in the case that the deblocking filter 600 is processing image data (e.g., such that the reconstructed frame is a reconstructed image), the psychovisual modeling stage 608 can use a larger threshold to determine visual perceptibility because the filtered frame 604 will be displayed for a long time. However, in the case that the deblocking filter 600 is processing video data (e.g., such that the reconstructed frame is a reconstructed video frame), the psychovisual modeling stage 608 can use a smaller threshold to determine visual perceptibility because the filtered frame 604 will only be displayed for a very short time.

[0083] In some implementations, the deblocking filter 600 can filter the frame in addition to determining and using the least invariant direction. For example, a machine learning algorithm can be used to determine the optimal filtering direction to use at the artifact removal stage 606. In another example, data indicating the filtering direction to be used can be explicitly signaled from the encoder to the decoder, such as within the frame header.

[0084] Figure 7 This is a block diagram representing the various parts of frame 700. This frame can be a video frame, for example... Figure 3A The video frame 306 shown; or the image frame, for example, Figure 3B Image frame 312 is shown. As shown, frame 700 includes four 64×64 blocks 710 arranged in two rows and two columns in a matrix or Cartesian plane. In some embodiments, the 64×64 block may be the maximum code unit, N=64. Each 64×64 block may include four 32×32 blocks 720. Each 32×32 block may include four 16×16 blocks 730. Each 16×16 block may include four 8×8 blocks 740. Each 8×8 block 740 may include four 4×4 blocks 750. Each 4×4 block 750 may include 16 pixels, which may be represented in four rows and four columns in each corresponding block of the Cartesian plane or matrix.

[0085] Pixels may include information representing the image captured in frame 700, such as luminance information, color information, and position information. In some embodiments, a block such as the 16×16 pixel block shown may include a luminance block 760, which may include luminance pixels 762; and two chrominance blocks 770, 780, such as a U or Cb chrominance block 770 and a V or Cr chrominance block 780. Chroma blocks 770, 780 may include chrominance pixels 790. For example, luminance block 760 may include 16×16 luminance pixels 762 and each chrominance block 770, 780 may include 8×8 chrominance pixels 790 as shown. Although one arrangement of blocks is shown, any arrangement can be used. Figure 7 An N×N block is shown; in some embodiments, an N×M block may be used. For example, 32×64 blocks, 64×32 blocks, 16×32 blocks, 32×16 blocks, or blocks of any other size may be used. In some embodiments, N×2N blocks, 2N×N blocks, or combinations thereof may be used.

[0086] In some implementations, coding the frame 700 can include in-order block-level coding. In-order block-level coding can include coding blocks of the frame in order, such as in a raster scan order, where blocks can be identified and processed starting from a top-left corner of the frame or a block in a portion of the frame and proceeding along the rows from left to right and from top row to bottom row, identifying each block in turn for processing. For example, the 64x64 block in the top row and left column of the frame can be the first block coded, and the 64x64 block immediately to the right of the first block can be the second block coded. The second row from the top can be the second row coded, such that the 64x64 block in the left column of the second row can be coded after the 64x64 block in the right-most column of the first row.

[0087] In some implementations, coding the blocks of the frame 700 can include using quadtree coding, which can include coding smaller block units within a block in a raster scan order. For example, the 64x64 block shown in the lower-left corner of the portion of the frame 700 can be coded using quadtree coding, where a top-left 32x32 block can be coded, then a top-right 32x32 block can be coded, then a bottom-left 32x32 block can be coded, and then a bottom-right 32x32 block can be coded. Each 32x32 block can be coded using quadtree coding, where a top-left 16x16 block can be coded, then a top-right 16x16 block can be coded, then a bottom-left 16x16 block can be coded, and then a bottom-right 16x16 block can be coded.

[0088] Each 16x16 block can be coded using quadtree coding, where a top-left 8x8 block can be coded, then a top-right 8x8 block can be coded, then a bottom-left 8x8 block can be coded, and then a bottom-right 8x8 block can be coded. Each 8x8 block can be coded using quadtree coding, where a top-left 4x4 block can be coded, then a top-right 4x4 block can be coded, then a bottom-left 4x4 block can be coded, and then a bottom-right 4x4 block can be coded. In some implementations, the 8x8 blocks can be omitted for 16x16 blocks, and the 16x16 blocks can be coded using quadtree coding, where a top-left 4x4 block can be coded, then the other 4x4 blocks in the 16x16 block can be coded in a raster scan order.

[0089] In some implementations, coding the frame 700 can include encoding information included in an original or input frame by, for example, omitting some information in the original frame from a corresponding encoded frame. For example, coding can include reducing spectral redundancy, reducing spatial redundancy, reducing temporal redundancy, or a combination thereof.

[0090] In some implementations, reducing spectral redundancy can include using a color model based on a luminance component (Y) and two chrominance components (U and V or Cb and Cr), which can be referred to as a YUV or YCbCr color model or color space. Using a YUV color model can include using a relatively large amount of information to represent a luminance component of a portion of a frame and using a relatively small amount of information to represent each corresponding chrominance component of the portion of the frame. For example, a portion of a frame can be represented by a high resolution luminance component (which can include a 16x16 block of pixels) and two lower resolution chrominance components (each of which represents the portion of the frame as an 8x8 block of pixels). A pixel can indicate a value, such as a value in a range from 0 to 255, and can be stored or transmitted using, for example, eight bits. Although the present disclosure is described with reference to a YUV color model, another color model can be used.

[0091] In some implementations, reducing spatial redundancy can include transforming a block into a frequency domain using, for example, a discrete cosine transform. For example, a unit of an encoder, such as transform stage 404 shown in FIG. 4, can perform a discrete cosine transform using transform coefficient values based on spatial frequencies. Figure 4

[0092] In some implementations, reducing temporal redundancy can include using similarities between frames to encode a frame using a relatively small amount of data based on one or more reference frames, which can be previously encoded, decoded, and reconstructed frames of a video stream. For example, a block or pixel of a current frame can be similar to a spatially corresponding block or pixel of a reference frame. In some implementations, a block or pixel of a current frame can be similar to a block or pixel of a reference frame at a different spatial location, and reducing temporal redundancy can include generating motion information indicating a spatial difference or translation between a location of a block or pixel in a current frame and a corresponding location of a block or pixel in a reference frame.

[0093] ​In some implementations, reducing temporal redundancy can include identifying a portion of a reference frame that corresponds to a current block or pixel of a current frame. For example, a search can be performed of a reference frame or a portion of a reference frame that can be stored in memory to identify a portion of the reference frame to use to generate a predictor that is used to encode the current frame or a pixel of the current frame with maximum efficiency. For example, the search can identify a portion of the reference frame for which a difference in pixel values between the current block and a predicted block generated based on the portion of the reference frame is minimized and can be referred to as a motion search. In some implementations, the portion of the reference frame searched can be limited. For example, a portion of the reference frame searched (which can be referred to as a search region) can include a limited number of rows of the reference frame. In an example, identifying a portion of a reference frame to use to generate a predictor can include calculating a cost function, such as a sum of absolute differences (SAD), between pixels of the portion of the search region and pixels of the current block.

[0094] In some implementations, a spatial difference between a location of a portion of a reference frame used to generate a predictor in the reference frame and a current block in a current frame can be represented as a motion vector. A difference in pixel values between a predictor block and the current block can be referred to as difference data, residual data, prediction error, or a residual block. In some implementations, generating a motion vector can be referred to as motion estimation, and pixels of the current block can be indicated based on location using Cartesian coordinates as f x，y . Similarly, pixels of a search region of the reference frame can be indicated based on location using Cartesian coordinates as r x，y A motion vector (MV) for the current block can be determined based on, for example, a SAD between pixels of the current frame and corresponding pixels of the reference frame.

[0095] Although described herein with reference to a matrix or Cartesian representation of a frame for clarity, a frame or any combination thereof can be stored, transmitted, processed in any data structure such that pixel values can be effectively represented for the frame or image. For example, a frame or any combination thereof can be stored, transmitted, processed in a two-dimensional data structure such as a matrix as shown or in a one-dimensional data structure such as a vector array. In implementations, a representation of a frame, such as a two-dimensional representation as shown, can correspond to a physical location when the frame is rendered as an image. For example, a location in a top-left corner of a block in the frame can correspond to a physical location in a top-left corner of an image when the frame is rendered as the image.

[0096] As described above, frame 700 can be a frame of a video sequence, or it can be an image. Regardless of whether frame 700 represents image data or video data, frame 700 includes a single picture to encode or decode. The picture included in frame 700 can have different types of detail in different regions. A region of a frame as described herein refers to an M x N size region of the frame, where M and N can be the same or different values. For example, a region of a frame can be a single block (e.g., an 8 x 8 block) within that frame. In another example, a region of a frame can be multiple blocks within the frame. In yet another example, different regions of a frame to encode can be blocks of different sizes (e.g., some 8x8, some 4x4, some 16x16, etc.).

[0097] For example, frame 700 can include a picture of a forest and a sky above the forest. The forest can be shown in the lower two 64 x 64 blocks 710, while the sky is shown in the upper two 64 x 64 blocks 710. The data included in the lower two 64 x 64 blocks 710 can be more detailed than the data included in the upper two 64 x 64 blocks 710. For example, the depiction of the forest can include many small details of the branches or leaves of the trees, as well as other flora and fauna present in the forest. In contrast, the depiction of the sky can reflect only a gradient of shades of blue. In this way, most of the information in frame 700 is located in the lower two 64 x 64 blocks 710.

[0098] Encoding or decoding frame 700 can include quantizing different regions (e.g., different blocks 710, 720, 730, 740, 750) of frame 700 by applying different weights to quantized values of frame 700 based on the data included in those regions. For example, a first region of an image that includes more information (e.g., more detail) can be quantized using a smaller weight than a second region that includes less information (e.g., less detail). The information contained within the first region is less quantized than the information contained within the second region. This results in less loss of information within the first region than the second region.

[0099] In the case of the example where frame 700 shows a forest and a sky, adaptive quantization field data for frame 700 can be generated to reflect that greater quantization was used to represent information within the blocks that depict the sky, and less quantization was used to represent information in the blocks that depict the forest. More specifically, the values of the adaptive quantization field data for the blocks that include sky information reflect that those blocks were quantized using greater weights than the blocks that include forest information. The values of the adaptive quantization field data for all four 64 x 64 blocks 710 can be controlled by an error level definition that represents the maximum quantization error resulting from the encoding of frame 700.

[0100] Referring now to Figure 8 and Figure 9 Techniques for encoding or decoding a video frame are described.Figure 8 is a flowchart of an example of a technique 800 for encoding or decoding a video frame or image frame using a spatially adaptive quantization-aware deblocking filter. Figure 9 is a flowchart of an example of a technique 900 for iteratively filtering a video frame or image frame from adaptive quantization field data.

[0101] One or both of the technique 800 or the technique 900 can be implemented as, for example, a software program that can be executed by a computing device such as the transmitting station 102 or the receiving station 104 shown in Figure 1 or otherwise by the computing device 200 shown in Figure 2 For example, the software program can be or otherwise include an encoder such as the encoder 400 shown in Figure 4 or a decoder such as the decoder 500 shown in Figure 5 For example, the software program can be or otherwise include an encoder such as the encoder 400 shown in

[0102] The software program can include machine-readable instructions that can be stored in a memory such as the memory 204 or the auxiliary storage 214 shown in Figure 2 and that, when executed by a processor such as the processor 202 shown in Figure 2 can cause the computing device to perform one or both of the technique 800 or the technique 900.

[0103] The technique 800 and / or the technique 900 or an encoder and / or a decoder (e.g., the encoder 400 and / or the decoder 500) used to perform the technique 800 and / or the technique 900 can be implemented using special-purpose hardware or firmware (e.g., integrated circuits). As explained above, some computing devices can have multiple memories or processors and can use multiple processors, memories, or both to distribute the operations described in the technique 800 and the technique 900.

[0104] To simplify the illustration, the technique 800 and the technique 900 are each depicted and described as a series of steps or operations. However, the steps or operations according to the present disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein can be used. Further, not all illustrated steps or operations can be required to implement a technique according to the disclosed subject matter.

[0105] Referring first to Figure 8 , a technique 800 for encoding or decoding a video frame or image frame using a spatially adaptive quantization-aware deblocking filter is shown. The technique 800 can be performed to encode a video frame or image into, for example, a bitstream. Alternatively, the technique 800 can be performed to, for example, decode an encoded video frame or encoded image such as from a bitstream. For example, during encoding, a video frame or image can be received from, for example, a video source or a bitstream. Figure 4The technique 800 can be performed by the reconstruction path (e.g., stages 410, 412, 414, and 416) of the encoder 400 shown in FIG. 4. In another example, during decoding, the technique 800 can be performed by stages 504, 506, 510, and 512 of the decoder 500 shown in FIG. 5. Figure 5 The stages 504, 506, 510, and 512 of the decoder 500 shown in FIG. 5 perform.

[0106] At 802, quantized transform coefficients associated with a video frame or image (hereinafter referred to as a frame) to be encoded or decoded are dequantized and inverse transformed. The quantized transform coefficients represent pixel values of the original video frame or original image after the pixel values have been transformed and quantized during encoding. The quantized transform coefficients are coefficients of a block of the video frame or image. By dequantizing and inverse transforming the quantized transform coefficients, a decoded block is produced. The decoded block can be, for example, a derivative residual block, such as where the frame is a video frame. At 804, the decoded video block is reconstructed into a reconstructed frame.

[0107] At 806, the reconstructed frame is filtered according to adaptive quantization field data associated with the frame. The adaptive quantization field data can be produced at a quantization stage of an encoder, for example, within a bitstream or received by a decoder. Filtering the reconstructed frame according to the adaptive quantization field data includes applying a deblocking filter to the reconstructed frame to remove some amount of blocking artifacts from the reconstructed frame. The amount of blocking artifacts removed by the deblocking filter is controlled by the adaptive quantization field data. As a result of filtering the reconstructed frame, a first filtered frame is produced.

[0108] One or more parameters of the deblocking filter can be used to control or otherwise configure the performance of the deblocking filter. As such, applying the deblocking filter to the reconstructed frame according to the adaptive quantization field data can include applying some or all of the filter parameters to the reconstructed frame, where those filter parameters are defined based on the adaptive quantization field data. These parameters can include, for example, a non-linearity selection parameter, a filter size parameter, a directional sensitivity parameter, and the like or combinations thereof. Moreover, assuming that the filter parameters are defined based on the adaptive quantization field data, the adaptive quantization field data can be used to modulate one or more parameters of the deblocking filter.

[0109] Modulating the parameters of the deblocking filter using the adaptive quantization field data can include modulating a non-linearity selection parameter of the deblocking filter according to the adaptive quantization field data to determine what type of data is to be preserved within the reconstructed frame. For example, the non-linearity selection parameter can be modulated to determine what type of data is to be preserved within the reconstructed frame. As a result, the deblocking filter does not process coefficients used to represent that type of data within the reconstructed frame.

[0110] For example, the non-linear selection parameter can be modulated according to a psycho-visual model. The psycho-visual model can indicate that certain types of data are more or less visually perceptible. Thus, the non-linear selection parameter can be modulated based on the psycho-visual model to cause more visually perceptible data within the frame to be preserved and less visually perceptible data within the frame to be modified.

[0111] The adaptive quantization field data indicates to the deblocking filter regions of the reconstructed frame that were processed using relatively higher or relatively lower quantization values. As described above, the adaptive quantization field data represents weights applied to quantization values used to encode blocks of the frame. Thus, the quantization values applied to local regions of the reconstructed frame are used to guide the processing of the deblocking filter.

[0112] For example, the reconstructed frame can include a first region that was processed using relatively higher quantization values during encoding and a second region that was processed using relatively lower quantization values during encoding. Thus, the deblocking filter can use the adaptive quantization field data for the reconstructed frame to apply a relatively stronger filter to the first region of the reconstructed frame and a relatively weaker filter to the second region. As a result, the deblocking filter removes a greater amount of blocking artifacts from the first region than from the second region.

[0113] At 808, a number of blocking artifacts remaining within the first filtered frame is determined. Determining the number of blocking artifacts remaining within the first filtered frame includes processing data of the first filtered frame using a psycho-visual model. The psycho-visual model may, for example, be a model configured to process video frame or image data based on human perceptibility. Reference is made to Figure 9 Implementations and examples for determining to adjust the adaptive quantization field data are further described.

[0114] At 810, the adaptive quantization field data is adjusted. Adjusting the adaptive quantization field data includes changing (e.g., increasing or decreasing) the weight applied to quantization values for at least some regions of the first filtered frame. For example, adjusting the adaptive quantization field data can include increasing a value of the adaptive quantization field data that is associated with one or more decoded blocks used to produce the reconstructed frame. For example, the value can reflect a weight applied to a quantization value in a particular block of those decoded blocks, another value produced by applying the weight to the quantization value, or another value reflected by the adaptive quantization field data.

[0115] Adjustments to the adaptive quantization field data can be limited by data associated with the frame being encoded or decoded. For example, increases to the adaptive quantization field data can be limited by an error level definition associated with the frame. The error level definition represents a maximum quantization error resulting from encoding of the frame. As such, the adaptive quantization field data can not be increased in a manner that causes the quantization error of the frame to exceed the error level definition. In another example, decreases to the adaptive quantization field data can be limited (e.g., during an encoding operation) by a file size definition for the frame. As such, the adaptive quantization field data can not be decreased in a manner that causes the total file size of the resulting encoded frame to exceed the file size definition.

[0116] At 812, the first filtered frame resulting from the earlier filtering of the reconstructed frame is refiltered according to the adjusted adaptive quantization field data. Refiltering the first filtered frame according to the adjusted adaptive quantization field data includes applying a deblocking filter to the first filtered frame to remove a quantity of blocking artifacts from the first filtered frame. The quantity of blocking artifacts removed by the deblocking filter is controlled by the adjusted adaptive quantization field data. As a result of filtering the first filtered frame, a second filtered frame is produced. Refiltering can be performed in the same or substantially the same manner as the earlier filtering of the reconstructed frame.

[0117] In some implementations of the technique 800 for decoding an encoded video frame or encoded image, the technique 800 includes decoding syntax data associated with the encoded video frame or encoded image, such as from a bitstream. The syntax data includes quantized transform coefficients for encoded blocks of the encoded video frame or encoded image. The syntax data also includes adaptive quantization field data.

[0118] The syntax data can be some or all of the data stored in a bitstream or another data store and associated with the encoded video frame or encoded image. For example, the syntax data can include quantized transform coefficients for some or all of the encoded blocks of the encoded video frame or encoded image. In another example, the syntax data can include metadata, such as from a frame header for the encoded video frame or from an image header for the encoded image.

[0119] In some implementations, the technique 800 includes determining whether to make an adjustment at the adaptive quantization field data prior to adjusting the adaptive quantization field data. For example, the determination can be made by comparing the quantity of blocking artifacts remaining in the first filtered frame to a threshold, such as a psycho-visual model threshold.

[0120] In some implementations, the technique 800 includes outputting a second filtered frame produced by refiltering the first filtered frame using the adjusted adaptive quantization field. For example, when the technique 800 is performed to encode a video frame or image, the second filtered frame can be output to a compressed bitstream, such asFigure 4 The compressed bitstream 420 shown may be stored, for example, at a receiving station 104 for later transmission. In another example, when technique 800 is performed to decode encoded video frames or encoded images, a second filtered frame may be output to an output video stream, such as... Figure 5 The output video stream 516 shown, or in a manner such as in Figure 2 The display shown is rendered at position 218.

[0121] In some implementations, technique 800 includes further adjustment of the adaptive quantization field data. For example, in response to re-filtering the reconstructed frame (e.g., to generate second filtered frame data), the adaptive quantization field data can be further adjusted, for example, based on the number of block artifacts within the second filtered video frame. For example, it can be determined (e.g., based on a threshold associated with a psychovisual model) that the first filtered frame data generated by the first filtering includes too many block artifacts. Further adjustment of the adaptive quantization field data can be based on this determination.

[0122] After further adjustments to the adaptive quantization field data, a deblocking filter can be used to further re-filter the reconstructed frame data, such as to generate a third filtered frame. The third filtered frame can then be output to, for example, a compressed bitstream, a compressed image memory, or for further processing (e.g., during encoding), or to an output video stream or for image rendering (e.g., during decoding). In some implementations, the third filtered frame can be output together with the second filtered frame. In some implementations, the third filtered frame can be output instead of the second filtered frame.

[0123] Next reference Figure 9 This illustrates a technique 900 for iteratively filtering video frames or image frames based on adaptive quantization field data. Technique 900 can be performed during encoding operations, such as encoding video frames or images into a bitstream. Alternatively, technique 900 can be performed during decoding operations, such as decoding encoded video frames or encoded images from a bitstream. For example, during encoding, techniques 900 can be used... Figure 6 The deblocking filter 600 shown (when included in the decoder) or in Figure 4 The deblocking filter stage 416 shown in the figure performs technique 900. In another example, during decoding, deblocking filter 600 (when included in the decoder) or Figure 5 The deblocking filter stage 512 shown is used to perform technique 900.

[0124] At 902, frame data and adaptive quantization field data are received. The frame data can be, for example, a reconstructed frame, which may be generated based on dequantized and inverse-transformed video or image coefficients. The adaptive quantization field data represents the weights applied to the quantization values ​​used to quantize those video or image coefficients. At 904, the frame data is filtered according to the adaptive quantization field data. (Refer to above) Figure 8 This describes the implementation and examples of filtering frame data based on adaptive quantization field data. At position 906, it identifies the number of block artifacts remaining in the filtered frame data.

[0125] At point 908, it is determined whether the number of block artifacts remaining in the filtered frame data exceeds a psychovisual model threshold. The psychovisual model threshold reflects the maximum number of visually perceptible block artifacts that can exist within a frame without causing a decrease in visual quality. Therefore, determining whether the number of block artifacts remaining in the filtered frame data exceeds the psychovisual model threshold involves using the psychovisual model to determine whether the number of block artifacts remaining in the filtered frame data exceeds the maximum number of visually perceptible block artifacts allowed within the filtered frame.

[0126] At 910, in response to determining that the number of block artifacts remaining in the filtered frame data exceeds a threshold of the psychovisual model, one or more values ​​of the adaptive quantization field data are increased, such as causing multiple quantizations to occur in one or more corresponding regions of the frame. Technique 900 then returns to 904, where the filtered frame data is further filtered based on the increased or already increased value of the adaptive quantization field data.

[0127] At 912, in response to determining that the number of block artifacts within the filtered frame data does not exceed a threshold of the psychovisual model, the filtered frame data is output. For example, in the case of performing technique 900 to encode the frame, from the encoder's reconstruction path (e.g., Figure 4 The stages 410, 412, and 414 shown in the diagram output filtered frame data. In another example, when performing technique 900 to decode the frame, the filtered frame data is output to the output video stream (e.g., ...). Figure 5 The output video stream 516 shown is output to a post-filtering stage (e.g., stage 514) before the output video stream.

[0128] In some implementations, after identifying the number of residual block artifacts within the filtered frame data, technique 900 includes determining whether the file size of the filtered frame data, including that number of residual block artifacts, exceeds a file size threshold. The file size threshold reflects a file size definition for the frame data, such as the maximum total file size of the encoded form of the frame data. Determining the file size of the filtered frame data may include encoding the filtered frame data into temporary encoded frames and identifying the total storage requirements of those temporary encoded frames.

[0129] In response to determining that the file size for the filtered frame data exceeds the file size threshold, one or more values of the adaptive quantization field data are decreased, such as causing less quantization to occur within one or more corresponding regions of the frame. In some implementations, the technique 900 can then return 904 to further filter the filtered frame data.

[0130] Figure 10 are illustrations of examples of the original video frame or image 1000A being reproduced using different filtering or no filter techniques. The original video frame or image 1000A represents a video frame or image prior to encoding and / or decoding by the encoding and decoding system 100 shown in FIG. 1. Figure 1 The first reproduction 1000B represents the original video frame or image 1000A after encoding and / or decoding without using a deblocking filter. The second reproduction 1000C represents the original video frame or image 1000A after encoding and / or decoding by using a constant deblocking filter. The third reproduction 1000D represents the original video frame or image 1000A after encoding and / or decoding using a spatially adaptive quantization perceptual deblocking filter (e.g., the deblocking filter 600 shown in FIG. 6). Figure 6 The fourth reproduction 1000E represents the original video frame or image 1000A after encoding and / or decoding using a spatially adaptive quantization perceptual deblocking filter (e.g., the deblocking filter 600 shown in FIG. 6) with a constant filter strength.

[0131] The above-described aspects of encoding and decoding illustrate some examples of encoding and decoding techniques and all or a portion of hardware components configured to perform the encoding and / or decoding techniques. However, it should be understood that encoding and decoding (as those terms are used in the claims) can mean compression, decompression, transformation, or any other processing or alteration of data.

[0132] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X includes A or B” is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances.

[0133] Also, the use of the articles “a” and “the” as used herein in reference to “one or more” of something is taken to mean that there is at least one and there can be more than one of something. Also, the use of terminology “implementation” or the use of the term “one implementation” throughout this disclosure is not intended to mean the same embodiment, unless described as such.

[0134] The implementation of transmitting station 102 and / or receiving station 104 (and the algorithms, methods, instructions, etc. stored thereon and / or executed thereon (including by encoder 400 and / or decoder 500 and including using technology 800 and / or technology 900)) can be implemented in hardware, software, or any combination thereof. Hardware may include, for example, a computer, intellectual property core, application-specific integrated circuit, programmable logic array, optical processor, programmable logic controller, microcode, microcontroller, server, microprocessor, digital signal processor, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the aforementioned hardware individually or in combination. The terms "signal" and "data" are used interchangeably. Furthermore, the portions of transmitting station 102 and receiving station 104 do not necessarily have to be implemented in the same manner.

[0135] The transmitting station 102 or the receiving station 104 can be implemented using a general-purpose computer or general-purpose processor with a computer program, which, when executed, performs any of the various methods, algorithms, and / or instructions described herein. Alternatively, for example, a special-purpose computer / processor can be used, which may include other hardware for performing any of the methods, algorithms, or instructions described herein.

[0136] Some or all of the embodiments of this disclosure may take the form, for example, a computer program product accessible from a computer-usable or computer-readable medium. A computer-usable or computer-readable medium may be, for example, any device capable of tangibly containing, storing, communicating, or transmitting a program for use by or in conjunction with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[0137] The above embodiments, implementations, and aspects have been described to facilitate understanding of this disclosure and are not intended to limit it. Rather, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which should be given the broadest interpretation permitted by law to include all such modifications and equivalent arrangements.

Claims

1. A method for decoding an encoded frame, the method comprising: decoding, from a bitstream into which the encoded frame is encoded, quantized transform coefficients of an encoded block of the encoded frame and adaptive quantization field data used to encode the encoded block, wherein the adaptive quantization field data represents weights applied to quantization values used to encode the encoded block of the encoded frame; producing a reconstructed frame, wherein producing the reconstructed frame comprises dequantizing and inverse transforming the quantized transform coefficients; filtering the reconstructed frame according to the adaptive quantization field data to produce a filtered frame based on the reconstructed frame, wherein filtering the reconstructed frame comprises modulating one or more of: a non-linear selection filter parameter, a filter size parameter, or a directional sensitivity filter parameter according to the adaptive quantization field data; and outputting the filtered frame for storage or display.

2. The method of claim 1, wherein, filtering the reconstructed frame according to the adaptive quantization field data comprises: producing filtering data by filtering the reconstructed frame according to the adaptive quantization field data; determining a number of block artifacts within the filtering data; adjusting at least some of the adaptive quantization field data based on the number of block artifacts within the filtering data; and producing the filtered frame by re-filtering the reconstructed frame according to the adjusted adaptive quantization field data.

3. The method of claim 2, wherein adjusting at least some of the adaptive quantization field data based on the number of block artifacts within the filtering data comprises: in response to determining that the number of block artifacts within the filtering data exceeds a threshold, increasing the adaptive quantization field data.

4. The method of claim 3, wherein, the threshold represents a maximum acceptable difference between a given region of the reconstructed frame and a corresponding region within an original frame that was encoded to produce the encoded frame.

5. The method of claim 4, wherein, the threshold is defined using a psycho-visual model that is used to analyze visually perceptible artifacts resulting from filtering.

6. The method of claim 2, wherein the filtering data is first filtering data, wherein producing the filtered frame by re-filtering the reconstructed frame according to the adjusted adaptive quantization field data comprises: producing second filtering data by filtering the reconstructed frame according to the adjusted adaptive quantization field data; determining a number of block artifacts within the second filtering data; further adjusting the adjusted adaptive quantization field data based on the number of block artifacts within the second filtering data; and producing the filtered frame by re-filtering the reconstructed frame according to the further adjusted adaptive quantization field data. the adjustments made to the adaptive quantization field data are limited by an error level definition that represents a maximum quantization error of the reconstructed frame.

7. The method of claim 2, wherein, 8. A method for decoding an encoded frame, the method comprising: receiving an encoded frame and adaptive quantization field data used to encode the encoded frame; producing a reconstructed frame using the encoded frame; ​ changing one or more of a non-linear selection filter parameter, a filter size parameter, or a directional sensitivity filter parameter available for filtering the reconstructed frame based on the adaptive quantization field data; producing filtered data by filtering a reconstructed frame according to the adaptive quantization field data and using the changed one or more of the non-linear selection filter parameter, the filter size parameter, or the directional sensitivity filter parameter; determining whether a number of block artifacts within the filtered data exceeds a threshold; in response to determining that the number of block artifacts within the filtered data exceeds a threshold, adjusting at least some of the adaptive quantization field data for further use in filtering the reconstructed frame; and producing a filtered frame by re-filtering the reconstructed frame according to the adjusted adaptive quantization field data.

9. The method of claim 8, wherein adjusting at least some of the adaptive quantization field data comprises: increasing a weight applied to quantization values used to process a first region of the reconstructed frame by a first amount; and increasing a weight applied to quantization values used to process a second region of the reconstructed frame by a second amount.

10. The method of claim 9, wherein, the first and second amounts are determined based on an amount by which the first and second regions of the reconstructed frame differ from corresponding regions within an original frame that was encoded to produce the encoded frame, the amount being a coefficient difference or a pixel value difference.

11. The method of claim 8, wherein, the change in the one or more of the non-linear selection filter parameter, the filter size parameter, or the directional sensitivity filter parameter is based on a visual perceptibility of data within the reconstructed frame.

12. The method of claim 11, wherein, the visual perceptibility of the data within the reconstructed frame is analyzed using a psychovisual model, wherein the threshold is defined using the psychovisual model.

13. The method of claim 12, wherein, changing the non-linear selection filter parameter results in certain types of data being preserved within the reconstructed frame.

14. The method of claim 13, wherein, the certain types of data within the reconstructed frame are a first set of reconstructed frame data, and remaining data within the reconstructed frame is a second set of reconstructed frame data, wherein the operation of filtering the reconstructed frame according to the adaptive quantization field data and using the changed one or more of the non-linear selection filter parameter, the filter size parameter, or the directional sensitivity filter parameter is limited to the second set of reconstructed frame data.

15. A method for decoding an encoded frame, the method comprising: producing a reconstructed frame from the encoded frame; producing filtered data by filtering the reconstructed frame according to adaptive quantization field data; adjusting the adaptive quantization field data of the encoded frame according to one or more details within the reconstructed frame, wherein adjusting the adaptive quantization field data according to the one or more details within the reconstructed frame comprises: adjusting at least some of the adaptive quantization field data based on a number of block artifacts within the filtered data; and controlling filtering of the reconstructed frame using the adjusted adaptive quantization field data.

16. The method of claim 15, wherein controlling filtering of the reconstructed frame using the adjusted adaptive quantization field data comprises: using the adaptive quantization field data to control one or both of a strength or a spatial size of the deblocking filter.

17. The method of claim 16, wherein the adaptive quantization field data indicates that a first region of the reconstructed frame is encoded using a first quantization value and a second region of the reconstructed frame is encoded using a second quantization value, wherein the first quantization value is higher than the second quantization value, wherein using the adaptive quantization field data to control one or both of a strength or a spatial size of the deblocking filter includes: filtering the first region of the reconstructed frame using a first filter strength and filtering the second region of the reconstructed frame using a second filter strength, wherein the first filter strength is stronger than the second filter strength.

18. The method of claim 15, wherein adjusting at least some of the adaptive quantization field data based on a number of the blocking artifacts within the filter data includes: determining that the number of the blocking artifacts within the filter data exceeds a threshold defined using a psycho-visual model for analyzing visually perceptible artifacts resulting from filtering.

19. The method of claim 15, wherein, the adjustment made to the adaptive quantization field data is limited by an error level definition, the error level definition representing a maximum quantization error of the reconstructed frame.

Images