Method for encoding a current block of image or video data

Abstract

A method for encoding a current block of image or video data is disclosed. This disclosure relates to a sparse matrix representation using non-zero coefficient boundaries. The sparse matrix representation of image or video data used for encoding or decoding uses boundaries of non-zero coefficients within the image or video data. A bounding box surrounds each non-zero coefficient within the image or video block. The coefficients surrounded by the bounding box are encoded into a bitstream along with size information that can be used to identify the bounding box within the image or video block during decoding. Coefficients not surrounded by the bounding box are not specifically encoded in the bitstream. The size information represents one or more of the shape, size, or position of the bounding box within the image or video block. The bounding box can be identified based on the scan order used to process the coefficients within the image or video block. The bounding box can be rectangular or non-rectangular.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese Invention Patent Application No. 202010940205.3, filed on September 9, 2020. Technical Field

[0003] This disclosure relates to sparse matrix representations using non-zero coefficient boundaries. Background Technology

[0004] Image content represents a large amount of online content. A webpage can include multiple images, and a significant portion of the time and resources spent rendering a webpage are dedicated to rendering those images for display. The amount of time and resources required to receive and render an image for display depends in part on how the image is encoded. Thus, images, and therefore webpages containing them, can be rendered faster by reducing the total data size of the image. Various schemes have been proposed to reduce the total data size of images, including encoding or decoding techniques.

[0005] Digital video streams can represent video using a series of frames or still images. Digital video can be used in a variety of applications, including video conferencing, high-definition video entertainment, video advertising, or sharing user-generated video. Digital video streams can contain large amounts of data and consume significant computing or communication resources of computing devices for processing, transmitting, or storing the video data. Various schemes have been proposed to reduce the amount of data in video streams, including encoding or decoding techniques. Summary of the Invention

[0006] This paper specifically discloses systems and techniques for encoding images or videos using sparse matrix representations that employ boundaries with non-zero coefficients.

[0007] A method for encoding an image block according to embodiments of the present disclosure includes: transforming data of the image block to generate transformed image data; quantizing the transformed image data to generate quantized image data, wherein the quantized image data includes coefficients arranged in a two-dimensional matrix; identifying bounding boxes of non-zero coefficients of the quantized image data enclosing within the two-dimensional matrix, wherein zero coefficients of the quantized image data are located outside the bounding boxes within the two-dimensional matrix; entropy encoding the non-zero coefficients enclosing within the bounding boxes into an encoded bitstream; and including size information of the bounding boxes in the encoded bitstream.

[0008] A method for decoding an encoded image block according to embodiments of the present disclosure includes: decoding bounding box size information from an encoded bitstream including the encoded image block; decoding syntax elements representing the encoded image block from the encoded bitstream using the size information, wherein the syntax elements correspond to non-zero value coefficients of image data enclosed within the bounding box during the encoding of the image data, wherein the syntax elements are arranged in a format other than a two-dimensional matrix format; dequantizing the syntax elements to generate transformed image data, wherein the transformed image data includes coefficients arranged in a two-dimensional matrix; performing an inverse transform on the transformed image data to generate decoded image data; and outputting the decoded image data for storage or display.

[0009] A method for encoding video blocks according to embodiments of the present disclosure includes: generating a prediction block for data of the video block; generating a prediction residual for the data of the video block using the prediction block; transforming the prediction residual to generate transform coefficients; quantizing the transform coefficients to generate quantized transform coefficients, wherein the quantized transform coefficients are arranged in a two-dimensional matrix; identifying bounding boxes of non-zero coefficients of the quantized transform coefficients enclosed within the two-dimensional matrix, wherein zero coefficients of the quantized transform coefficients are located outside the bounding boxes within the two-dimensional matrix; entropy encoding the non-zero coefficients enclosed within the bounding boxes into an encoded bitstream; and including size information of the bounding boxes in the encoded bitstream.

[0010] A method for decoding an encoded video block according to embodiments of the present disclosure includes: decoding bounding box size information from an encoded bitstream including the encoded video block; decoding syntax elements representing the encoded video block from the encoded bitstream using the size information, wherein the syntax elements correspond to non-zero coefficients of the video data enclosed within the bounding box during encoding of the video data, wherein the syntax elements are arranged in a format other than a two-dimensional matrix format; dequantizing the syntax elements to generate transform coefficients, wherein the transform coefficients include coefficients arranged in a two-dimensional matrix; performing an inverse transform on the transform coefficients to generate a prediction residual; reconstructing the prediction residual to generate a decoded video block; and outputting the decoded video block for storage or display. Attached Figure Description

[0011] This disclosure can be best understood in conjunction with the accompanying drawings, and is further explained in the following detailed description. It should be emphasized that, by convention, the features in the drawings are not drawn to scale. Instead, for clarity, the dimensions of the features have been arbitrarily enlarged or reduced.

[0012] Figure 1 This is a block diagram of an example image or video coding system.

[0013] Figure 2 It is a block diagram of an example of the internal configuration of a computing device that can be used in an image or video coding system.

[0014] Figure 3 This is an example diagram of an image that is to be encoded and subsequently decoded.

[0015] Figure 4 This is a diagram of an example of a video stream that is to be encoded and subsequently decoded.

[0016] Figure 5 This is a block diagram of an example image encoder.

[0017] Figure 6 This is a block diagram of an example image decoder.

[0018] Figure 7 This is a block diagram of an example video encoder.

[0019] Figure 8 This is a block diagram of an example video decoder.

[0020] Figure 9 This is an illustration of examples of parts of an image or video frame.

[0021] Figure 10-11 This is an illustration of an example of a sparse matrix representation of an image or video patch using boundaries with non-zero coefficients.

[0022] Figure 12 This is a flowchart illustrating an example of a technique for encoding image or video blocks using a sparse matrix representation that employs boundaries with non-zero coefficients.

[0023] Figure 13 This is a flowchart illustrating an example of a technique for decoding encoded image or video blocks using a sparse matrix representation that employs boundaries with non-zero coefficients. Detailed Implementation

[0024] Image and video compression schemes involve dividing corresponding image or video frames into smaller parts, such as blocks, and generating an output bitstream using techniques to limit the information included in each block in the output. The encoded bitstream can be decoded to reconstruct the source image or video frame from the limited information. Typically, coefficients arranged in a two-dimensional matrix format are used to represent the image or video data to be encoded into or subsequently decoded from the encoded bitstream. In at least some cases, the two-dimensional matrix of coefficients may include some zero-value coefficients. In many such cases, the zero-value coefficients can be found along the top and / or bottom of the two-dimensional matrix.

[0025] Zero-valued coefficients typically do not indicate important information and therefore may not be signaled within the encoded bitstream, for example. Thus, compression efficiency can be improved by skipping at least some zero-valued coefficients. However, a specific arrangement of zero-valued coefficients may not be efficient for transmission within a two-dimensional matrix structure. For example, if the zero-valued coefficients are not grouped together, they can be encoded into the encoded bitstream and then decoded from it, for example, because otherwise isolating only non-zero-valued coefficients would be difficult or resource-intensive.

[0026] In this regard, conventional schemes for improving transmission efficiency include using scan order patterns such as raster order or zigzag order to convert the two-dimensional matrix into a one-dimensional signal. However, at least in some cases, scan order patterns may still not be efficient enough in grouping zero-value coefficients, thus potentially resulting in many zero-value coefficients being encoded. Furthermore, using scan order patterns alone to identify the position of the last non-zero coefficient based on the scan order pattern may require signaling additional data in the encoded bitstream, such as an end-of-block (EOB) message indicating the position of the last non-zero coefficient.

[0027] Embodiments of the present invention address problems such as those associated with sparse matrix representations of image or video data, where such representations utilize boundaries with non-zero coefficients. Two-dimensional matrix representations of image or video frame data, such as data corresponding to blocks of an image or video frame, are processed by identifying bounding boxes with non-zero coefficients of the blocks. These bounding boxes may have a rectangular shape. Alternatively, they may have a non-rectangular shape. For example, the bounding box may have a generally convex or concave shape on one or more sides. In another example, the bounding box may have a shape corresponding to a triangle, circle, rhombus, or other geometry. Coefficients located within the bounding box, along with information that can be used by the decoder to identify the shape and / or position of the bounding box within the block during decoding, are encoded into a coded bitstream. Thus, coefficients located outside the bounding box are not encoded into the coded bitstream, thereby improving compression and computational efficiency.

[0028] This paper first describes in more detail the techniques for encoding images or videos using sparse matrix representations, which employ boundaries with non-zero coefficients, by referring to systems in which such techniques can be implemented. Figure 1 This is a diagram illustrating an example of an image or video coding system 100. The image or video coding system 100 includes a transmitting station 102, a receiving station 104, and a network 106. The image or video coding system 100 can be used, for example, to encode and decode some or all of an image or video sequence.

[0029] Transmitting station 102 is a computing device for encoding and transmitting images. Alternatively, transmitting station 102 may include two or more distributed devices for encoding and transmitting image or video sequences. Receiving station 104 is a computing device for receiving and decoding encoded images or encoded video. Alternatively, receiving station 104 may include two or more distributed devices for receiving and decoding encoded images or encoded video. (Refer to below) Figure 2 An example of a computing device for implementing one or both of transmitting station 102 or receiving station 104 is described.

[0030] Network 106 connects transmitting station 102 and receiving station 104 for encoding, transmitting, receiving, and decoding images. Network 106 can be, for example, the Internet. Network 106 can also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or other means for transmitting images from transmitting station 102 to receiving station 104.

[0031] The implementation of the encoding system 100 can be related to... Figure 1 The embodiments shown and described differ. In some embodiments, the network 106 may be omitted from the image or video encoding system 100. In some embodiments, the image or video stream may be encoded and then stored for later transmission to the receiving station 104 or other device with memory. In some embodiments, the receiving station 104 may receive (e.g., via the network 106, a computer bus, and / or some communication path) the encoded image or encoded video and store the encoded image or encoded video for later decoding. For example, Real-time Transport Protocol (RTP), a video streaming protocol based on Hypertext Transfer Protocol (HTTP), or other protocols may be used to transmit the encoded image or encoded video over the network 104.

[0032] In some implementations, the image or video encoding system 100 may be used in a video conferencing system. For example, sending station 102 and / or receiving station 106 may include the ability to encode and decode video streams as described below. For example, receiving station 106 may be a participant's video conferencing device that receives encoded video bitstreams from a video conferencing server (e.g., sending station 102) for decoding and viewing, and also encodes and transmits video bitstreams to the video conferencing server for other participants to decode and view.

[0033] In some implementations, the functions of the sending station 102 and the receiving station 104 may vary based on the specific operations performed. For example, during the encoding of an image or video stream, the sending station 102 may be a computing device for uploading the image or video stream to a server for encoding, and the receiving station 104 may be a server that receives the image or video stream from the sending station 102 and encodes the image or video stream for later use. In another example, during the decoding of an encoded image or video, the sending station 102 may be a server that decodes the encoded image or video, and the receiving station 104 may be a computing device that receives the decoded image or video from the sending station 102 and renders the decoded image or video.

[0034] Figure 2 It can be used in image encoding and decoding systems (e.g., Figure 1 The image encoding system 100 shown is illustrated in the block diagram of an example of the internal configuration of the computing device 200. The computing device 200 may, for example, implement one or both of a transmitting station 102 or a receiving station 104. The computing device 200 may be in the form of a computing system comprising multiple computing devices, or in the form of a single computing device, such as a mobile phone, tablet computer, laptop computer, notebook computer, or desktop computer, etc.

[0035] The processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be another type of device, existing or developed in the future, capable of manipulating or processing information. For example, although the disclosed embodiments can be practiced with a single processor (e.g., processor 202) shown, speed and efficiency advantages can be achieved by using more than one processor.

[0036] In one embodiment, the memory 204 in the computing device 200 may be a read-only memory (ROM) device or a random access memory (RAM) device. However, other suitable types of storage devices may be used as memory 204. Memory 204 may include code and data 206 accessed by the processor 202 using bus 212. Memory 204 may further include an operating system 208 and an application program 210, which includes at least one program that allows the processor 202 to perform the techniques described herein. For example, application program 210 may include applications 1 to N, which also include image or video encoding and / or decoding software for performing some or all of the techniques described herein. The computing device 200 may also include an auxiliary storage device 214, which may be, for example, a memory card for moving the computing device. For example, images may be stored wholly or partially in the auxiliary storage device 214 and loaded into memory 204 as needed for processing.

[0037] The computing device 200 may also include one or more output devices, such as a display 218. In one example, the display 218 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 218 may be coupled to the processor 202 via a bus 212. In addition to or as an alternative to the display 218, other output devices may be provided that allow the user to program or otherwise use the computing device 200. When the output device is a display or includes a display, the display may be implemented in various ways, including as a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light-emitting diode (LED) display (e.g., an organic LED (OLED) display).

[0038] The computing device 200 may also include or communicate with an image sensing device 220, such as an existing or future camera or other image sensing device, which can sense images such as images of a user operating the computing device 200. The image sensing device 220 may be positioned such that it faces the user operating the computing device 200. For example, the position and optical axis of the image sensing device 220 may be configured such that the field of view includes an area directly adjacent to the display 218, from which the display 218 can be seen.

[0039] The computing device 200 may also include or communicate with a sound sensing device 222, such as an existing or future microphone or other sound sensing device, which can sense sounds in the vicinity of the computing device 200. The sound sensing device 222 may be positioned to point towards a user operating the computing device 200 and may be configured to receive sounds emitted by the user, such as speech or other talking, when the user operates the computing device 200.

[0040] The implementation of computing device 200 may differ from that regarding Figure 2 The embodiments shown and described are as follows. In some embodiments, the operation of processor 202 may be distributed across multiple machines (each machine may have one or more processors), which may be directly coupled or coupled across a local area network or other network. In some embodiments, memory 204 may be distributed across multiple machines, such as network-based memory or memory in multiple machines performing the operation of computing device 200. In some embodiments, bus 212 of computing device 200 may consist of multiple buses. In some embodiments, auxiliary memory 214 may be directly coupled to other components of computing device 200 or may be accessed via a network, and may include an integrated unit such as a memory card, or multiple units such as multiple memory cards.

[0041] Figure 3 This is an example diagram of an image 300 that will be encoded and subsequently decoded. Image 300 may be divided into a series of planes or segments 302. Segments 302 may be, for example, subsets of images that allow for parallel processing. Segments 302 may also be subsets of images, or alternatively, subsets of images whose color data can be separated into individual colors. For example, image 300 of color video data may include a luminance plane and two chrominance planes. The segments 302 may be sampled at different resolutions.

[0042] Regardless of whether frame 300 is divided into segments 302, image 300 may be further subdivided into blocks 304, which may contain data corresponding to, for example, 16x16 pixels in image 300. Block 304 may also be arranged to include data from one or more segments 302 of pixel data. Block 304 may also have any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise stated, the terms "block" and "macroblock" are used interchangeably herein.

[0043] Figure 4 This is a diagram of an example of a video stream 400 that will be encoded and subsequently decoded. Video stream 400 includes video sequence 402. At the next level, video sequence 402 includes multiple adjacent frames 404. While three frames are depicted as adjacent frames 404, video sequence 402 can include any number of adjacent frames 404. The adjacent frames 404 can then be further subdivided into individual frames, for example, frame 406.

[0044] At the next level, frame 406 can be divided into a series of planes or segments 408. Segments 408 can be, for example, subsets of frames that allow for parallel processing. Segments 408 can also be subsets of frames that can separate video data into separate color planes. For example, frame 406 of color video data can include a luma plane and two chroma planes. The segments 408 can be sampled at different resolutions.

[0045] Regardless of whether frame 406 is divided into segments 408, frame 406 can be further subdivided into blocks 410, which may contain data corresponding to, for example, 16x16 pixels in frame 406. Block 410 may also be arranged to include data from one or more segments 408 of the pixel data. Block 410 may also have any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise stated, the terms "block" and "macroblock" are used interchangeably herein.

[0046] Figure 5This is a block diagram of an example image encoder 500. The image encoder 500 can be, for example, in a... Figure 1 The image encoder 500, as shown in the image encoding system 100, implements an image decoder at the transmitting station 102 of the image encoding system. The image encoder 500 receives and encodes the input image 502 (e.g., Figure 3 Image 300 shown is used to generate an encoded image 504, which can be output to a decoder (e.g., by a decoder such as...) Figure 1 (The receiving station 104 shown is implemented by a receiving station) or used for storage.

[0047] Image encoder 500 includes a transform stage 506, a quantization stage 508, and an entropy coding stage 510. Transform stage 506 transforms blocks of input image 502 to the frequency domain. For example, transform stage 506 can use a discrete cosine transform (DCT) to transform blocks of input image 502 from the spatial domain to the frequency domain. Alternatively, transform stage 506 can use another Fourier correlation transform or discrete Fourier transform to transform blocks of input image 502 from the spatial domain to the frequency domain. As another alternative, transform stage 506 can use other block-based transforms to transform blocks of input image 502 from the spatial domain to the frequency domain.

[0048] Quantization stage 508 quantizes the transform coefficients produced as the output of transform stage 506. Quantization stage 508 uses a quantization factor to convert these transform coefficients into discrete quantum values, called quantized transform coefficients. For example, the transform coefficients can be divided by the quantization factor and truncated.

[0049] Entropy coding level 510 uses lossy or lossless coding techniques to entropy code the quantized transform coefficients output from quantization level 508. For example, the lossless coding technique used by entropy coding level 510 to entropy code the quantized transform coefficients can be or includes Huffman coding, arithmetic operations, variable-length coding, or other coding techniques. The encoded image 504 is generated based on the output of entropy coding level 510. The encoded image 504 can be stored in a server (e.g., a database or similar data storage) for later retrieval and decoding. For example, the encoded image 504 can be an image hosted on a website or provided for display on a webpage.

[0050] Figure 6 This is a block diagram of an example image decoder 600. The image decoder 600 can be, for example, in... Figure 1The image encoding system 100 shown has an image decoder implemented at the receiving station 104. The image decoder 600 (e.g., from a storage device or memory) receives and decodes the encoded image 602 to produce an output image 604, which can be output for display or storage. The output image 604 is perceptually equivalent to the input image encoded using the encoder (e.g., ...). Figure 3 The input image 502 and image encoder 500 shown are the same or similar. However, if the encoding results in the encoded image 602 being lossy, the output image 604 may look substantially the same as the input image, but does not have to be exactly the same as the input image.

[0051] The image decoder 600 includes an entropy decoding stage 606, an inverse quantization stage 608, an inverse transform stage 610, and a filtering stage 612. The entropy decoding stage 606 uses lossless coding techniques to perform entropy decoding on the encoded image data from the encoded image 602. For example, the lossless coding techniques used by the entropy decoding stage 606 to perform entropy decoding on the encoded image data from the encoded image 602 can be or include Huffman coding, arithmetic operations, variable-length coding, or other coding techniques.

[0052] Entropy decoding stage 606 performs entropy decoding on the encoded image data to generate quantization transform coefficients. Dequantization stage 608 dequantizes the quantization transform coefficients output from entropy decoding stage 606, for example, by multiplying the quantization transform coefficients by a quantization factor used to generate the encoded image 602. Inverse transform stage 610 performs an inverse transform on the dequantized transform coefficients, for example, by inverse transforming the dequantized transform coefficients from the frequency domain to the spatial domain.

[0053] Filtering stage 612 performs filtering to remove artifacts caused by the encoding of the encoded image 602. For example, filtering stage 612 can filter the coefficients output from inverse transform stage 610 of the block-by-block filtering of the encoded image 602 according to the main filtering direction of the block.

[0054] Figure 7 This is a block diagram of an example video encoder 700. As described above, encoder 700 can be implemented in transmitting station 102, for example, by providing a computer software program stored in memory (e.g., memory 204). The computer software program may include machine instructions that, when executed by a processor such as processor 202, cause transmitting station 102 to... Figure 7 The video data is encoded in the manner described herein. The encoder 700 can also be implemented as dedicated hardware included, for example, in the transmitting station 102. In some embodiments, the encoder 700 may be a hardware encoder.

[0055] The encoder 700 has the following stages to perform various functions in the forward path (shown by solid connecting lines) to use the input video stream 704 (e.g., Figure 4 The video stream 400 shown is used as input to generate an encoded or compressed bitstream 702: an intra / inter-frame prediction stage 706, a transform stage 708, a quantization stage 710, and an entropy coding stage 712. The encoder 700 may also include reconstruction paths (shown by dashed connecting lines) to reconstruct frames for encoding future blocks. Figure 7 In this encoder 700, the following stages are used to perform various functions in the reconstruction path: inverse quantization stage 714, inverse transform stage 716, reconstruction stage 718, and loop filtering stage 720. Other architectural variations of the encoder 700 can be used to encode the video stream 300.

[0056] When the input video stream 704 is presented for encoding, adjacent frames can be processed in blocks. At the intra / inter-frame prediction level 706, individual blocks can be encoded using either intra-frame prediction (also known as intra-prediction) or inter-frame prediction (also known as inter-prediction). In either case, prediction blocks can be formed. In the case of intra-frame prediction, prediction blocks can be formed from samples that have been previously encoded and reconstructed in the current frame. In the case of inter-frame prediction, prediction blocks can be formed from samples in one or more previously constructed reference frames.

[0057] Next, the prediction block can be subtracted from the current block at the intra / inter-frame prediction stage 706 to produce a residual block (also called a residual). The transform stage 708 transforms the residual into transform coefficients in, for example, the frequency domain using a block-based transform. The quantization stage 710 converts the transform coefficients into discrete quantum values, referred to as quantized transform coefficients, using a quantization value or quantization level. For example, the transform coefficients can be divided by the quantization value and truncated.

[0058] The quantization transform coefficients are then entropy encoded by entropy coding level 712. The entropy-encoded coefficients, along with other information for decoding the block (which may include, for example, syntax elements such as those indicating the prediction type, transform type, motion vector, or quantization value used), are then output to compressed bitstream 702. Compressed bitstream 702 can be formatted using various techniques such as variable-length coding (VLC) or arithmetic coding. Compressed bitstream 702 may also be referred to as encoded video stream or encoded video bitstream, and these terms will be used interchangeably herein.

[0059] The reconstruction path (shown by the dashed connecting line) can be used to ensure the encoder 700 and decoder (e.g., see below) Figure 8 The decoder 800 described uses the same reference frame to decode the compressed bitstream 420. The reconstruction path performs the functions that occur during the decoding process (as described below). Figure 8 Similar functionality (as described) includes dequantizing the quantized transform coefficients in dequantization stage 714 and inverse transforming the dequantized transform coefficients in inverse transform stage 716 to produce a derivative residual block (also known as derivative residual).

[0060] At reconstruction stage 718, the predicted block from intra / inter-frame prediction stage 706 can be added to the derivative residual to create a reconstruction block. Loop filtering stage 720 can be applied to the reconstruction block to reduce distortion such as block artifacts. In some implementations, loop filtering stage 720 can be replaced by other filtering stages.

[0061] Other variations of encoder 700 can be used to encode the compressed bitstream 702. In some embodiments, for certain blocks or frames, a non-transform-based encoder can directly quantize the residual signal without transform stage 708. In some embodiments, the encoder may have a quantization stage 710 and an inverse quantization stage 714 combined in a common stage.

[0062] Figure 8 This is a block diagram of an example video decoder 800. For example, the decoder 800 can be implemented in the receiving station 106 by providing a computer software program stored in memory 204. The computer software program may include machine instructions that, when executed by a processor such as processor 202, cause the receiving station 106 to... Figure 8 The decoder 800 decodes video data in the manner described herein. The decoder 800 can also be implemented in hardware, such as in a transmitting station 102 or a receiving station 106.

[0063] Similar to Figure 7 The reconstruction path of encoder 700 shown, decoder 800 may include the following stages in one example to perform various functions to extract data from compressed bitstream 804 (e.g., ...). Figure 7 The compressed bitstream 702 shown generates an output video stream 802 consisting of an entropy decoding stage 806, an inverse quantization stage 808, an inverse transform stage 810, an intra / inter-frame prediction stage 812, a reconstruction stage 814, a loop filtering stage 816, and a deblocking filtering stage 818. Other structural variations of the decoder 800 can be used to decode the compressed bitstream 804.

[0064] When the compressed bitstream 804 is presented for decoding, the data elements within the compressed bitstream 804 can be decoded by the entropy decoding stage 806 to produce a set of quantization transform coefficients. The dequantization stage 808 dequantizes the quantization transform coefficients (e.g., by multiplying the quantization transform coefficients by the quantization value), and the inverse transform stage 810 inverse transforms the dequantized transform coefficients to produce a derivative residual that is identical to the derivative residual created by the inverse transform stage 716 in the encoder 700. Using the header information decoded from the compressed bitstream 804, the decoder 800 can use the intra / inter-frame prediction stage 812 to create a prediction block identical to the prediction block created in the encoder 700 (e.g., at the intra / inter-frame prediction stage 706).

[0065] In reconstruction stage 814, the predicted block can be added to the derivative residual to create a reconstruction block. Loop filtering stage 816 can be applied to the reconstruction block to reduce block artifacts. In some embodiments, loop filtering stage 816 can be replaced by other filtering stages. Other filters can also be applied to the reconstruction block. For example, deblocking filtering stage 818 can be applied to the reconstruction block to reduce block distortion, and the result is output as output video stream 802. Output video stream 802 can also be referred to as decoded video stream, and the terms will be used interchangeably herein.

[0066] Other variations of decoder 800 can be used to decode the compressed bitstream 804. In some embodiments, decoder 800 can produce an output video stream 802 without the need for a deblocking filter stage 818.

[0067] Figure 9 This is an illustration of examples of portions of an image or video frame 900. As shown, the image or video frame 900 comprises four 64×64 blocks 910 arranged in two rows and two columns in a matrix or Cartesian plane. In some embodiments, the 64×64 block may be a maximum coding unit, N=64. Each 64×64 block may include four 32×32 blocks 920. Each 32×32 block may include four 16×16 blocks 930. Each 16×16 block may include four 8×8 blocks 940. Each 8×8 block 940 may include four 4×4 blocks 950. Each 4×4 block 950 may include 16 pixels, which may be represented in four rows and four columns in each corresponding block of the Cartesian plane or matrix.

[0068] The pixels may include information representing the image captured in image or video frame 900, such as luminance information, color information, and position information. In some embodiments, a block such as the 16×16 pixel block shown may include: a luminance block 960, which may include luminance pixels 562; and two chrominance blocks 970, 980, such as a U or Cb chrominance block 970 and a V or Cr chrominance block 980. Chroma blocks 970, 980 may include chrominance pixels 990. For example, luminance block 960 may include 16×16 luminance pixels 962, and each chrominance block 970, 980 may include 8×8 chrominance pixels 990, as shown. Although one arrangement of the boxes is shown, any arrangement can be used. Figure 9 An N×N block is shown, but in some implementations, an N×M block can be used, where N and M are different numbers. For example, 32×64 blocks, 64×32 blocks, 16×32 blocks, 32×16 blocks, or any other size of block can be used. In some implementations, N×2N blocks, 2N×N blocks, or combinations thereof can be used.

[0069] In some implementations, encoding the image or video frame 900 may include ordered block-level encoding. Ordered block-level encoding may include encoding blocks of the image or video frame 900 in an order such as raster scan order, wherein each block may be identified and processed sequentially, starting with the top-left block of the image or video frame 900 or a portion of the image or video frame 900, and proceeding along rows from left to right and from top to bottom. For example, a 64×64 block in the top row and left column of the image or video frame 900 may be the first block to be encoded, and a 64×64 block immediately to the right of the first block may be the second block to be encoded. The second row, starting from the top, may be the second row to be encoded, such that a 64×64 block in the left column of the second row may be encoded after a 64×64 block in the rightmost column of the first row.

[0070] In some implementations, encoding blocks of image or video frame 900 may include using quadtree coding, which may involve encoding smaller block units within a block in raster scan order. For example, quadtree coding may be used to encode the 64×64 block shown in the lower left corner of this portion of image or video frame 900, wherein the top-left 32×32 block may be encoded, then the top-right 32×32 block, then the bottom-left 32×32 block, and then the bottom-right 32×32 block. Quadtree coding may be used to encode each 32×32 block, wherein the top-left 16×16 block may be encoded, then the top-right 16×16 block, then the bottom-left 16×16 block, and then the bottom-right 16×16 block.

[0071] Quadtree coding can be used to encode each 16×16 block, where the top-left 8×8 block can be encoded, then the top-right 8×8 block, then the bottom-left 8×8 block, and then the bottom-right 8×8 block. Alternatively, quadtree coding can be used to encode each 8×8 block, where the top-left 4×4 block can be encoded, then the top-right 4×4 block, then the bottom-left 4×4 block, and then the bottom-right 4×4 block. In some implementations, the 8×8 blocks can be omitted from the 16×16 blocks, and quadtree coding can be used to encode the 16×16 blocks, where the top-left 4×4 block can be encoded, and the other 4×4 blocks in the 16×16 blocks can be encoded according to the raster scan order.

[0072] In some implementations, encoding the image or video frame 900 may include, for example, encoding information included in the original version of the image or video frame by omitting some information from the original version of the image or video frame from the corresponding encoded image or video frame. For example, the encoding may include reducing spectral redundancy, reducing spatial redundancy, or a combination thereof.

[0073] Reducing spectral redundancy may include using a color model based on a luminance component (Y) and two chrominance components (U and V or Cb and Cr), which may be referred to as the YUV or YCbCr color model or color space. Using the YUV color model may involve using a relatively large amount of information to represent the luminance component of a portion of an image or video frame 900, and using a relatively small amount of information to represent each corresponding chrominance component of that portion of the image or video frame 900. For example, a portion of an image or video frame 900 may be represented by a high-resolution luminance component and two lower-resolution chrominance components, the luminance component comprising 16×16 pixel blocks, and each chrominance component representing that portion of the image as an 8×8 pixel block. Pixels may indicate a value, for example, a value in the range of 0 to 255, and may be stored or transmitted using, for example, eight bits. Although this disclosure is described with reference to the YUV color model, other color models may be used.

[0074] Reducing spatial redundancy can include transforming the blocks to the frequency domain, for example, using a discrete cosine transform. For example, encoder units, such as... Figure 5 Transformation stage 506 shown Figure 7 The transform stage 708 shown can perform discrete cosine transform using transform coefficient values ​​based on spatial frequency.

[0075] Although the image or video frame 900 is described herein with reference to a matrix or Cartesian representation for clarity, the image or video frame 900 may be stored, transmitted, processed, or a combination thereof in a data structure such that pixel values ​​can be effectively represented for the image or video frame 900. For example, the image or video frame 900 may be stored, transmitted, processed, or any combination thereof in a two-dimensional data structure (such as a matrix as shown) or a one-dimensional data structure (such as a vector array).

[0076] Furthermore, although this document describes an image where U and V are chroma-subsampled at half the resolution of Y, the color channels of image or video frame 900 can have different configurations. For example, still referring to the YUV color space, full resolution can be used for all color channels of image or video frame 900. In another example, a color space other than the YUV color space can be used to represent the resolution of the color channels of image or video frame 900.

[0077] Now refer to Figure 10-11 This describes an example of the boundaries used for the non-zero coefficients in a sparse matrix representation of an image or video block. A sparse matrix representation using boundaries of non-zero coefficients refers to image or video encoding using bounding boxes that indicate or represent non-zero value coefficients within a block of an image or video frame. At the encoder, the non-zero value coefficients enclosed by the bounding boxes are encoded in the encoded bitstream. The remaining coefficients are zero value coefficients outside the bounding boxes and are not specifically encoded in the encoded bitstream. For example, the remaining coefficients can be represented in the encoded bitstream by zero-value runs. In another example, the remaining coefficients can be completely omitted from the encoded bitstream. Dimensional information indicating the shape, size, and / or position of the coefficients relative to the subject block is also encoded in the encoded bitstream. For example, informational size information can be encoded as the block header of the subject block. Then, at the decoder, the encoded non-zero value coefficients are decoded from the encoded bitstream. The decoder uses the size information to determine the coefficients on which computational resources should be spent, for example, because those coefficients are enclosed by the bounding boxes and therefore include non-zero value coefficients. The decoder determines that the remaining coefficients of the block are zero-value coefficients. The decoder can determine that the remaining coefficients are zero-value coefficients, either independently of or based on the data encoded within the coded bitstream.

[0078] First refer to Figure 10 Block 1000 is shown. Block 1000 can be an image (e.g., Figure 3 The image 300 shown) or video frame (e.g., Figure 4The block 1000 (shown in video frame 406) comprises coefficients 1002 enclosed by a bounding box indicated by boundary 1004 and coefficients 1006 outside boundary 1004 and therefore outside the bounding box. Thus, block 1000 can be considered a two-dimensional matrix of coefficients. Coefficients 1002 include non-zero coefficients and, in some cases, zero coefficients. Coefficients 1006 are restricted to zero coefficients. Symbol 1008 represents some coefficients 1006 outside the bounding box.

[0079] Encoding block 1000 includes encoding and subsequently decoding the coefficients 1002, as they are surrounded by the bounding box. However, since coefficient 1006 is outside the bounding box, it is not encoded or subsequently decoded. For example, alternatively, zero-value runs can be encoded into an encoded bitstream and then decoded from the encoded bitstream to indicate the number of coefficients 1006 in block 1000.

[0080] The size information of the bounding box is included in the encoded bitstream. For example, based on a portion of a two-dimensional matrix corresponding to block 1000 on which the bounding box is located, the size information may be, include, or refer to information indicating the shape of the bounding box. In another example, the size information may be, include, or refer to information indicating one or more dimensions of the bounding box. For example, information indicating one or more dimensions may indicate one or more of the width or height of the bounding box. In some cases, such as when the bounding box is non-rectangular, information indicating one or more dimensions may indicate multiple widths and / or multiple heights to indicate, for example, a dimensional variation in the overall shape of the bounding box. In yet another example, the size information may be, include, or refer to information indicating the position of the bounding box within a block. For example, information indicating the position of the bounding box may by default indicate that the bounding box begins at the top-left corner of the two-dimensional matrix representation corresponding to the block. In another example, information indicating the position of the bounding box may indicate that the bounding box begins and / or ends at other locations within the block. In some implementations, the size information of the bounding box may be, include, or represent information indicating a combination of the shape of the bounding box, one or more dimensions of the bounding box, and / or the position of the bounding box within the block.

[0081] Size information may correspond to, for example, coordinates along the X and Y axes within a two-dimensional matrix representation of the block. In some embodiments, size information included in the encoded bitstream indicates the full size, shape, and / or location of the boundary block. In some embodiments, size information included in the encoded bitstream indicates a portion of the size, shape, and / or location of the boundary block.

[0082] In some implementations, encoded syntax element values ​​can be used to represent size information included within the encoded bitstream. For example, a set of encoded syntax element values ​​can be defined for a shape (e.g., rectangle, rhombus, circle, triangle, or other geometry), in which case the encoding of this set of encoded syntax element values ​​indicates to the decoder the bounding box of the defined shape. In another example, a set of syntax element values ​​can be defined for height and / or width, in which case the encoding of this set of encoded syntax element values ​​indicates to the decoder the height and / or width of the bounding box. In yet another example, a set of encoded syntax can be defined for the start and / or end positions within a block, in which case the encoding of this set of encoded syntax element values ​​indicates to the decoder the start and / or end positions of the bounding box within the block. In some implementations, syntax elements such as binary flags can be used to signal whether a bounding box is used. In some such implementations, the decoder can check the value of this flag before proceeding to process the corresponding encoded block using the bounding box.

[0083] In some implementations, the size information for the bounding box can be omitted from the encoded bitstream. In some such implementations, the decoder used to subsequently decode the encoded block corresponding to the bounding box can rely on a default configuration or determine or identify the bounding box relative to the coefficients of the encoded block. For example, the default configuration might indicate that the height of the bounding box is equal to the height of the encoded block. In another example, the default configuration might indicate that the width of the bounding box is equal to the width of the encoded block. In yet another example, the default configuration might indicate that the height and / or width of the bounding box is identified based on the XY position of the last non-zero coefficient within the encoded block.

[0084] In some implementations, adaptive bits may be included in the bitstream for some or all of the coefficients 1002 and / or for some or all of the coefficients 1006 to indicate whether such coefficients are enclosed within the bounding box associated with boundary 1004. For example, the adaptive bits may be binary random variables with probabilities that are updated each time a zero-value coefficient and / or a non-zero-value coefficient is encountered. In some implementations, adaptive bits may be used to indicate the run length of zero values ​​for coefficients 1006. For example, if there are fewer than five consecutive zero-value coefficients 1006, adaptive bits may be used to indicate those coefficients 1006. However, if there are 5 to 16 consecutive zero-value coefficients 1006, the number of zeros can be encoded using syntax elements in the image or video frame header corresponding to block 1000. Furthermore, if there are more than 16 consecutive zero-value coefficients 1006, entropy encoding of those zero-value coefficients 1006 can be used, for example, in Golomb notation.

[0085] Next reference Figure 11Block 1000 is again shown using symbols 1100 and 1102. Symbol 1100 represents the pattern along the scan sequence (e.g., as shown in the image). Figure 10-11 The position of the zigzag pattern shown is indicated by the symbol 1100, which has crossed both sides of boundary 1004. Symbol 1100 indicates the EOB position of the coefficients enclosed within the bounding box corresponding to boundary 1004. Symbol 1102 indicates the position of the pattern along the scan sequence after which all coefficients are non-zero coefficients. Symbols 1100 and 1102, along with their positions, can be included in the encoded bitstream to which block 1000 is encoded.

[0086] In some implementations, the bounding boxes used in conjunction with the encoding of block 1000 can enclose information other than coefficients. For example, bounding boxes can be used during encoding to enclose other digital information representing image or video data. In some such implementations, where the numerical information enclosed by the bounding boxes is generated before using scan-order processing, additional iterations can be performed to identify the bounding boxes. For example, these additional iterations can perform the same or similar operations as scan-order processing, such as those described below relative to... Figure 12 As described.

[0087] Now refer to Figure 12-13 Describes a technique for sparse matrix representation using boundaries with non-zero coefficients. Figure 12 This is a flowchart illustrating an example of a technique 1200 for encoding image or video blocks using sparse matrix representations, wherein the sparse matrix representations use boundaries with non-zero coefficients. Figure 13 This is a flowchart illustrating an example of a technique 1300 for decoding encoded image or video blocks using sparse matrix representations, wherein the sparse matrix representation uses boundaries with non-zero coefficients.

[0088] One or more of techniques 1200 or 1300 may be implemented, for example, as a software program executable by a computing device such as transmitting station 102 or receiving station 104. For example, the software program may include machine-readable instructions that may be stored in a memory such as memory 204 or auxiliary memory 214, and when executed by a processor such as processor 202, may cause the computing device to perform one or more of techniques 1200 or 1300. One or more of techniques 1200 or 1300 may be implemented using dedicated hardware or firmware. As described above, some computing devices may have multiple memories or processors, and multiple processors, memories, or both may be used to distribute the operations described in one or more of techniques 1200 or 1300.

[0089] For the sake of simplicity, techniques 1200 and 1300 are depicted and described as a series of steps or operations, respectively. However, the steps or operations according to this disclosure may occur in various orders and / or simultaneously. Additionally, other steps or operations not presented and described herein may be used. Furthermore, implementing the techniques according to the disclosed subject matter may not require all the steps or operations shown.

[0090] First refer to Figure 12 The diagram illustrates a flowchart of an example of technique 1200 for encoding a current block (e.g., an image block of an image being encoded or a video block of a video frame being encoded) using a sparse matrix representation, wherein the sparse matrix representation uses boundaries with non-zero coefficients. At 1202, pixel values ​​are transformed to produce coefficients. Pixel values ​​can be pixel values ​​before or after performing prediction. Thus, in some cases, pixel values ​​can be or refer to values ​​of the prediction residual.

[0091] At 1204, the coefficients are quantized to produce quantized transform coefficients. The quantized transform coefficients are arranged in a two-dimensional matrix representation. For example, the two-dimensional matrix may include or be expressed as a transform block corresponding to the current block. In some embodiments, the two-dimensional matrix may include or be expressed as multiple transform blocks.

[0092] At 1206, a bounding box is identified to enclose the non-zero coefficients of the quantization transform coefficients. Identifying the bounding box may include locating the non-zero coefficients along the scan sequence. Many scan sequences can be used to scan the quantization transform coefficients. For example, the scan sequence may start from the top of the transform block corresponding to the current block (e.g., raster or horizontal scan) and scan the quantization transform coefficients row by row, or start from the left side of such a transform block (e.g., vertical scan) and scan the coefficients column by column. In another example, the scan sequence may process the coefficients in directions that are not entirely horizontal or vertical (e.g., diagonal scan, zigzag scan, etc.).

[0093] While typical scan-order processing converts a two-dimensional matrix of quantized transform coefficients into a one-dimensional sequence by traversing the coefficients in a specific order set by the pattern of the scan order used, the scan-order processing for identifying bounding boxes includes traversing the quantized transform coefficients to identify non-zero coefficients in the current block. For example, the location of the non-zero coefficients can be identified within the current block. In some implementations, a block end marker can also be identified to indicate the location of the last non-zero coefficient along the scan-order pattern.

[0094] The bounding box is identified based on the non-zero coefficients. Thus, the size information of the bounding box, representing the shape, size, and / or position of the bounding box in the two-dimensional matrix representation corresponding to the current block, is determined based on the position of the non-zero coefficients in the current block. For example, the position of the non-zero coefficients identified in the current block along the scan sequence indicates the boundary of the bounding box, where each non-zero coefficient is enclosed within the bounding box. In some cases, the bounding box will only enclose the non-zero coefficients. In other cases, the bounding box will enclose both the non-zero coefficients and one or more zero coefficients.

[0095] Consider the size of the bounding box. Smaller bounding boxes generally produce better compression performance compared to larger bounding boxes, for example, because smaller bounding boxes have fewer bounding coefficients in total. Thus, the boundaries of the bounding box are set to achieve this exclusion, in order to exclude zero-value coefficients from the bounding box, such that the resulting bounding box is smaller than it would be if it contained those zero-value coefficients.

[0096] At 1208, the non-zero coefficients enclosed within the bounding box are entropy-encoded into a coded bitstream. Entropy-encoding the non-zero coefficients into the coded bitstream involves processing the non-zero coefficients enclosed within the bounding box according to their order in the scan order mode. For example, after performing the operation to identify the bounding box, the two-dimensional matrix of quantized transform coefficients of the current block is converted into a one-dimensional sequence using the scan order. In another example, during the operation to identify the bounding box, the scan order can be used to convert the two-dimensional matrix of quantized transform coefficients of the current block into a one-dimensional sequence.

[0097] As part of entropy coding, zero-value coefficients enclosed within a bounding box are also encoded into the bitstream. However, zero-value coefficients not enclosed within the bitstream are not specifically entropy-coded into the bitstream. In some implementations, zero-value coefficients not enclosed within the bitstream are completely skipped, such that the information encoded into the bitstream is limited to the coefficients enclosed within the bounding box and information that the decoder can use to identify the bounding box (e.g., size information as described below). In some implementations, adaptive bits, syntax elements, or Golomb representations, such as the run size based on the zero-value coefficients, can be used to indicate the run size of the zero-value coefficients within the bitstream. For example, the decoder can use data indicating the run size of the zero-value coefficients to reconstruct the current block by padding zero-value coefficients at specific locations.

[0098] At 1210, the size information of the bounding box is encoded within the encoded bitstream. Encoding the size information within the encoded bitstream may include, for example, encoding the size information as a block header for the current block, an image or video frame header including the current block, or other portions of the bitstream. Encoding the size information within the bitstream instructs a decoder that subsequently receives the bitstream how to identify the bounding box used for decoding the current block.

[0099] In contrast to technique 1200, the operation for encoding non-zero coefficients enclosed within a bounding box into a bitstream and the operation for encoding size information within the bitstream are described as separate operations. However, in some implementations, the operation for encoding non-zero coefficients enclosed within a bounding box into a bitstream and the operation for encoding size information within the bitstream can be combined. For example, a single encoding operation can be performed to encode both non-zero coefficients and size information into a bitstream.

[0100] In some implementations, technique 1200 may include encoding syntax elements that indicate whether a bounding box is used to encode the block into a bitstream. For example, the syntax element may be a binary flag. The binary flag may be included in a block header for the encoded block, an image or video frame header for an image or video frame that includes the encoded block, or other portions of the bitstream.

[0101] In some implementations, the quantization transform coefficients of the current block can be processed for each of a plurality of candidate scan sequences to identify the scan sequence used to identify the bounding box. For example, different candidate scan sequences may result in different shapes, sizes, and / or positions of the bounding box within the current block. Consequently, different candidate scan sequences may result in different total numbers of coefficients (non-zero and zero values) enclosed by the resulting bounding box. Therefore, the scan sequence used to identify the bounding box is identified based on the grouping of non-zero coefficients within the two-dimensional matrix representation corresponding to the current block. Furthermore, the scan sequence ultimately identified for identifying the bounding box is identified as the candidate scan sequence that yields the closest grouping of non-zero coefficients. Therefore, the shape and size of the bounding box can be based on the arrangement of its closest grouping of non-zero coefficients.

[0102] In some such implementations, processing the quantization transform coefficients of the current block for multiple candidate scan sequences may include, for each candidate scan sequence, using that candidate scan sequence to identify candidate bounding boxes and determine the total number of coefficients enclosed within the candidate bounding boxes. The candidate bounding box with the lowest total number of coefficients can then be selected or identified as the bounding box for the current block.

[0103] In some such implementations, processing the quantization transform coefficients of the current block for multiple candidate scan orders may include using a first candidate scan order to identify candidate bounding boxes and determining the total number of coefficients enclosed within the candidate bounding boxes. The candidate bounding boxes themselves, or indications used to represent or indicate the candidate bounding boxes, may then be stored as pending best candidate bounding boxes. The remaining candidate scan orders are then traversed. For each new bounding box identified in this process, it may be determined whether the new bounding box encloses a lower total number of coefficients than the currently stored bounding box. If the new bounding box does indeed enclose such a lower total number of coefficients, the currently stored bounding box may be evicted from memory and replaced with the new bounding box. If the new bounding box does not enclose such a lower total number of coefficients, the new bounding box may be discarded. This process may be repeated until every available candidate scan order has been traversed, and then the bounding boxes stored at the end of the process may be selected or identified as the bounding boxes for the current block.

[0104] In some implementations, identifying the bounding box may include using a reference bounding box previously identified for a previously encoded block or image or video frame. For example, the reference bounding box may be a bounding box previously identified for a neighboring block of the current block being encoded. In another example, the reference bounding box may be the bounding box of a juxtaposed block within a reference image or video frame, such as a reference image or video frame for inter-frame prediction of the current block, or a reference image or video frame for inter-frame prediction of another block included in the same image or video frame as the current block. In some implementations, using the reference bounding box may include differentially encoding into the bitstream, wherein the differential signals to the decoder how to use the reference bounding box to represent the bounding box of the current block. In some implementations, the reference bounding box itself is signaled within the bitstream. In some implementations, both the differential and the reference bounding box are signaled within the bitstream.

[0105] In some implementations, multiple reference bounding boxes may be available during the encoding of the current block. In some such implementations, a buffer may be used to store data corresponding to one or more reference bounding boxes. For example, reference bounding boxes may be stored for some or all of the reference frames stored in a reference frame buffer that can be used to encode the current block. In another example, reference bounding boxes may be stored for some or all of the current block's adjacent blocks. In yet another example, reference bounding boxes may be stored for some or all of the reference frames and for some or all of the adjacent blocks.

[0106] In some implementations where multiple reference bounding boxes are available during the encoding of the current block, identifying the bounding box for the current block may include traversing some or all of the multiple reference bounding boxes to identify the best one for encoding the current block. For example, the current block may be traversed (e.g., using scan order processing or other processes for scanning its coefficients) to identify the non-zero coefficients of the current block within some or all of the multiple reference bounding boxes. The reference bounding box surrounding each non-zero coefficient of the current block and the minimum total number of coefficients in the current block can then be selected or identified as the bounding box for the current block.

[0107] In some implementations, identifying the bounding box for the current block may include identifying the bounding box without using a scan order. For example, the bounding box may be identified by traversing different bounding box candidates defined for the encoder, rather than by encoding previous blocks. For example, one or more bounding box candidates of different sizes, shapes, and / or positions within the current block may be processed for the quantization transform coefficients of the current block. For example, the bounding box candidates may surround the top leftmost M×N block of the quantization transform coefficients. In this case, both M and N are smaller than the total width or height of the current block. In some cases, the values ​​of M and N may be different. Other examples based on size, shape, and / or position are also possible.

[0108] In some such implementations, processing the quantization transform coefficients of the current block for bounding box candidates may include determining, for each bounding box candidate, the total number of coefficients enclosing the candidate bounding box. The bounding box candidate with the lowest total number of coefficients can then be selected or identified as the bounding box for the current block. In some such implementations, where two or more bounding box candidates share the lowest total number of coefficients, other criteria may be used to select or identify one of those bounding box candidates as the bounding box for the current block. For example, a bounding box candidate with a smaller size and / or a bounding box candidate whose DC coefficients are closest to the current block may be selected or identified as the bounding box for the current block. In some such implementations, a first bounding box candidate that satisfies a threshold score may be selected or identified as the bounding box for the current block. For example, the threshold score may correspond to or be based on a compression throughput target, the maximum total number of coefficients that can be enclosing the bounding box, and / or other criteria.

[0109] In some such implementations, processing the quantization transform coefficients of the current block for the bounding box candidate may include using machine learning. For example, by analyzing multiple bounding boxes identified for previously encoded blocks, a machine learning model can be trained to identify the optimal size, shape, and / or position of the two-dimensional matrix representation corresponding to the bounding box.

[0110] In some implementations, identifying the bounding box for the current block may include determining whether a reference bounding box is used in conjunction with the encoding of the current block. For example, determining whether to use a reference bounding box in conjunction with the encoding of the current block may include determining the amount of motion within the current block. For example, if the amount of motion reaches a threshold, it may indicate that too much information about a reference block from which the reference bounding box will be used differs from the information about the current block. In this case, the reference bounding box may not be used. The reference bounding box may be a single reference bounding box available during the encoding of the current block. Alternatively, the reference bounding box may be one of a plurality of reference bounding boxes available during the encoding of the current block.

[0111] Next reference Figure 13 The diagram illustrates a flowchart of an example of technique 1300 for decoding encoded image or video blocks using a sparse matrix representation, where the sparse matrix representation uses non-zero coefficients for the boundaries. At 1302, the size information of the bounding boxes is decoded from the encoded bitstream.

[0112] At 1304, the size information is used to decode syntax elements representing non-zero coefficients encoded into the encoded bitstream. For example, using the size information to decode the syntax elements may include using the size information to determine the position of the non-zero coefficients within the block being decoded.

[0113] At 1306, the syntax element is dequantized to generate transform coefficients. The transform coefficients are arranged in a two-dimensional matrix corresponding to the block being decoded.

[0114] At 1308, the transform coefficients are inversely transformed to produce decoded pixel data. If the decoding involves prediction residuals, the decoded pixel data may be, include, or refer to pixel values ​​before or after reconstructing the prediction residuals.

[0115] At 1310, the decoded pixel data is output for storage or display. For example, outputting the decoded pixel data for storage or display may include rendering the decoded pixel data as an image within a webpage. In another example, outputting the decoded pixel data for storage or display may include outputting an output video stream containing the decoded pixel data to a device for playing the video stream.

[0116] In some implementations, technique 1300 may include decoding a syntax element that indicates whether a bounding box is used to encode a block of code from the bitstream. For example, the syntax element may be a binary flag. The binary flag may be included in a block header for the encoded block, in an image or video frame header for an image or video frame that includes the encoded block, or in another portion of the bitstream. In some such implementations, the value of the syntax element indicating whether the bounding box is used may be checked as a preprocessing step, for example, before performing other operations described with respect to technique 1300. For example, in response to determining that the syntax element indicates that the bounding box is not used for encoding the block, some operations described with respect to technique 1300 may be skipped. In some such implementations, in response to determining that the syntax element indicates that the bounding box is not used to encode the block, operations for decoding syntax elements representing size information and / or operations for using the size information to decode the block may be skipped.

[0117] The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques and hardware components configured to perform all or part of those examples of encoding and / or decoding techniques. However, it should be understood that, as used in the claims, encoding and decoding can refer to the encoding, decoding, transformation, or other processing or alteration of data.

[0118] The word “example” is used herein to mean something used as an example, instance, or illustration. Any aspect or design described herein as an “example” is not necessarily to be construed as being more preferred or superior to other aspects or designs. Rather, the use of the word “example” is intended to present a concept in a specific manner. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless the context otherwise specifies or expressly indicates, the statement “X comprises A or B” is intended to mean any natural inclusion. That is, if X comprises A; X comprises B; or X comprises both A and B, then “X comprises A or B” is satisfied in any of the foregoing cases. Furthermore, the articles “a” and “an” used in this application and the appended claims should generally be interpreted as meaning “one or more” unless the context otherwise specifies or expressly indicates that it is for the singular form. Moreover, the terms “implementation” or “an embodiment” used throughout this disclosure are not intended to claim the same embodiment unless so described.

[0119] All or part of the embodiments of this disclosure may take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium may be, for example, any means that can tangibly contain, store, communicate, or transmit the program for use by or in conjunction with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media may also be used.

[0120] The above embodiments, examples, and aspects have been described to facilitate a clear understanding of this disclosure and are not intended to limit it. Rather, this disclosure is intended to cover various modifications and equivalents included within the scope of the appended claims, and this scope should be interpreted in the broadest possible sense permitted by law to cover all such modifications and equivalents.

Claims

1. A method for decoding coded blocks of image or video data, the method comprising: Decode the size information of the bounding box of the coded block from the bitstream; The size information is used to decode the syntax elements corresponding to the non-zero coefficients enclosed in the bounding box from the bitstream, wherein the syntax elements are arranged in a one-dimensional sequence format; The syntax elements are dequantized to generate transformed data, wherein the transformed data includes coefficients arranged in a two-dimensional matrix format; The transformed data is inversely transformed to generate decoded data; and The decoded data is output for storage or display.

2. The method of claim 1, wherein decoding the syntax element comprises: The size information is used to determine the position of the non-zero coefficient within the coding block.

3. The method according to claim 1, comprising: Decode from the bitstream the syntax element that indicates the bounding box was used to encode the coded block.

4. The method according to claim 3, wherein, The syntax element is decoded from the block header corresponding to the encoded block.

5. The method according to claim 3, wherein, The syntax elements are decoded from the image header or video frame header corresponding to the image or video frame that includes the coded block.

6. The method according to claim 1, wherein, The bounding box is based on the position of the non-zero coefficients according to the scan order used to encode the coded block.

7. The method of claim 1, wherein the bounding box has a non-rectangular shape.

8. The method of claim 1, wherein at least some zero-value coefficients of the coded block are outside the bounding box.

9. The method according to claim 1, wherein, The size information corresponds to one or both of the shape of the bounding box and the size of the bounding box.

10. The method of claim 1, wherein the bounding box only encloses the non-zero coefficients of the coded block.

11. A method for decoding coded blocks of image or video data, the method comprising: Decode the size information of the bounding box of the coded block from the bitstream, the bounding box corresponding to the boundary of the non-zero coefficient; The size information is used to decode the syntax elements, including non-zero coefficients enclosed in the bounding box, from the bitstream; Decoded data is generated by dequantizing and inverse transforming the syntax elements; and The decoded data is output for storage or display.

12. The method according to claim 11, wherein, All non-zero coefficients of the coded block are enclosed within the bounding box, and all coefficients of the coded block outside the bounding box are zero coefficients.

13. The method of claim 11, wherein the syntax elements are arranged in a one-dimensional sequence format and the decoded data is arranged in a two-dimensional matrix format.

14. The method of claim 11, comprising: Determine whether the value of the syntax element encoded into the bitstream indicates that the bounding box is used to encode the coded block; and In response to determining the value of the syntax element indicating that the bounding box is used to encode the coded block, the size information is decoded from the bitstream.

15. The method according to claim 11, wherein, The size information corresponds to one or both of the shape of the bounding box and the size of the bounding box.

16. A method for decoding coded blocks of image or video data, the method comprising: Identify the size information of the bounding box of the coded block; Identify syntax elements corresponding to coefficients enclosed within the bounding box, wherein the syntax elements are arranged in a one-dimensional sequence format; The coded block is decoded by dequantizing and inverse transforming the syntax elements to generate decoded data, wherein the decoded data is arranged in a two-dimensional matrix format; and The decoded data is output for storage or display.

17. The method according to claim 16, wherein, All non-zero coefficients of the coded block are enclosed within the bounding box, and all coefficients of the coded block outside the bounding box are zero coefficients.

18. The method according to claim 16, wherein, The size information and the syntax elements are decoded from the bitstream that includes the coded block.

19. The method of claim 16, wherein, The size information is identified based on the values ​​of syntax elements encoded into a bitstream, the bitstream including the encoded blocks that indicate the bounding boxes used to encode the encoded blocks.

20. The method of claim 16, wherein, The bounding box is based on the position of non-zero coefficients according to the scan order used to encode the coded block.

Images