Intra block copy with extended reference area

Abstract

Methods, systems, and computer-readable media for predicting pixels of a current block of a current frame using intra block copy with extended reference area are disclosed. A disclosed technique extends the available reference area for IntraBC prediction by identifying available local reference sub-blocks for a current sub-block within a video frame. This identification is based on a pre-determined mapping that links an index of the current sub-block to a corresponding set of reference sub-block indices. After the available reference sub-blocks are identified, pixel values for the current sub-block are predicted. This prediction utilizes a block vector to reference specific pixel values within the identified available local reference sub-blocks, thereby improving compression efficiency by enabling the use of a more effective and expanded set of predictor blocks.

Description

INTRA BLOCK COPY WITH EXTENDED REFERENCE AREACROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This disclosure claims the benefit of U.S. Provisional Patent Application No.63 / 711,652 filed October 24, 2024, the disclosure of which is incorporated by reference herein in its entirety.BACKGROUND

[0002] Digital video streams may represent video using a sequence of frames or still images. Digital video can be used for various applications including, for example, video conferencing, high- definition video entertainment, video advertisements, or sharing of usergenerated videos. A digital video stream can contain a large amount of data and consume a significant amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including compression and other coding techniques. These techniques may include both lossy and lossless coding techniques.SUMMARY

[0003] This disclosure relates generally to encoding and decoding video data and more particularly relates to intra block copy with extended reference area.

[0004] An aspect of the disclosed implementations is a method for predicting pixels of a current block of a current frame using intra block copy with extended reference area. The method includes identifying available local reference sub-blocks for a current sub-block based on a pre-determined mapping between a current sub-block index, corresponding to the current sub-block, and reference sub-block indices corresponding to the available local reference sub-blocks, and predicting pixel values of the current sub-block based on a block vector and pixel values of the identified available local reference sub-blocks.

[0005] An aspect of the disclosed implementations is a device including a memory and a processor configured to execute instructions stored in the memory to perform the foregoing method.

[0006] An aspect of the disclosed implementations is a non-transitory computer-readable storage medium including executable instructions that, when executed by a processor, facilitate performance of operations that perform the foregoing method.

[0007] An aspect of the disclosed implementations is a non-transitory computer-readable storage medium having stored thereon an encoded bitstream comprising a syntax element for decoding by a decoder or generated by an encoder using the foregoing method.

[0008] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims, and the accompanying figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The description herein refers to the accompanying drawings described below wherein like reference numerals refer to like parts throughout the several views.

[0010] FIG. 1 is a schematic of a video encoding and decoding system.

[0011] FIG. 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station.

[0012] FIG. 3 is a diagram of an example of a video stream to be encoded and subsequently decoded.

[0013] FIG. 4 is a block diagram of an encoder.

[0014] FIG. 5 is a block diagram of a decoder.

[0015] FIG. 6 is a block diagram illustrating an IntraBC mode.

[0016] FIG. 7 is a block diagram illustrating sub-pixel positions.

[0017] FIG. 8 is a block diagram illustrating global and local reference areas for an intraBC mode.

[0018] FIG. 9 is a block diagram illustrating an extended local reference area for an intraBC mode.

[0019] FIG. 10 is a flowchart describing a technique for intra block copy with extended reference area.DETAILED DESCRIPTION

[0020] As mentioned above, compression schemes related to coding video streams may include breaking images (i.e., original or source images) into blocks and generating a digital video output bitstream using one or more techniques to limit the information included in the output. A received encoded bitstream can be decoded to re-create the blocks and the source images from the limited information. Encoding a video stream, or a portion thereof, such as a frame or a block, can include using temporal or spatial similarities in the videostream to improve coding efficiency. For example, a current block of a video stream may be encoded based on identifying a difference (residual) between previously coded pixel values and those in the current block. In this way, only the residual and parameters used to generate the residual need be added to the encoded bitstream. The residual may be encoded using a lossy quantization step. Decoding (i.e., reconstructing) an encoded block from such a residual often results in a distortion between the original and the reconstructed block.

[0021] Encoding using temporal similarities is known as inter prediction. Inter prediction uses a motion vector that represents the temporal displacement of a previously coded block relative to the current block. The motion vector can be identified using a method of motion estimation, such as a motion search. In the motion search, a portion of a reference frame can be translated to a succession of locations to form a predictor block that can be subtracted from a portion of a current frame to form a series of residuals. The horizontal and / or vertical translations corresponding to the location having, e.g., the smallest residual can be selected as the motion vector. The motion vector can be encoded in the encoded bitstream along with an indication of the reference frame.

[0022] Encoding using spatial similarities is known as intra prediction. Using an intra-prediction mode, intra prediction can attempt to predict the pixel values of a current block of a current frame of a video stream using pixels peripheral to the current block. The pixels peripheral to the current blocks are pixels within the current frame but that are outside the current block. The pixels peripheral to the block can be pixels adjacent to the current block. Which pixels peripheral to the block are used can depend on the intra-prediction mode and / or a scan order of the blocks of a frame. For example, in a raster scan order, peripheral pixels above a current block (i.e., the block being encoded or decoded) and / or peripheral pixels to the left of the current block may be used.

[0023] Intra Block Copy (IntraBC) is a technique that utilizes a block vector (BV) to obtain a predictor for the current block from a previously coded block of pixels in the same frame. The block vector identifies the location of a block of pixels to obtain the predictor from, such as by specifying an x, y offset from the current block. IntraBC is conventionally limited in its ability to predict pixels as it is limited to frames categorized as screen content, block vectors do not utilize sub-pixel precision, and compound prediction is not available. In addition, IntraBC may take an undesirable amount of time for prediction as compared to other intra modes because the block vector may refer to pixels no longer stored in a cache (e.g., an on-chip storage with limited space but with faster access times) accessible to theencoder or decoder. Instead, the pixels may need to be accessed from a non-cache memory unit (e.g., RAM) which has slower access times.

[0024] Implementations according to this disclosure solve problems such as these including by using a pre-determined mapping to identify available local reference sub-blocks usable for IntraBC prediction using a block vector. The available local reference sub-blocks may be stored in a local cache with faster access than a non-cache memory unit where previously coded sub-blocks are stored. In some implementations, the number of available local reference sub-blocks is 864x64 blocks. The pre- determined mapping allows the encoder and decoder to select the same blocks to be stored in the local cache and to map the blocks to the correct location in the frame. In some implementations, the block vector may also reference pixels outside the available local reference sub-blocks (e.g., which may be referred to as global reference sub-blocks). In some implementations, some sub-blocks may be unavailable for use in prediction (e.g., one or two previously coded / decoded superblocks), to account for a delay in storing those superblocks in non-cache memory. Implementations of this disclosure may arrange sub-blocks in superblocks, for example, 128x128 or 256x256 superblocks of 64x64 sub-blocks.

[0025] In some implementations, a weighted combination of pixels predicted using an IntraBC mode and a different intra mode may be used to generate a predictor for a sub-block. In some implementations, the block vector for the IntraBC mode may be extended to support sub-pixel prediction, such as to (A pixel or 14 pixel precision.

[0026] A screen content categorization, for example, may be applied to a frame or video that includes content having repeated patterns and / or sharp edges, such as may be found in a video capture of a computer screen. When a frame or video is categorized as screen content, a different set of coding and decoding tools may be utilized to code and decode the frame or video. For example, conventionally, an intra block copy mode would only be available for use if a frame or video were categorized as screen content.

[0027] In some implementations of this disclosure, an encoder and decoder may be configured to permit the selection of IntraBC mode for all frames, and not just those categorized as screen content. For example, in some implementations, an IntraBC mode may always be available for selection. For example, in some implementations, enabling or disabling the availability of an IntraBC mode may be effectuated at the frame level. For example, an intraBC_enable_flag may be signaled for a frame irrespective of the value of other screen-content-related flags, thereby making the IntraBC tool universally available for both screen content and natural, camera-captured content.

[0028] Implementations of this disclosure may provide for a coding gain when encoding and decoding video. For example, coding gains of between .11% and .24% may be achieved.

[0029] Further details of intra block copy with extended reference area are described herein with initial reference to a system in which it can be implemented. FIG. 1 is a schematic of a video encoding and decoding system 100. A transmitting station 102 can be, for example, a computer having an internal configuration of hardware such as that described in FIG. 2. However, other suitable implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 can be distributed among multiple devices.

[0030] A network 104 can connect the transmitting station 102 and a receiving station 106 for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station 102 and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network or any other means of transferring the video stream from the transmitting station 102 to, in this example, the receiving station 106.

[0031] The receiving station 106, in one example, can be a computer having an internal configuration of hardware such as that described in FIG. 2. However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0032] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation can omit the network 104. In another implementation, a video stream can be encoded and then stored for transmission at a later time to the receiving station 106 or any other device having memory. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and / or some communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network 104. In another implementation, a transport protocol other than RTP may be used, e.g., a Hypertext Transfer Protocol (HTTP) video streaming protocol.

[0033] When used in a video conferencing system, for example, the transmitting station 102 and / or the receiving station 106 may include the ability to both encode and decode a video stream as described below. For example, the receiving station 106 could be avideo conference participant who receives an encoded video bitstream from a video conference server (e.g., the transmitting station 102) to decode and view and further encodes and transmits its own video bitstream to the video conference server for decoding and viewing by other participants.

[0034] FIG. 2 is a block diagram of an example of a computing device 200 (e.g., an apparatus) that can implement a transmitting station or a receiving station. For example, the computing device 200 can implement one or both of the transmitting station 102 and the receiving station 106 of FIG. 1. The computing device 200 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.

[0035] A CPU 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the CPU 202 can be any other type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed. Although the disclosed implementations can be practiced with one processor as shown, e.g., the CPU 202, advantages in speed and efficiency can be achieved using more than one processor.

[0036] A memory 204 in computing device 200 can be a read only memory (ROM) device or a random-access memory (RAM) device in an implementation. Any other suitable type of storage device can be used as the memory 204. The memory 204 can include code and data 206 that is accessed by the CPU 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, the application programs 210 including at least one program that permits the CPU 202 to perform the techniques described here. For example, the application programs 210 can include applications 1 through N, which further include a video coding application that performs the techniques described here. Computing device 200 can also include a secondary storage 214, which can, for example, be a memory card used with a mobile computing device. Because the video communication sessions may contain a significant amount of information, they can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0037] The computing device 200 can also include one or more output devices, such as a display 218. The display 218 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 218 can be coupled to the CPU 202 via the bus 212. Other output devices that permit a user to program or otherwise use the computing device 200 can be provided in addition toor as an alternative to the display 218. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display or light emitting diode (LED) display, such as an organic LED (OLED) display.

[0038] The computing device 200 can also include or be in communication with an image-sensing device 220, for example a camera, or any other image-sensing device 220 now existing or hereafter developed that can sense an image such as the image of a user operating the computing device 200. The image-sensing device 220 can be positioned such that it is directed toward the user operating the computing device 200. In an example, the position and optical axis of the image- sensing device 220 can be configured such that the field of vision includes an area that is directly adjacent to the display 218 and from which the display 218 is visible.

[0039] The computing device 200 can also include or be in communication with a sound-sensing device 222, for example a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device 200. The sound- sensing device 222 can be positioned such that it is directed toward the user operating the computing device 200 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device 200.

[0040] Although FIG. 2 depicts the CPU 202 and the memory 204 of the computing device 200 as being integrated into one unit, other configurations can be utilized. The operations of the CPU 202 can be distributed across multiple machines (wherein individual machines can have one or more of processors) that can be coupled directly or across a local area or other network. The memory 204 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device 200. Although depicted here as one bus, the bus 212 of the computing device 200 can be composed of multiple buses. Further, the secondary storage 214 can be directly coupled to the other components of the computing device 200 or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. The computing device 200 can thus be implemented in a wide variety of configurations.

[0041] FIG. 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of adjacent frames 304. While three frames are depicted as the adjacent frames 304, the video sequence 302 can include any number ofadjacent frames 304. The adjacent frames 304 can then be further subdivided into individual frames, e.g., a frame 306. At the next level, the frame 306 can be divided into a series of planes or segments 308. The segments 308 can be subsets of frames that permit parallel processing, for example. The segments 308 can also be subsets of frames that can separate the video data into separate colors. For example, a frame 306 of color video data can include a luminance plane and two chrominance planes. The segments 308 may be sampled at different resolutions.

[0042] Whether or not the frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also be of any other suitable size such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise noted, the terms block and macro-block are used interchangeably herein.

[0043] FIG. 4 is a block diagram of an encoder 400. The encoder 400 can be implemented, as described above, in the transmitting station 102 such as by providing a computer software program stored in memory, for example, the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the CPU 202, cause the transmitting station 102 to encode video data in the manner described in FIG. 4. The encoder 400 can also be implemented as specialized hardware included in, for example, the transmitting station 102. In one particularly desirable implementation, the encoder 400 is a hardware encoder.

[0044] The encoder 400 has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy encoding stage 408. The encoder 400 may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. In FIG. 4, the encoder 400 has the following stages to perform the various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filtering stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300.

[0045] When the video stream 300 is presented for encoding, respective frames 304, such as the frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, respective blocks can be encoded using intra-frame prediction (also called intra-prediction) or inter- frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames.

[0046] Next, still referring to FIG. 4, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also called a residual). The transform stage 404 transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage 406 converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated. The quantized transform coefficients are then entropy encoded by the entropy encoding stage 408. The entropy-encoded coefficients, together with other information used to decode the block, which may include for example the type of prediction used, transform type, MVs and quantizer value, are then output to the compressed bitstream 420. The compressed bitstream 420 can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

[0047] The reconstruction path in FIG. 4 (shown by the dotted connection lines) can be used to ensure that the encoder 400 and a decoder 500 (described below) use the same reference frames to decode the compressed bitstream 420. The reconstruction path performs functions that are similar to functions that take place during the decoding process that are discussed in more detail below, including dequantizing the quantized transform coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also called a derivative residual). At the reconstruction stage 414, the prediction block that was predicted at the intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. The loop filtering stage 416 can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

[0048] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. For example, a non-transform-based encoder can quantize the residual signal directly without the transform stage 404 for certain blocks or frames. In anotherimplementation, an encoder can have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0049] FIG. 5 is a block diagram of a decoder 500. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the CPU 202, cause the receiving station 106 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.

[0050] The decoder 500, similar to the reconstruction path of the encoder 400 discussed above, includes in one example the following stages to perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512 and a post-loop filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0051] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage 412 in the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 can use the intra / inter prediction stage 508 to create the same prediction block as was created in the encoder 400, e.g., at the intra / inter prediction stage 402. At the reconstruction stage 510, the prediction block can be added to the derivative residual to create a reconstructed block. The loop filtering stage 512 can be applied to the reconstructed block to reduce blocking artifacts.

[0052] Other filtering can be applied to the reconstructed block. In this example, the post-loop filtering stage 514 is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the terms will be used interchangeably herein. Other variations of the decoder 500 can be used to decode thecompressed bitstream 420. For example, the decoder 500 can produce the output video stream 516 without the post-loop filtering stage 514.

[0053] FIG. 6 is a block diagram illustrating the conventional IntraBC mode. FIG. 6 illustrates a current frame 600 being coded and a current block 602 within the current frame 600. It is also noted that the IntraBC mode is limited to intra-coded frames.

[0054] In the IntraBC mode, an encoder (e.g., the encoder 400 of FIG. 4) searches at least a subset of a reconstructed area 603 of the current frame 600 to identify a reference block (e.g., a reference block 604) that best matches the current block 602. The reference block 604 is used as a prediction block for the current block 602. A difference (i.e., residual) between the current block 602 and the reference block 604 is encoded in a compressed bitstream (e.g., the compressed bitstream 420 of FIG. 4) along with a BV, such as a block vector 606. As such, IntraBC can be compared to “motion compensation”, but within the same frame (e.g., the current frame 600), where a BV is used instead of a motion vector. The encoder encodes a flag (e.g., an IntraBC flag) indicating that the current block 602 is encoded using the IntraBC mode and also encodes the BV.

[0055] When decoding the current block 602, a decoder, such as the decoder 500 of FIG. 5, decodes the IntraBC flag to determine whether the current block 602 is to be decoded using the IntraBC mode. If so, then the decoder decodes the block vector 606 from the compressed bitstream to identify the reference block 604 therewith obtaining the prediction block for the current block 602. A residual block can then be decoded from the compressed bitstream to add to the prediction block therewith reconstructing the current block 602.

[0056] FIG. 7 is a block diagram illustrating sub-pixel positions that may be used by block vectors with sub-pixel precision. Conventionally, an IntraBC mode BV refers to full pixel positions as illustrated by int-pel positions 702. A BV with sub-pixel positions may additionally refer to fractional pixels in the x and / or y directions, such as half-pel positions 704 and quarter-pel positions 706. In some implementations, the availability of sub-pixel positions may be constrained by the availability of local reference sub-blocks. The generation of a sub-pixel prediction value includes applying an interpolation filter to surrounding integer-pixel samples. Consequently, the availability of sub-pixel precision for a block vector may be constrained by the boundaries of the available reference sub-blocks. For instance, if a block vector points to a location near a top or bottom boundary of a reference area, the integer-pixel samples required for vertical interpolation may not be available. In such a scenario, the vertical component of the block vector may be restricted to integer-pixel precision, while the horizontal component may still utilize sub-pixel precision. When a BVrefers to a sub-pixel position, the pixel value for that sub-pixel position is inferred from surrounding pixel values, such as by using a vertical and a horizontal multi-tap filters to estimate the pixel value at the sub-pixel position. For example, implementations of an intraBC mode, such as described with respect to FIGS. 8, 9, or 10, may utilize a BV with subpixel precision, such as a half-pel or quarter-pel precision.

[0057] FIG. 8 is a block diagram illustrating global and local reference areas for an intraBC mode. Frame 802 includes super-blocks which include sub-blocks and an uncoded / not decoded area 804. As depicted, super-blocks are 128x128 and sub-blocks are 64x64, but any size blocks may be used. The legend of FIG. 8 indicates how super-blocks 812, sub-blocks 814, global reference area 822, local reference area / current block 824, and unavailable areas 826 are depicted.

[0058] In frame 802, there are multiple current blocks because intra coding can be performed concurrently on a row-by-row basis once sufficient blocks from prior row(s) have been coded / decoded to perform intra prediction. On a row basis, the prior two super-blocks that have been coded / decoded are identified as unavailable due to the delay in storing coded / decoded blocks in non-cache memory. Depending on the implementation, the number of prior blocks that are unavailable may vary.

[0059] The sub-blocks that are identified as a current block are also available as a local reference area. For example, intra coding may be performed on a 4x4 block basis and previously coded 4x4 blocks in the sub-block may be available for prediction using intra BC. Blocks denoted as within the global reference area 822 may also be used for prediction using intra BC. Blocks that are within an unavailable area 826 or are in the have not yet been coded / decoded area 804 are not available for prediction using intra BC.

[0060] FIG. 9 is a block diagram illustrating an extended local reference area (i.e., extended reference area) for an intra BC mode. The illustration of FIG. 9 may be understood as an extension of the intraBC mode illustrated in FIG. 8. FIG. 9 includes illustrations of extended local reference areas utilized with 256x256 super-blocks 902 and 128x128 superblocks 906. Specifically, super-blocks 902 illustrate an application of an extended local reference area for 256x256 super-blocks and super-blocks 906 illustrate an application of an extended local reference area for 128x128 super-blocks. In both illustrations, sub-blocks are illustrated as 64x64. FIG. 9 depicts super-blocks relevant to determining the extended local reference area for a given current block.

[0061] The respective super-block indices 904, 908 may be applied to corresponding sub-block locations in respective super-blocks 902, 906 to identify, for a current block, whichblocks are in a local reference area. The super-block indices 904, 908 may provide indices for two or more adjacent super-blocks, depending on the available buffer size for the extended local reference area. The sub-blocks within the extended local reference area (as identified according to local reference area identification 952) are identified based on respective superblock indices 904, 908 and a pre-determined mapping between the index of the current block (as identified according to current block identification 950) and reference sub-block indices. Blocks that are within an unavailable area 826 or are in the have not yet been coded / decoded area 804 are not available for prediction. For example, the pre-determined mapping may take the form of a reference table known to both the encoder and decoder, such as the tables below (which, for a given current block index, provides the indices of the reference blocks in the local reference area):64x64 sub-block of current super-block Reference 64x64 sub-blocks0 0, 4, 5, 6, 7, 9, 10, 111 0, 1, 4, 5, 6, 7, 10, 112 0, 1, 2, 4, 5, 6, 7, 113 0, 1, 2, 3, 4, 5, 6, 7(Table 1) 64x64 sub-block of current super-block Reference 64x64 sub-blocks0 0, 25, 26, 27, 28, 29, 30, 311 0, 1, 26, 27, 28, 29, 30, 312 0, 1, 4, 5, 2, 27, 30, 313 0, 1, 4, 5, 2, 3, 30, 314 0, 1, 4, 26, 27, 29, 30, 315 0, 1, 4, 5, 26, 27, 30, 316 0, 1, 2, 3, 4, 5, 6, 317 0, 1, 2, 3, 4, 5, 6, 78 1, 2, 3, 4, 5, 6, 7, 89 2, 3, 4, 5, 6, 7, 8, 910 3, 6, 7, 8, 9, 12, 13, 1011 6, 7, 8, 9, 10, 11, 12, 1312 2, 3, 5, 6, 7, 8, 9, 1213 2, 3, 6, 7, 8, 9, 12, 1314 7, 8, 9, 10, 11, 12, 13, 14 15 8, 9, 10, 11, 12, 13, 14, 15(Table 2)

[0062] In the foregoing example tables, Table 1 refers to an example pre-determined mapping for 128x128 super-blocks and Table 2 refers to an example pre- determined mapping for 256x256 super-blocks.

[0063] Variations on the illustrated super-block sizes, sub-block sizes, super-block indices, and pre-determined mappings are possible depending on the implementation. For example, in some implementations, the pre-determined mapping may be generated algorithmically, for example, as a function of a current block index which produces a tuple of reference block indices. For example, in some implementations, a pre- determined mapping maps a current block index to a different number of reference blocks (e.g., other than 8). Tables 3 and 4, below, are example reference tables that may be utilized if, respectively, five or six reference blocks are utilized.Superblock Size 64x64 sub-block of Reference 64x64 sub-blocks current super-block128x128 superblock 0 0, 4, 5, 6, 71 0, 1, 5, 6, 72 0, 1, 2, 6, 73 0, 1, 2, 3, 7256x256 superblock 0 0, 26, 27, 30, 311 0, 1, 27, 30, 312 0, 1, 2, 4, 53 1, 2, 3, 4, 54 0, 1, 4, 30, 315 0, 1, 4, 5, 316 2, 3, 4, 5, 67 2, 3, 5, 6, 78 2, 3, 6, 7, 89 3, 6, 7, 8, 910 8, 9, 10, 12, 1311 9, 10, 11, 12, 1312 6, 7, 8, 9, 1213 7, 8, 9, 12, 1314 10, 11, 12, 13, 1415 10, 11, 13, 14, 15(Table 3)Superblock Size 64x64 sub-block of Reference 64x64 sub-blocks current super-block128x128 superblock 0 0, 4, 5, 6, 7, 111 0, 1, 4, 5, 6, 72 0, 1, 2, 5, 6, 73 0, 1, 2, 3, 6, 7256x256 superblock 0 0, 29, 26, 27, 30, 311 0, 1, 26, 27, 30, 312 0, 1, 2, 4, 5, 313 0, 1, 2, 3, 4, 54 0, 1, 4, 27, 30, 315 0, 1, 4, 5, 30, 316 1, 2, 3, 4, 5, 67 2, 3, 4, 5, 6, 78 2, 3, 5, 6, 7, 89 2, 3, 6, 7, 8, 910 7, 8, 9, 10, 12, 1311 8, 9, 10, 11, 12, 1312 3, 6, 7, 8, 9, 1213 6, 7, 8, 9, 12, 1314 9, 10, 11, 12, 13, 1415 10, 11, 12, 13, 14, 15(Table 4)

[0064] FIG. 10 is a flowchart describing a technique 1000 for intra block copy with extended reference area. Technique 1000 may be performed by an encoder or decoder, such as encoder 400 using intra / inter prediction stage 402 or decoder 500 using intra / inter prediction stage 508. Technique 1000 may be used with the super-blocks, indices, mapping tables, and other configurations shown and described with respect to FIG. 9.

[0065] In some implementations, technique 1000 may be performed responsive to a determination that an intra block copy mode is to be utilized that is not based on the current frame being categorized as a screen content type. In some implementations, an encoder may make a determination to use an intra block copy mode for a block. A syntax element may be coded in a compressed bitstream with respect to a frame, block, or combination thereof that indicates whether an intra block copy mode is to be utilized. The decoder may utilize such a syntax element to determine that intra block copy mode was utilized to encode the block and cause the decoder to perform technique 1000 to recreate the prediction utilized by the encoder.

[0066] Step 1002 includes identifying available local reference sub-blocks for a current sub-block based on a pre- determined mapping between a current sub-block index and reference sub-block indices. Examples of identified available local reference sub-blocks are illustrated in FIG. 9 as local reference areas in super-blocks 902, 906 (as indicated by local reference area identification 952). Examples of implementing a pre-determined mapping using a reference table are illustrated above. When a reference table is used, it may include rows mapping each sub-block index of a current super-block to reference sub-block indices corresponding to the current super-block and at least one prior super-block in raster order. In some implementations, there are eight reference sub-block indices corresponding to each subblock index. In some implementations, the current sub-block is in a current super-block and the pre-determined mapping is based on a size of the current super-block.

[0067] As shown with respect to FIG. 9, an implementation may have different predetermined mappings for 128x128 and 256x256 super-blocks. As shown with respect to FIG.9, an implementation may have current sub-block and available local reference sub-blocks that are 64x64. In such implementations utilizing 128x128 super-blocks, the current superblock and two previously coded super-blocks in the vicinity of the current super-block each include at least one of the available local reference sub-blocks. Different block sizes are possible depending on the implementation.

[0068] Step 1004 includes predicting pixel values of the current sub-block based on a block vector and pixel values of the identified available local reference sub-blocks. For example, with respect to FIG. 9, pixel values in a current block in one of super-blocks 902, 906 (as indicated by current block identification 950) may be predicted from pixel values in the local reference areas that are identified using a block vector (as illustrated according to local reference area identification 952). In some implementations, the block vector may have single-pixel precision, sub-pixel precision, or a combination thereof. When sub-pixelprecision is utilized, for example, the block vector refers to a sub-pixel position in one of the identified available local reference sub-blocks. Sub-pixel prediction is further illustrated with respect to FIG. 7. For example, a precision syntax element may be coded or decoded indicating whether to utilize a 2-pixel, 1 -pixel, or Vi- pixel precision followed by a block vector that is coded / decoded based on the precision syntax element value. In some implementations, a binary syntax element may be utilized to signal whether 1 -pixel or 14-pixel precision is utilized. Depending on the implementation, one or more interpolation filters, such as bicubic or bilinear filters, may be utilized to generate the interpolated pixels. For example, because there is not a pixel value between pixels, one or more filters may be applied to pixels (or previously-generated sub-pixels) above and below and / or to the left and the right of the sub-pixel position in order to estimate the pixel value at the sub-pixel position.

[0069] In some implementations, the signaling of the block vector precision may be separated from the signaling of the block vector components. Specifically, a precision indicator may be signaled in the bitstream prior to the block vector data. In some implementations, this precision indicator may be a ternary symbol that is signaled at the block level to indicate one of three supported precision modes, such as a 2-pixel, an integerpixel (1-pel), or a quarter-pixel (14-pel) precision. In another implementation, the indicator may be a binary symbol to select between two modes, such as 1-pel and half-pixel C / i-pel) precision. A decoder, upon parsing this symbol, determines the precision for the current block and subsequently parses the block vector components (e.g., such as by using motion vector difference (MVD) coding) according to the indicated precision. This two-step signaling process provides flexibility while maintaining coding efficiency, as the block vector representation can be adapted without altering the parsing logic.

[0070] In some implementations, predicting pixel values of the current sub-block is based on a weighted combination of the pixel values of the identified available local reference sub-blocks identified based on the block vector and pixel values obtained using a different intra prediction mode. For example, if a 4x4 block of pixels are being predicted within the sub-block, a first predicted block of pixels are predicted using intra block copy and a second predicted block of pixels are predicted using a different, second intra mode. This second intra mode may be a predefined mode known to both the encoder and decoder, or it may be explicitly signaled in the bitstream for a given block. In some implementations, the second intra mode is a DC prediction mode, where prediction is based on an average of neighboring pixel values. In another example, a SMOOTH prediction mode may be used.Other modes may be utilized, such as paeth, recursive, directional, or other intra prediction methodologies that may be available, e.g., in published or in development video coding standards.

[0071] In some implementations, a compound intraBC mode is signaled and the different intra mode used is fixed (e.g., always SMOOTH or DC mode). In some implementations, a compound intraBC mode is signaled as well as a mode indicating which intra prediction mode to use for the different intra mode. In some implementations, the different intra mode may be determined or constrained based on contextual information previously encoded or decoded. The first predicted block and the second predicted block may be combined into a prediction block using a weighted average, such as by using the following equation:p = (w* Pl + (64-w) * P2) » 6where P is the prediction block, w is a weighting factor (e.g., between 1 and 63), Pl is the first predicted block, P2 is the second predicted block, and calculations are performed for each pixel value in the blocks.(Formula 1)

[0072] In some implementations, the foregoing weighting factor may be a predetermined value, a value signaled in the bitstream, or a combination thereof. For example, in some implementations, the weighting factor may be a fixed value known to both the encoder and decoder, such as 32. For example, in some implementations, the weighting factor may be signaled to the decoder at a block level or at a frame level.

[0073] Information signaled from encoder to decoder may be encoded into one or more syntax elements of a compressed bitstream. For example, respective syntax elements, or combinations thereof, may be used to signal a block vector, block vector precision, whether intra block copy mode is available for use, whether intra block copy mode is utilized for encoding and decoding a particular block, or whether a combination of intra block copy mode and another prediction mode is utilized for a given block. A decoder may decode such syntax element(s) and utilize the decoded information to decode a block in accordance with technique 1000.

[0074] For simplicity of explanation, the technique 1000 of FIG. 10 is depicted and described as a series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustratedsteps or operations may be required to implement a method in accordance with the disclosed subject matter.

[0075] The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

[0076] Some implementations are described below as numbered examples (Example A, B, C, etc.). These examples are provided as examples only and do not limit the other implementations disclosed herein.

[0077] Example A is a method for predicting pixels of a current block of a current frame using intra block copy with extended reference area, including: identifying available local reference sub-blocks for a current sub-block based on a pre-determined mapping between a current sub-block index, corresponding to the current sub-block, and reference sub-block indices corresponding to the available local reference sub-blocks; and predicting pixel values of the current sub-block based on a block vector and pixel values of the identified available local reference sub-blocks.

[0078] Example B is the method of Example A, where the pre-determined mapping is a reference table that includes rows mapping each sub-block index of a current super-block to reference sub-block indices corresponding to the current super-block and at least one prior super-block in raster order.

[0079] Example C is the method of Example B, where there are eight reference subblock indices.

[0080] Example D is the method of any of Examples A to C, where the current subblock is in a current super-block and where the pre-determined mapping is based on a size of the current super-block.

[0081] Example E is the method of Example D, where the size of the current superblock is 128x128, a size of the current sub-block and identified available local reference subblocks is 64x64, and the current super-block and two previously coded super-blocks in a vicinity of the current super-block each include at least one of the available local reference sub-blocks.

[0082] Example F is the method of any of Examples A to E, where identifying available local reference sub-blocks and predicting pixel values are performed responsive to a determination that an intra block copy mode is to be utilized that is not based on the current frame being categorized as a screen content type.

[0083] Example G is the method of any of Examples A to F, where the block vector refers to a sub-pixel position in one of the identified available local reference sub-blocks.

[0084] Example H is the method of any of Examples A to G, where predicting pixel values of the current sub-block is based on a weighted combination of the pixel values of the identified available local reference sub-blocks identified based on the block vector and pixel values obtained using a different intra prediction mode.

[0085] Example I is the method of Example H, where the weighted combination is based on one of a pre-determined value or a value signaled in a compressed bitstream.

[0086] Example J is a device, including: a processor that is configured to perform the method of any of Examples A to I.

[0087] Example K is a device, including: a memory; and a processor, the processor configured to execute instmctions stored in the memory to perform the method of any of Examples A to I.

[0088] Example L is a non-transitory computer-readable storage medium, including executable instructions that, when executed by a processor, facilitate performance of operations, including operations that perform the method of any of Examples A to I.

[0089] Example M is a non-transitory computer-readable storage medium having stored thereon an encoded bitstream including a syntax element decodable by a decoder using the method of any of Examples A to I or generated by an encoder using the method of any of Examples A to I.

[0090] Example N is a computer program product, including executable instructions that, when executed by a processor, facilitate performance of operations, including operations that perform the method of any of Examples A to I.

[0091] Example O is a computer program product including an encoded bitstream, where the encoded bitstream is decodable by a decoder using the method of any of Examples A to I or was generated by an encoder using the method of any of Examples A to I.

[0092] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X includes A or B” is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoinginstances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. Moreover, use of the term “an implementation” or “one implementation” throughout is not intended to mean the same embodiment or implementation unless described as such.

[0093] Implementations of the transmitting station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc., stored thereon and / or executed thereby, including by the encoder 400 and the decoder 500) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0094] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 can be implemented using a general-purpose computer or general-purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms and / or instructions described herein. In addition, or alternatively, for example, a special purpose computer / processor can be utilized which can contain other hardware for carrying out any of the methods, algorithms, or instructions described herein.

[0095] The transmitting station 102 and the receiving station 106 can, for example, be implemented on computers in a video conferencing system. Alternatively, the transmitting station 102 can be implemented on a server and the receiving station 106 can be implemented on a device separate from the server, such as a hand-held communications device. In this instance, the transmitting station 102 can encode content using an encoder 400 into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder 500. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station 102. Other suitable transmitting and receiving implementation schemes are available. For example, the receiving station 106 can be a generally stationary personal computer ratherthan a portable communications device and / or a device including an encoder 400 may also include a decoder 500.

[0096] Further, all or a portion of implementations of the present disclosure can take the form of a computer program product accessible from, for example, a non-transitory computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport a program including instructions for use by or in connection with any processor. For example, a processor may be configured to perform executed instructions stored in the memory (e.g., computer readable medium) to perform techniques embodied in the instructions. For example, a non-transitory computer-readable storage medium may include executable instructions that, when executed by a processor, facilitate performance of operations corresponding to techniques described in this disclosure. For example, a non-transitory computer-readable storage medium may store an encoded bitstream that is encodable or decodable using techniques described in this disclosure. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable mediums are also available.

[0097] The above-described embodiments, implementations and aspects have been described in order to allow easy understanding of the present invention and do not limit the present invention. On the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structure as is permitted under the law.

Claims

What is claimed is:

1. A method for predicting pixels of a current block of a current frame using intra block copy with extended reference area, comprising:identifying available local reference sub-blocks for a current sub-block based on a pre-determined mapping between a current sub-block index, corresponding to the current sub-block, and reference sub-block indices corresponding to the available local reference subblocks; andpredicting pixel values of the current sub-block based on a block vector and pixel values of the identified available local reference sub-blocks.

2. The method of claim 1, wherein the pre-determined mapping is a reference table that includes rows mapping each sub-block index of a current super-block to reference sub-block indices corresponding to the current super-block and at least one prior super-block in raster order.

3. The method of claim 2, wherein there are eight reference sub-block indices.

4. The method of any of claims 1 to 3, wherein the current sub-block is in a current super-block and wherein the pre-determined mapping is based on a size of the current super-block.

5. The method of claim 4, wherein the size of the current super-block is 128x128, a size of the current sub-block and identified available local reference sub-blocks is 64x64, and the current super-block and two previously coded super-blocks in a vicinity of the current super-block each include at least one of the available local reference sub-blocks.

6. The method of any of claims 1 to 5, wherein identifying available local reference sub-blocks and predicting pixel values are performed responsive to a determination that an intra block copy mode is to be utilized that is not based on the current frame being categorized as a screen content type.

7. The method of any of claims 1 to 6, wherein the block vector refers to a subpixel position in one of the identified available local reference sub-blocks.

8. The method of any of claims 1 to 7, wherein predicting pixel values of the current sub-block is based on a weighted combination of the pixel values of the identified available local reference sub-blocks identified based on the block vector and pixel values obtained using a different intra prediction mode.

9. The method of claim 8, wherein the weighted combination is based on one of a pre-determined value or a value signaled in a compressed bitstream.

10. A device, comprising:a processor that is configured to perform the method of any of claims 1 to 9.

11. A device, comprising:a memory; anda processor, the processor configured to execute instructions stored in the memory to perform the method of any of claims 1 to 9.

12. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising operations that perform the method of any of claims 1 to 9.

13. A non-transitory computer-readable storage medium having stored thereon an encoded bitstream comprising a syntax element for decoding by a decoder using the method of any of claims 1 to 9 or generated by an encoder using the method of any of claims 1 to 9.

14. A computer program product, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising operations that perform the method of any of claims 1 to 9.

15. A computer program product comprising an encoded bitstream including a syntax element for decoding by a decoder using the method of any of claims 1 to 9 or was generated by an encoder using the method of any of claims 1 to 9.

Images