Content adaptive segmentation prediction
Patent Information
- Application Number
- JP2024008793
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-05-27
- Filing Date
- 2024-01-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-06-16
AI Technical Summary
Existing video coding technologies struggle with efficient compression of diverse video content, particularly in scenarios with varying textures and occlusions, as they often assume a single texture pattern within a block, leading to inefficiencies in encoding and decoding processes.
Implementing a method that segments video blocks into multiple segments, applying different prediction modes to each segment, such as intra block copy (IBC) for some samples and intra prediction using adjacent reconstructed samples for others, based on threshold calculations or segmentation maps derived from reference blocks.
Enhances video coding efficiency by adapting to varying textures and occlusions, improving compression performance and reducing hardware complexity by optimizing memory usage and prediction accuracy.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 17 / 332,227, filed May 27, 2021, which claims priority to U.S. Provisional Application No. 63 / 073,629, filed September 2, 2020, the disclosure of which is incorporated herein by reference in its entirety.
[0002] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to a set of advanced video coding techniques, and more specifically, to content-adaptive segmented prediction modes for image and video compression. [Background technology]
[0003] AOMedia Video (AV1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by the Alliance for Open Media (AOMedia), a consortium founded in 2015 that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors. Many of the components of the AV1 project are derived from previous research efforts by Alliance members. Individual contributors began working on experimental technology platforms several years ago, such as Xiph / Mozilla's Daala, which released code in 2010, Google's experimental VP9 evolution project VP10, announced on September 12, 2014, and Cisco's Thor, which was released on August 11, 2015. Built on the VP9 code base, AV1 incorporates additional technologies, some of which were developed in these experimental formats. The first version of the AV1 reference codec, version 0.1.0, was released on April 7, 2016. The Alliance announced the release of the AV1 Bitstream Specification, along with reference software-based encoders and decoders, on March 28, 2018. Validated version 1.0.0 of the specification was released on June 25, 2018. On January 8, 2019, the "AV1 Bitstream&Decoding Process Specification" was released, which is the validated version 1.0.0 with errata 1 of the specification. The AV1 Bitstream Specification includes reference video codecs. "AV1 Bitstream&Decoding Process Specification" (Version 1.0.0 with Errata 1), The Alliance for Open Media, January 8, 2019, is incorporated herein by reference in its entirety.
[0004] The High Efficiency Video Coding (HEVC) standard was jointly developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG) standardization bodies. To develop the HEVC standard, these two standardization bodies have been working together in a partnership known as the Joint Collaborative Team on Video Coding (JCT-VC). The first edition of the HEVC standard was completed in January 2013, resulting in a harmonized text published by both ITU-T and ISO / IEC. Additional work has since been done to extend the standard to support several additional application scenarios, including the use of an extended range of precision and enhanced color format support, scalable video coding, and 3-D / stereo / multi-view video coding. At ISO / IEC, the HEVC standard became MPEG-H Part 2 (ISO / IEC 23008-2), and at ITU-T, it became ITU-T Recommendation H.265.
[0005] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Since then, they have been studying the potential need for standardization of future video coding technologies that could significantly exceed HEVC in compression capabilities. In October 2017, they announced a joint Call for Proposal (CfP) for video compression with capabilities beyond HEVC. By February 15, 2018, 22 CfP responses for standard dynamic range (SDR), 12 CfP responses for high dynamic range (HDR), and 12 CfP responses for 360 video category were submitted, respectively. In April 2018, all received CfP responses were evaluated at the 122 MPEG / 10th Joint Video Exploration Team-Joint Video Expert Team (JVET) meeting. After careful evaluation, JVET officially started the standardization of the next generation video coding beyond HEVC, namely the so-called Versatile Video Coding (VVC).
[0006] Prior to HEVC, in December 2001, VCEG and the Moving Picture Experts Group (MPEG) ISO / IEC JTC 1 / SC 29 / WG 11 formed the Joint Video Team (JVT) and developed a charter to finalize a draft of a new video coding standard for submission for formal ratification as H.264 / AVC in March 2003. H.264 / AVC was approved by the ITU-T as Recommendation H.264 and by ISO / IEC as International Standard 14 496-10 (MPEG-4 part 10) Advanced Video Coding (AVC). Summary of the Invention [Means for solving the problem]
[0007] According to one or more embodiments, a method is provided for execution by at least one processor, the method including receiving a coded picture and reconstructing a current block of the coded picture, the reconstruction including segmenting samples of the current block into a plurality of segments including a first segment and a second segment, predicting the first segment of the current block of the coded picture using a first prediction mode, and predicting the second segment of the current block of the coded picture using a second prediction mode different from the first prediction mode.
[0008] According to one embodiment, the first prediction mode is an intra block copy (IBC) mode and the second prediction mode is an intra prediction mode that performs intra prediction using neighboring reconstructed samples.
[0009] According to one embodiment, the step of reconstructing the current block further comprises the step of signaling the first prediction mode.
[0010] According to one embodiment, reconstructing the current block comprises signaling the second prediction mode.
[0011] According to one embodiment, the segmentation comprises the steps of calculating at least one threshold based on the samples of the current block and segmenting the samples of the current block based on the at least one threshold.
[0012] According to one embodiment, the step of reconstructing the current block further comprises a step of segmenting samples of a reference block of the coded picture into a plurality of segments to obtain segmentation information of the reference block, and the step of segmenting samples of the current block comprises a step of mapping the segmentation information of the reference block to the current block.
[0013] According to one embodiment, predicting the first segment of the current block includes identifying the reference block based on a block vector before segmenting samples of the reference block and before segmenting samples of the current block, and mapping includes mapping segmentation information of the reference block to the current block based on the block vector.
[0014] According to one embodiment, the step of predicting the second segment of the current block includes a step of obtaining a prediction block predicting the second segment of the current block, and the step of predicting the first segment of the current block further includes a step of obtaining a combined prediction block by combining the prediction block of the current block with a segment of the reference block corresponding to the first segment of the current block using the segmentation information.
[0015] According to one embodiment, the segmentation information is a segmentation map.
[0016] According to one embodiment, a step of segmenting the samples of the current block is performed before predicting the first segment.
[0017] According to one or more embodiments, a system is provided, the system including at least one memory configured to store computer program code and at least one processor configured to access and operate as instructed by the computer program code. The computer program code includes a reconstruction code configured to cause the at least one processor to reconstruct a current block of a received coded picture. The reconstruction code includes a current block segmentation code configured to cause the at least one processor to segment samples of the current block into a plurality of segments including a first segment and a second segment, a first prediction code configured to cause the at least one processor to predict the first segment of the current block of the coded picture using a first prediction mode, and a second prediction code configured to cause the at least one processor to predict the second segment of the current block of the coded picture using a second prediction mode different from the first prediction mode.
[0018] According to one embodiment, the first prediction mode is an intra block copy (IBC) mode and the second prediction mode is an intra prediction mode that performs intra prediction using neighboring reconstructed samples.
[0019] According to one embodiment, the reconfiguration code further includes signaling code configured to cause the at least one processor to signal the first prediction mode.
[0020] According to one embodiment, the reconfiguration code further includes signaling code configured to cause the at least one processor to signal the second prediction mode.
[0021] According to one embodiment, the current block segmentation code is further configured to cause the at least one processor to calculate at least one threshold based on the samples of the current block and to segment the samples of the current block based on the at least one threshold.
[0022] According to one embodiment, the reconstruction code further includes a reference block segmentation code configured to cause the at least one processor to segment samples of a reference block of the coded picture into a plurality of segments to obtain segmentation information of the reference block, and the current block segmentation code configured to cause the at least one processor to map the segmentation information of the reference block to the current block.
[0023] According to one embodiment, the first predictive code is configured to cause the at least one processor to identify the reference block based on the block vector before the at least one processor segments samples of the reference block and samples of the current block, and the current block segmentation code is configured to cause the at least one processor to map segmentation information of the reference block to the current block based on the block vector.
[0024] According to one embodiment, the second predictive code is configured to cause the at least one processor to obtain a predictive block predicting a second segment of the current block, and the first predictive code is configured to cause the at least one processor to obtain a combined predictive block by combining the predictive block of the current block with a segment of the reference block corresponding to the first segment of the current block, using the segmentation information.
[0025] According to one embodiment, the segmentation information is a segmentation map.
[0026] According to one or more embodiments, a non-transitory computer-readable medium storing computer instructions is provided that, when executed by at least one processor, is configured to cause the at least one processor to reconstruct a current block of a received coded picture by segmenting samples of the current block into a plurality of segments including a first segment and a second segment, predicting the first segment of the current block of the coded picture using a first prediction mode, and predicting the second segment of the current block of the coded picture using a second prediction mode different from the first prediction mode. [Brief description of the drawings]
[0027] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.
[0028] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 3] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 5A] FIG. 1 illustrates a first exemplary partition structure for VP9. [Figure 5B] FIG. 13 illustrates a second exemplary partition structure for VP9. [Figure 5C] FIG. 13 is a diagram showing a third exemplary partition structure of VP9. [Figure 5D] FIG. 11 is a diagram showing a fourth exemplary partition structure of VP9. [Figure 6A] A diagram showing a first exemplary partition structure of AV1. [Figure 6B]A diagram showing a second exemplary partition structure of AV1. [Figure 6C] A diagram showing a third exemplary partition structure of AV1. [Figure 6D] A figure showing a fourth exemplary partition structure of AV1. [Figure 6E] A figure showing a fifth exemplary partition structure of AV1. [Figure 6F] A figure showing a sixth exemplary partition structure of AV1. [Figure 6G] A figure showing a seventh exemplary partition structure of AV1. [Figure 6H] A figure showing an eighth exemplary partition structure of AV1. [Figure 6I] A figure showing a ninth exemplary partition structure of AV1. [Figure 6J] A figure showing a tenth exemplary partition structure of AV1. [Figure 7A] FIG. 13 is a diagram showing a vertical binary split type in a multi-type tree structure. [Figure 7B] FIG. 13 is a diagram showing a horizontal binary split type in a multi-type tree structure. [Figure 7C] FIG. 13 is a diagram showing a vertical ternary split type in a multi-type tree structure. [Figure 7D] FIG. 13 is a diagram showing a horizontal ternary split type in a multi-type tree structure. [Figure 8] FIG. 13 is a diagram showing an example of a CTU divided into multiple CUs having a quadtree and nested multi-type tree coding block structure; [Figure 9] FIG. 1 shows eight nominal angles in AV1. [Figure 10] FIG. 2 shows a current block and samples. [Figure 11] FIG. 13 illustrates an example of intra block copy compensation. [Figure 12]FIG. 2 illustrates an example of a current processing block, a constrained immediate block, and an allowed predicted block for one embodiment of intra block copy prediction. [Figure 13A] FIG. 1 is a diagram of intra-picture block compensation using a search range of one CTU size according to one embodiment. [Figure 13B] FIG. 1 is a diagram of intra-picture block compensation using a search range of one CTU size according to one embodiment. [Figure 13C] FIG. 1 is a diagram of intra-picture block compensation using a search range of one CTU size according to one embodiment. [Figure 13D] FIG. 1 is a diagram of intra-picture block compensation using a search range of one CTU size according to one embodiment. [Figure 14] FIG. 2 illustrates an example of intra block copy applied to video content. [Figure 15] FIG. 2 illustrates a single block to which multiple prediction modes are applied, according to an embodiment of the present disclosure. [Figure 16] FIG. 1 illustrates a process according to an embodiment of the present disclosure. [Figure 17] FIG. 2 is a schematic diagram of a decoder according to one embodiment of the present disclosure. [Figure 18] FIG. 1 illustrates a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] Embodiments of the present disclosure include a video coding approach that can be applied on top of multiple existing video coding standards, including but not limited to H.264 / AVC, HEVC, VVC, and AV1.
[0030] In this disclosure, the term "block" may be interpreted as a prediction block, a coding block, or a coding unit (CU), where the term "block" may also be used to refer to a transform block.
[0031] FIG. 1 shows a simplified block diagram of a communication system (100) according to one embodiment of the present disclosure. The system (100) may include at least two terminals (110, 120) interconnected via a network (150). In the case of one-way data transmission, the first terminal (110) may code video data at a local location for transmission to the other terminal (120) via the network (150). The second terminal (120) may receive the other terminal's coded video data from the network (150), and may decode the coded data and display the restored video data. One-way data transmission may be common in media serving applications, etc.
[0032] 1 illustrates a second pair of terminals (130, 140) provided to support two-way transmission of coded video, such as might occur during a video conference. For two-way transmission of data, each terminal (130, 140) can code video data captured at a local location for transmission over a network (150) to the other terminal. Each terminal (130, 140) can also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.
[0033] In FIG. 1, the terminals (110-140) may be depicted as servers, personal computers, smartphones, and / or any other type of terminal. For example, the terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that convey coded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (150) may not be important to the operation of the present disclosure unless otherwise described herein below.
[0034] 2 illustrates the placement of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0035] As shown in FIG. 2, the streaming system (200) may include a capture subsystem (213) that may include a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera and may be configured to create an uncompressed video sample stream (202). The uncompressed video sample stream (202) may provide a high amount of data compared to an encoded video bitstream and may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204) may include a lower amount of data compared to the sample stream and may be stored on a streaming server (205) for future use. One or more streaming clients (206) may access the streaming server (205) to retrieve a video bitstream (209), which may be a copy of the encoded video bitstream (204).
[0036] In an embodiment, the streaming server (205) may also function as a Media-Aware Network Element (MANE). For example, the streaming server (205) may be configured to prune the encoded video bitstream (204) in order to tailor the potentially different bitstreams to one or more streaming clients (206). In an embodiment, a MANE may be provided separate from the streaming server (205) in the streaming system (200).
[0037] The streaming client (206) may include a video decoder (210) and a display (212). The video decoder (210) may, for example, decode a video bitstream (209) that is an incoming copy of the encoded video bitstream (204) and create an outgoing video sample stream (211) that may be rendered on a display (212) or another rendering device (not shown). In some streaming systems, the video bitstreams (204, 209) may be encoded according to a particular video coding / compression standard. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. The video coding standard under development is informally known as Versatile Video Coding (VVC). Embodiments of the present disclosure may be used in the context of VVC.
[0038] FIG. 3 shows an exemplary functional block diagram of a video decoder (210) attached to a display (212) according to one embodiment of the present invention.
[0039] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder / parser (320), a scaler / inverse transform unit (351), an intra prediction unit (352), a motion compensated prediction unit (353), an aggregator (355), a loop filter unit (356), a reference picture memory (357), and a current picture memory (). In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. The video decoder (210) may also be embodied partially or wholly in software running on one or more CPUs with associated memory.
[0040] In this and other embodiments, the receiver (310) can receive one or more coded video sequences to be decoded by the decoder (210), one coded video sequence at a time, where the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequences may be received from a channel (312), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (310) can receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to a respective using entity (not shown). The receiver (310) can separate the coded video sequences from the other data. To address network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter the "parser"). When the receiver 310 is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 315 may not be used or may be small. For use with best effort packet networks such as the Internet, a buffer 315 may be required, which may be relatively large and may be of adaptive size.
[0041] The video decoder (210) may include a parser (320) for reconstructing symbols (321) from the entropy coded video sequence. These categories of symbols include, for example, information used to manage the operation of the decoder (210) and potentially information for controlling a rendering device, such as a display (212), which may be coupled to the decoder as shown in FIG. 2. The control information for the rendering device may be in the form of, for example, a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (320) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (320) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (320) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.
[0042] The parser (320) may perform entropy decoding / parsing operations on the video sequence received from the buffer (315) to produce symbols (321).
[0043] The reconstruction of the symbols (321) may involve several different units, depending on the type of coded video picture or part thereof (e.g., inter-picture and intra-picture, inter-block and intra-block), and other factors. Which units are involved, and how, may be controlled by subgroup control information parsed from the coded video sequence by the parser (320). The flow of such subgroup control information between the parser (320) and the following units is not shown for clarity.
[0044] In addition to the functional blocks already mentioned, the decoder (210) may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may also be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate:
[0045] One unit may be a scalar / inverse transform unit (351), which may receive quantized transform coefficients as symbols (321) from the parser (320), as well as control information including the transform to use, block size, quantization coefficients, quantization scaling matrix, etc. The scalar / inverse transform unit (351) may output blocks containing sample values, which may be input to an aggregator (355).
[0046] In some cases, the output samples of the scaler / inverse transform unit (351) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory (358). The aggregator (355) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351).
[0047] In other cases, the output samples of the scaler / inverse transform unit (351) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (353) may access the reference picture memory (357) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (321) associated with the block, these samples may be added by the aggregator (355) to the output of the scaler / inverse transform unit (351) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (357) from which the motion compensated prediction unit (353) fetches the prediction samples may be controlled by a motion vector. The motion vector may be available to the motion compensated prediction unit (353) in the form of a symbol (321), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values fetched from the reference picture memory (357) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.
[0048] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in a loop filter unit (356). Video compression techniques may include in-loop filter techniques, which are controlled by parameters contained in the coded video bitstream and made available to the loop filter unit (356) as symbols (321) from the parser (320), but which may also be responsive to meta-information obtained during decoding of a previous (in decode order) portion of the coded picture or coded video sequence, or to previously reconstructed and loop filtered sample values.
[0049] The output of the loop filter unit (356) may be a sample stream that can be output to a rendering device such as a display (212), as well as stored in a reference picture memory (357) for use in future inter-picture prediction.
[0050] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (320)), the current reference picture can become part of the reference picture memory (357), and a new current picture memory can be reallocated before beginning reconstruction of the next coded picture.
[0051] The video decoder (210) may perform decoding operations according to a given video compression technique, which may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to a syntax specified by the video compression technique or standard being used, in the sense of conforming to the syntax of the video compression technique or standard as specified in the video compression technique document or standard, particularly the profile documents therein. Also, to conform to some video compression techniques or standards, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.
[0052] In one embodiment, the receiver (310) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.
[0053] FIG. 4 illustrates an exemplary functional block diagram of a video encoder (203) associated with a video source (201) according to one embodiment of the present disclosure.
[0054] The video encoder (203) may include, for example, an encoding source coder (430), a coding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter (440), an entropy coder (445), a controller (450), and a channel (460).
[0055] The video encoder (203) can receive video samples from a video source (201) (not part of the encoder) that can capture video images to be encoded by the encoder (203).
[0056] The video source (201) may provide a source video sequence to be coded by the encoder (203) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCB 4:2:0, Y CrCB 4:4:4). In a media serving system, the video source (201) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of separate pictures that give motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. In the following, the description will focus on samples.
[0057] According to one embodiment, the encoder (203) can code and compress pictures of a source video sequence into a coded video sequence (443) in real-time or under any other time constraint required by the application. Applying an appropriate coding rate is one function of the controller (450). The controller (450) can also control and be operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (450) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller (450) as they may be relevant for a video encoder (203) optimized for a particular system design.
[0058] Some video encoders operate in a "coding loop" that is easily recognizable to those skilled in the art. As an oversimplified explanation, the coding loop can consist of an encoding part of a source coder (430) (responsible for creating symbols based on the input picture to be coded and reference pictures) and a (local) decoder (433) embedded in the encoder (203) that reconstructs the symbols to create sample data, which a (remote) decoder also creates if the compression between the symbols and the coded video bitstream is lossless for a particular video compression technique. That reconstructed sample stream can be input to a reference picture memory (434). Since the decoding of the symbol stream gives bit-accurate results regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, for example due to channel errors) is known to those skilled in the art.
[0059] The operation of the "local" decoder (433) may be the same as the operation of the "remote" decoder (210) already described in detail above in connection with Figure 3. However, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (445) and parser (320) may be lossless, the entropy decoding portion of the decoder (210), including the channel (312), receiver (310), buffer (315), and parser (320), may not be fully implemented in the local decoder (433).
[0060] An observation that can be made at this point is that any decoder technology, except for analysis / entropy decoding, present in a decoder may need to be present in a corresponding encoder in substantially identical functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of the encoder technology may be omitted, since it may be the inverse of the decoder technology described generically. Only certain areas require more detailed description, which is provided below.
[0061] As part of its operation, the source coder (430) can perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as “reference frames.” In this manner, the coding engine (432) codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.
[0062] The local video decoder (433) can decode the coded video data of a frame that may be designated as a reference frame based on the symbols created by the source coder (430). The operation of the coding engine (432) can advantageously be a lossy process. If the coded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can be a replica of the source video sequence, usually with some errors. The local video decoder (433) can replicate the decoding process that may be performed by the video decoder on the reference frame and store the reconstructed reference frame in the reference picture memory (434). In this way, the video encoder (203) can locally store a copy of the reconstructed reference frame that has common content as the reconstructed reference frame obtained by the far-end video decoder (free of transmission errors).
[0063] The predictor (435) can perform a prediction search for the coding engine (432). That is, for a new frame to be coded, the predictor (435) can search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that can serve as suitable prediction references for the new picture. The predictor (435) can operate on a pixel block-by-sample block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input picture may have prediction references extracted from multiple reference pictures stored in the reference picture memory (434).
[0064] The controller (450) can manage the coding operations of the video coder (430), including, for example, setting the parameters and subgroup parameters used to encode the video data.
[0065] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (445), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, e.g., Huffman coding, variable length coding, arithmetic coding, etc.
[0066] The transmitter (440) can buffer the coded video sequence produced by the entropy coder (445) for transmission over a communication channel (460), which can be a software / hardware link to a storage device that stores the encoded video data. The transmitter (440) can merge the coded video data from the video coder (430) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).
[0067] The controller (450) can manage the operation of the encoder (203). During coding, the controller (450) can assign a particular coded picture type to each coded picture, which can affect the coding technique that can be applied to the respective picture. For example, pictures may often be assigned as intra pictures (I pictures), predicted pictures (P pictures), or bidirectionally predicted pictures (B pictures).
[0068] An intra picture (I-picture) may be one that can be coded and decoded without using any other frame in a sequence as a source of prediction. In some video codecs, various types of intra pictures are possible, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.
[0069] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values of each block.
[0070] Bidirectionally predicted pictures (B-pictures) may be those that can be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.
[0071] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be non-predictively coded via spatial prediction with reference to one previously coded reference picture, or via temporal prediction. Blocks of a B-picture may be non-predictively coded via spatial prediction with reference to one or two previously coded reference pictures, or via temporal prediction.
[0072] The video coder (203) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operation, the video coder (203) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.
[0073] In one embodiment, the transmitter (440) can transmit additional data along with the encoded video. The video coder (430) can include such data as part of the coded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.
[0074] [Coding block division in VP9 and AV1]
[0075] Referring to the partition structures (502)-(508) in Figures 5A-5D, VP9 uses four types of partition trees, from the 64x64 level down to the 4x4 level, with some additional restrictions on blocks 8x8. Note that the partitions designated as R in Figure 5D refer to recursion in that the same partition tree is repeated at lower scales until the lowest 4x4 level is reached.
[0076] With reference to the partition structures (511)-(520) in Figures 6A-6J, AV1 not only extends the partition tree to 10 types of structures, but also increases the maximum size (called superblock in VP9 / AV1 terminology) such as from 128x128. Note that this includes the 4:1 / 1:4 rectangular partitions that did not exist in VP9. The partition type with three subpartitions as shown in Figures 6C-6F is called a "T-type" partition. The rectangular partitions may not need to be further divided. In addition to the coding block size, the coding tree depth can be defined to indicate the division depth from the root node. Specifically, for example, for a 128x128 root node, the coding tree depth is set to 0, and after one further division of the tree block, the coding tree depth is increased by 1.
[0077] Instead of enforcing a fixed transform unit size as in VP9, AV1 allows luma coding blocks to be divided into transform units of multiple sizes, which can be represented by recursive partitions going up to two levels down. To incorporate AV1's extended coding block partitions, square, 2:1 / 1:2, and 4:1 / 1:4 transform sizes from 4x4 to 64x64 may be supported. For chroma blocks, only the largest possible transform units may be allowed.
[0078] [Block division in HEVC]
[0079] In HEVC, a coding tree unit (CTU) may be divided into coding units (CUs) by using a quadtree (QT) structure, denoted as a coding tree, to adapt to various local characteristics. The decision of whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction can be made at the CU level. Each CU may be further divided into one, two, or four prediction units (PUs) according to the PU partition type. Within one PU, the same prediction process may be applied, and related information is transmitted to the decoder on a PU basis. After applying the prediction process based on the PU partition type to obtain the residual block, the CU may be divided into transform units (TUs) according to another quadtree structure, such as the coding tree of the CU. One of the important features of the HEVC structure is that it has multiple partition concepts, including CUs, PUs, and TUs. In HEVC, a CU or TU can only take a square shape, while a PU can take a square or rectangular shape for an inter-prediction block. In HEVC, a coding block can be further divided into four square sub-blocks, and a transform is performed on each sub-block (i.e., TU). Each TU can be further divided recursively (using a quad-tree partition) into smaller TUs, which are called Residual Quad-Tree (RQT).
[0080] At picture boundaries, HEVC uses implicit quad-tree partitioning such that blocks undergo quad-tree partitioning until their size fits the picture boundary.
[0081] [Quadtree with nested multi-type tree coding block structure in VVC]
[0082] In VVC, a quadtree with nested multi-type trees using binary and ternary split segmentation structures replaces the concept of multiple partition unit types. That is, VVC does not include the separation of CU, PU, and TU concepts, except as required for CUs whose size is too large for the maximum transform length, and provides more flexibility in CU partition shapes. In the coding tree structure, CUs can have either square or rectangular shapes. The coding tree unit (CTU) is first partitioned by a quadtree (aka quadtree) structure. The leaf nodes of the quadtree can then be further partitioned by a multi-type tree structure. As shown in diagrams (532), (534), (536), and (538) of Figures 7A to 7D, the multi-type tree structure has four split types: vertical binary split (SPLIT_BT_VER) shown in Figure 7A, horizontal binary split (SPLIT_BT_HOR) shown in Figure 7B, vertical ternary split (SPLIT_TT_VER) shown in Figure 7C, and horizontal ternary split (SPLIT_TT_HOR) shown in Figure 7D. The leaf nodes of the multi-type tree may be called coding units (CUs), and as long as the CUs are not too large for the maximum transform length, this segmentation can be used for prediction and transform processing without further splitting. This means that in most cases, CUs, PUs, and TUs have the same block size in a quad tree with nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of the CU. An example of block split for one CTU is shown in Figure 8. 8 shows a CTU (540) divided into multiple CUs with a quad-tree and nested multi-type tree coding block structure, where the bold edges represent the quad-tree partitions and the dashed edges represent the multi-type tree partitions. The quad-tree with nested multi-type tree partitions provides a content adaptive coding tree structure composed of CUs.
[0083] In VVC, the maximum supported luma transform size is 64 x 64, and the maximum supported chroma transform size is 32 x 32. If the width or height of the CB is larger than the maximum transform width or height, the CB may be automatically split in the horizontal and / or vertical directions to meet the transform size limit in that direction.
[0084] In VTM7, the coding tree scheme supports the ability for luma and chroma to have separate block tree structures. For P and B slices, the luma and chroma CTBs in one CTU may need to share the same coding tree structure. However, for I slices, luma and chroma can have separate block tree structures. When the separate block tree mode is applied, the luma CTB is divided into CUs by one coding tree structure, and the chroma CTB is divided into chroma CUs by another coding tree structure. This means that a CU in an I slice may consist of a coding block for the luma component or a coding block for two chroma components, and a CU in a P or B slice may consist of coding blocks for all three color components, unless the video is monochrome.
[0085] [Directional intra prediction in AV1]
[0086] VP9 supports eight directional modes, corresponding to angles from 45 degrees to 207 degrees. To exploit more diverse spatial redundancy in the directional texture, in AV1, the directional intra mode is extended to a finer set of angles. The original eight angles are slightly modified to become nominal angles, and these eight nominal angles are named V_PRED (542), H_PRED (543), D45_PRED (544), D135_PRED (545), D113_PRED (546), D157_PRED (547), D203_PRED (548), and D67_PRED (549), shown in FIG. 9 for the current block (541). For each nominal angle, there are seven finer angles, so there are a total of 56 directional angles in AV1. The prediction angle is expressed as the nominal intra angle plus the angle delta, where the angle delta is -3 to 3 times the step size of 3 degrees. To implement directional prediction modes in AV1 in a generic way, all 56 directional intra prediction modes in AV1 are implemented using a joint directional predictor that projects each pixel to a reference sub-pixel location and interpolates the reference pixels with a 2-tap bilinear filter.
[0087] [Non-directional smooth intra predictor in AV1]
[0088] There are five non-directional smooth intra prediction modes in AV1: DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H. For DC prediction, the average of the left and top neighboring samples is used as the predictor of the block to be predicted. For PAETH predictor, the top, left, and top-left reference samples are fetched first, and then the closest value to (top+left-top-left) is set as the predictor of the pixel to be predicted. Figure 10 shows the location of the top sample (554), left sample (556), and top-left sample (558) of the current pixel (552) in the current block (550). For SMOOTH, SMOOTH_V, and SMOOTH_H modes, the current block (550) is predicted using quadratic interpolation in the vertical or horizontal direction, or an average in both directions.
[0089] [Saturation predicted from luminance]
[0090] For the chroma component, in addition to the 56 directional and 5 non-directional modes, chroma from luma (CfL) is a chroma-only intra prediction mode that models chroma pixels as linear functions of the simultaneously reconstructed luma pixels. CfL prediction can be expressed as shown in Equation 1 below.
[0091]
number
[0092] Here, L AC where α denotes the AC contribution of the luma component, α denotes the parameter of the linear model, and DC denotes the DC contribution of the chroma component. Specifically, the reconstructed luma pixel is subsampled to the chroma resolution, and then the mean is subtracted to form the AC contribution. To approximate the chroma AC component from the AC contribution, instead of requiring the decoder to calculate the scaling parameters as in some background art, AV1 CfL determines the parameters α based on the original chroma pixels and signals them in the bitstream. This reduces the decoder complexity and results in a more accurate prediction. For the DC contribution of the chroma component, it can be calculated using the intra DC mode, which is sufficient for most chroma content and has a mature and fast implementation.
[0093] When signaling chroma intra-prediction modes, first, eight nominal directional modes, five non-directional modes, and a CfL mode can be signaled. The context for signaling these modes may depend on the corresponding luma mode of the top-left position of the current block. Then, if the current chroma mode is a directional mode, one additional flag can be signaled to indicate the delta angle relative to the nominal angle.
[0094] [Intrablock copy]
[0095] Block-based compensation from different pictures may be called motion compensation. Similarly, block compensation may be done from previously reconstructed regions in the same picture. This may be called intra-picture block compensation, current picture referencing (CPR), or intra-block copy (IBC). The displacement vector indicating the offset between the current block and the reference block may be called block vector (BV). Unlike the motion vector in motion compensation, which can be any value (positive or negative in either x or y direction), the block vector has some constraints to ensure that the reference block it is directed to is available and already reconstructed. Also, some reference regions that are tile boundaries or wavefront ladder shape boundaries may be excluded to allow for parallel processing.
[0096] The coding of block vectors may be explicit or implicit. In explicit modes (called AMVP modes in inter-coding), the difference between a block vector and its predictor may be signaled, while in implicit modes, a block vector may be reconstructed purely from its predictor in a manner similar to motion vectors in merge modes. In some implementations, the resolution of block vectors is limited to integer positions, while in other systems they may be allowed to point to fractional positions.
[0097] The use of intra block copy at the block level can be signaled using a block level flag (e.g., IBC flag). In one embodiment, this flag is signaled when the current block is not coded in merge mode. Alternatively, it can be signaled by a reference index approach. This can be done by treating the current decoded picture as a reference picture. In HEVC SCC, such a reference picture can be placed at the last position of the list. This special reference picture may also be managed together with other temporary reference pictures in the decoded picture buffer (DPB).
[0098] There are also some variants of intra block copy, such as treating intra block copy as a third mode different from either intra prediction or inter prediction mode. This separates block vector prediction in merge and AMVP modes from the normal inter modes. For example, a separate merge candidate list is defined for intra block copy mode, where all entries in the list are block vectors. Similarly, the block vector prediction list in intra block copy AMVP mode consists of block vectors only. A general rule that applies to both lists is that, in terms of the candidate derivation process, they may follow the same logic as the inter merge candidate list or the AMVP predictor list. For example, five spatially adjacent locations in HEVC or VVC inter merge modes are accessed for intra block copy to derive its own merge candidate list. An example of intra block copy is shown in FIG. 11, which shows a current picture (560) in which a current block (562) is predicted based on a reference block (564) whose location is indicated by a block vector (566).
[0099] IBC (also called Intra BC) is very effective for coding screen content, but it also brings many difficulties to hardware design. To facilitate hardware design, AV1 may adopt the following modifications.
[0100] Modification 1: When IBC is permitted, the loop filter becomes invalid, which includes the deblocking filter, CDEF (Constrained Directional Enhancement Filter), and loop restoration. This allows the picture buffer of the reconstructed samples to be shared between IBC and inter prediction.
[0101] Modification 2: To facilitate parallel decoding, prediction cannot exceed a restricted area. For one superblock, if the coordinates of the upper-left position are (x0, y0), IBC can access prediction at position (x, y) if y < y0 and x < x0 + 2 * (y0 - y)3.
[0102] To enable hardware write-back delay, IBC prediction may not be able to access the immediately reconstructed area. The restricted immediately reconstructed area can be 1 to n superblocks. Therefore, in addition to Modification 2, for one superblock, if the coordinates of the upper-left position are (x0, y0), IBC can access prediction at position (x, y) if y < y0 and x < x0 + 2 * (y0 - y) - D, where D indicates the restricted immediately reconstructed area. If D is one superblock, the prediction area can be as shown in Figure 12. In Figure 12, multiple current processing blocks (572) are shown in diagonal stripes, multiple restricted immediate blocks (574) are shown in cross-hatching, and multiple permitted prediction blocks (576) are shown in a dark pattern.
[0103] In VVC, the search range of IBC mode may be constrained to be within the current CTU. The effective memory requirement for storing reference samples of IBC mode may be one CTU-sized sample. Considering the existing reference sample memory for storing reconstructed samples in the current 64x64 region, three more 64x64-sized reference sample memories may be required. Based on this fact, the method of the embodiment of the present disclosure can extend the effective search range of IBC mode to a part of the left CTU while keeping the total memory requirement for storing reference pixels unchanged (one CTU-sized, four 64x64 reference sample memories in total). An example of such a memory reuse mechanism is shown in Figures 13A-13D, where the diagonally striped blocks are the current coding region, the samples in the dotted pattern box are the coded samples, and the strikethrough regions (marked with "X") are not available for reference because they have been replaced in the reference sample memory by the coding region of the current CTU.
[0104] For example, in Figure 13A, reference sample (612a), marked with an X, is unavailable to current sample (611). Similarly, in Figure 13B, reference samples (622a) and (622b) are unavailable to current sample (621). In Figure 13C, reference samples (632a), (632b), and (632c) are unavailable to current sample 631, and in Figure 13D, reference samples (642a), (642b), (642c), and (642d) are unavailable to current sample (641).
[0105] [Comparison technique coding mode issue]
[0106] IBC and other coding modes may assume a single texture pattern within one block. However, in many typical video contents, objects have mutual occlusions. For example, stationary text or logos float above main video content that has a completely different texture pattern or motion. For example, as shown in the video frame (700) shown in FIG. 14, we can see that there are two matching characters "o" (702) and "E" (704). However, these matching characters are located above different backgrounds and IBC cannot capture both text and background efficiently because it assumes that all samples share the same block vector.
[0107] [Exemplary aspects of the present disclosure]
[0108] The embodiments of the present disclosure may be used separately or in any order in combination. Furthermore, each embodiment (e.g., the methods, encoders, and decoders) may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.
[0109] Embodiments of the present disclosure may incorporate any number of the aspects described above. Embodiments of the present disclosure may also incorporate one or more of the aspects described below to solve the above problems and / or other problems.
[0110] A. First Aspect
[0111] According to an embodiment, instead of applying a single prediction mode to a block, multiple prediction modes can be applied to a single block, and a segmentation process can be performed to divide samples in a single block into different segments, each segment being associated with one of the selected prediction modes. Then, for samples belonging to a first segment, the associated first prediction mode is used to generate predicted samples of the first segment, for samples belonging to a second segment, the associated second prediction mode is used to generate predicted samples of the second segment, and so on. Figure 15 shows an example of such a process, where samples of a current block (722) are classified (segmented) into a first segment (A) and a second segment (B). Samples of the first segment (A) can be predicted by applying an IBC mode that uses a block vector (726) to fetch a predicted block (724), and samples of the second segment (B) can be predicted by applying a normal intra prediction mode that uses neighboring reconstructed samples to generate predicted values.
[0112] According to one embodiment, multiple prediction modes can be signaled for a block (e.g., the current block (722)). In one example, the multiple prediction modes include an IBC prediction mode (including block vectors) and an intra prediction mode (including an intra prediction mode or direction) that performs intra prediction using neighboring reconstructed samples.
[0113] According to one embodiment, IBC is applied to a single block (e.g., the current block (722)) together with one or more default intra prediction modes, and only mode information related to IBC (e.g., block vectors) is signaled. For samples within a single block, IBC modes are applied to some samples, and one or more default intra prediction modes are applied to the remaining samples. The default intra prediction modes include, but are not limited to, DC, Planar, SMOOTH, SMOOTH_H, SMOOTH_V, Paeth, and Plane modes.
[0114] According to one embodiment, IBC is applied to a single block (e.g., the current block (722)) together with one or more default intra prediction modes, and mode information (e.g., block vectors) related to IBC is estimated. For samples in a single block, IBC modes are applied to some samples, and one or more default intra prediction modes are applied to the remaining samples. The estimated IBC information (block vectors) can be derived (e.g., by the decoder) from a block vector predictor list such as that used to generate the merge list. If there are multiple possible block vector candidates, an index value can be signaled to select a block vector from the block vector predictor list. For example, the index value can be received by the decoder from the encoder. Otherwise, the encoder does not send an index and the decoder does not receive an index, and a default selection can be made. In this case, for example, it may be assumed (e.g., by the decoder) that the first block vector candidate in the predictor list is used. Default intra prediction modes include, but are not limited to, DC, Planar, SMOOTH, SMOOTH_H, SMOOTH_V, Paeth, and Plane modes.
[0115] An embodiment (e.g., a decoder) may perform the segmentation process in different ways. According to an embodiment, the segmentation process may be performed on the current block (722). In one example, first a threshold value of samples in a single current block (722) is calculated. According to the threshold value, the samples are classified into different segments by comparing their values to the threshold value. The threshold value includes, but is not limited to, the mean, the median, or other values derived from the mean or median. In another example, a histogram of the distribution of the values of the samples is calculated. According to the histogram, one or more threshold values on the sample counts are derived, and the samples are classified into different segments by comparing their sample counts in the histogram with the threshold value. In another example, a convolutional neural network is applied using the samples of the current block (722) as input, and the output of the convolutional neural network is the segment index of each sample. In another example, an edge detection method may be applied to the block. The segmentation may be performed according to possible edge boundaries.
[0116] According to one or more embodiments, referring to FIG. 16, segmentation is first applied to the reference block (730), and then the segmentation information is further mapped to the current block (740) to derive a sample segmentation of the current block (740). In one example, a block vector is used to identify the reference block (730) in the neighboring reconstructed region, segmentation is applied to the reference block (730), and then the segmentation information is mapped to the current block (740) using the block vector. Referring to FIG. 16, the reference block (730) for the current block (740) is derived using a block vector in the same picture. Then, a segmentation process is applied to the reference block (730), for example, dividing the reference block (730) into multiple parts (e.g., a first segment (732) and a second segment (734)). In the example shown in FIG. 16, the first segment (732) includes the area outside the image of the bird, and the second segment (734) includes the image of the bird. In the segmentation process, a segmentation map (736) is obtained that indicates the segment index of each sample. Referring to FIG. 16, the black portion of the segmentation map (736) corresponds to the first segment (732), and the white portion of the segmentation map (736) corresponds to the second segment (734). The segmentation map (736) is then mapped to the current block (740) using the block vector to derive segmentation information for the current block (740). Based on the segmentation information, the current block (740) is segmented into multiple segments (e.g., two segments corresponding to the first segment (732) and the second segment (734). For example, the first segment of the current block (740) includes the area outside the image of the bird, and the second segment of the current block (740) includes the image of the bird. A conventional prediction process is then applied to the current block (740) to derive a conventional prediction block (750). For example, a conventional prediction process can be applied to a first segment of the current block (740) that includes the area outside the image of the bird, but not to a second segment of the current block (740).Finally, the conventional prediction block (750) and the second segment (734) from the reference block (730) are combined using the segmentation map (736) to derive a combined prediction block (760).
[0117] For each segment, prediction samples may be generated within the segment boundaries. After predicting the samples for each possible segment, the predicted segments can be merged together to form a final predictor for the current block (740). This prediction block, along with a possible decoded residual block signal, can form a reconstructed block (before loop filtering) by adding the two together.
[0118] According to an embodiment, the identification of the reference block (730), the prediction using a second segment (734) of the reference block (730), and the combination of the second segment (734) with a conventional prediction block (750) can be considered as part of a prediction mode other than the conventional prediction process used to obtain the combined prediction block (760).
[0119] B. Second Aspect
[0120] According to the example described above with respect to the first aspect, for a segmented prediction mode, it may be necessary to specify a block vector of an intra block copy mode and an intra prediction direction (or a non-directional mode option). A non-limiting example process of selection that may be performed (e.g., by a decoder) is described below.
[0121] In one embodiment, the IBC mode block vector is signaled to indicate the reference block location. This signaling can be based on a vector prediction and a coded difference. The coded difference can be the difference between the block vector and its predictor and may be received by the decoder from the encoder.
[0122] In another embodiment, a block vector candidate list is obtained and one candidate is selected from the list by signaling the index of an entry in the list, and the selected candidate is used as the block vector without coding the difference.
[0123] In another embodiment, the decoder obtains a block vector candidate list and selects one candidate from the list by selecting a default entry position in the list. For example, the decoder may select the first entry in the list as the default entry position. In this case, there may be no need to signal an index. The selected candidate is used as the block vector without coding the difference.
[0124] In another embodiment, an intra mode is signaled to indicate an intra prediction method for one of the segments. The signaling may be based on a separate set of intra prediction modes. In one example, normal intra prediction may have M prediction modes, while the separate set of intra prediction modes may be N modes. N may not be equal to M. A Most Probable Mode (MPM) list may be obtained to predict the selected intra mode. In one example, the N intra prediction modes may be a subset of the full set of M normal intra prediction modes.
[0125] In another embodiment, an intra-prediction mode candidate list is obtained and one candidate is selected from the list by signaling an index of an entry in the list, and the selected candidate is used as the intra-prediction mode without coding the difference.
[0126] In another embodiment, the decoder obtains an intra-prediction mode candidate list and selects one candidate from the list by selecting a default entry position in the list. For example, the decoder may select the first entry in the list as the default entry position. In this case, there may be no need to signal an index. The selected candidate is used as the intra-prediction mode without coding the difference.
[0127] According to an embodiment, at least one processor and a memory storing computer program instructions may be provided. The computer program instructions, when executed by the at least one processor, may implement an encoder or a decoder and may perform any number of functions described in this disclosure. For example, with reference to FIG. 17, the at least one processor may implement a decoder (800). The computer program instructions may include, for example, a reconstruction code (810) configured to cause the at least one processor to reconstruct a current block of a coded picture received (e.g., from an encoder). The reconstruction code (810) may include, for example, a signaling or estimation code (820), a segmentation code (830), a first predictive code (840), and / or a second predictive code (850).
[0128] According to embodiments of the present disclosure, the signaling or estimation code (820) may be configured to cause at least one processor to signal values of syntax elements (e.g., flags, indices, etc.), which may be obtained by the decoder 800 from an encoder or in other manner. Alternatively or additionally, according to embodiments of the present disclosure, the signaling or estimation code (820) may be configured to cause at least one processor to estimate information (e.g., a default intra-prediction mode and / or mode information related to IBC).
[0129] According to an embodiment of the present disclosure, the segmentation code (830) may include a current block segmentation code configured to cause at least one processor to segment samples of a current block into a plurality of segments, including a first segment and a second segment. For example, the current block segmentation code may be configured to cause at least one processor to calculate at least one threshold based on samples of the current block and segment samples of the current block based on the at least one threshold. In such cases and other cases, segmentation of samples of the current block may be performed before applying the first prediction mode. Alternatively, referring to the description of FIG. 16, the current block segmentation code may be configured to cause at least one processor to segment samples of the current block by mapping segmentation information (e.g., a segmentation map) of the obtained reference block to the current block. In such a case, the segmentation code (830) may further include a reference block segmentation code configured to cause at least one processor to segment samples of a reference block of the coded picture into a plurality of segments to obtain segmentation information of the reference block. In such cases and in other cases, the segmentation of the samples may be performed, at least in part, simultaneously with the application of the first prediction mode.
[0130] According to an embodiment of the present disclosure, the first predictive code (840) may be configured to cause the at least one processor to apply a first prediction mode for prediction of a first segment of a current block of a coded picture. According to an embodiment, the first prediction mode may be an IBC mode. According to an embodiment with reference to FIG. 16, the first predictive code (840) may be configured to cause the at least one processor to identify a reference block based on the block vector before segmenting samples of the reference block and before segmenting samples of the current block. In such a case, the first predictive code (840) may be configured to cause the at least one processor to obtain a combined predictive block by combining a predictive block of the current block with a segment of the reference block corresponding to the first segment of the current block using the segmentation information.
[0131] The second prediction code (850) may be configured to cause the at least one processor to apply a second prediction mode, different from the first prediction mode, for prediction of a second segment of a current block of the coded picture. According to an embodiment, the second prediction mode may be an intra prediction mode performing intra prediction using neighboring reconstructed samples. According to an embodiment with reference to FIG. 16, the second prediction code (850) may be configured to cause the at least one processor to obtain a prediction block predicting the second segment of the current block, the prediction block being used to obtain a joint prediction block.
[0132] According to an embodiment, an encoder-side process corresponding to the above process may be implemented by an encoding code for encoding a picture, as would be understood by one of ordinary skill in the art based on the above description.
[0133] The techniques of the above-described embodiments of the present disclosure can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 18 shows a computer system (900) suitable for implementing embodiments of the disclosed subject matter.
[0134] Computer software may be coded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly, or through interpretation, microcode execution, etc.
[0135] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.
[0136] 18 for computer system (900) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (900).
[0137] The computer system (900) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).
[0138] The input human interface devices may include one or more of a keyboard (901), a mouse (902), a trackpad (903), a touch screen (910), a data glove, a joystick (905), a microphone (906), a scanner (907), and a camera (908) (only one of each is shown).
[0139] The computer system (900) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (910), data gloves, or joystick (905), although there may also be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (speakers (909), headphones (not shown), etc.), visual output devices (screens (910) including CRT screens, LCD screens, plasma screens, OLED screens, etc., each of which may or may not have touch screen input capability, each of which may or may not have haptic feedback capability, some of which may output two-dimensional visual output or three-dimensional or higher output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).
[0140] The computer system (900) may also include human accessible storage and associated media such as optical media or similar media (921), including CD / DVD ROM / RW (920) with CDs / DVDs, thumb drives (922), removable hard drives or solid state drives (923), legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices (not shown) such as security dongles.
[0141] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.
[0142] The computer system (900) may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, automotive and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial networks including CANBus, etc. Certain networks generally require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (949) (e.g., a USB port of the computer system (900)), while other networks are generally integrated into the core of the computer system 900 by connecting to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (900) may communicate with other entities. Such communications may be, for example, one-way to other computer systems using local or wide area digital networks, receive only (e.g., broadcast TV), transmit only one way (e.g., CANbus to a particular CANbus device), or bidirectional. Such communications may include communications to a cloud computing environment (955). As noted above, specific protocols and protocol stacks may be used with each of these networks and network interfaces.
[0143] The aforementioned human interface devices, human accessible storage, and network interfaces (954) may be connected to the core (940) of the computer system (900).
[0144] The cores (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), specialized programmable processing units (943) in the form of field programmable gate areas (FPGAs), hardware accelerators (944) for specific tasks, etc. These devices may be connected via a system bus (948) along with read-only memory (ROM) (945), random access memory (946), internal mass storage devices (947) such as internal hard drives or SSDs that are not user accessible. In some computer systems, the system bus (948) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected directly to the core's system bus (948) or via a peripheral bus (949). Peripheral bus architectures include PCI, USB, etc. A graphics adapter (950) may be included in the core (940).
[0145] The CPU (941), GPU (942), FPGA (943), and accelerator (944) may execute certain instructions, which may combine to constitute the aforementioned computer code. The computer code may be stored in a ROM (945) or a RAM (946). Transient data may also be stored in the RAM (946), while persistent data may be stored, for example, in an internal mass storage device (947). A cache memory, which may be closely associated with one or more of the CPU (941), GPU (942), mass storage device (947), ROM (945), RAM (946), etc., may be used to enable fast storage and retrieval from any memory device.
[0146] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.
[0147] As an example, and not by way of limitation, the architecture, and in particular the computer system (900) having the core (940), may provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage devices introduced above, as well as specific storage devices of the core (940) that are non-transitory in nature, such as the core internal mass storage (947) or ROM (945). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (940). The computer-readable media may include one or more memory devices or chips, as appropriate. The software may cause the core (940), and in particular the processor therein (including CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (946) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (944)) that may work in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, as appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.
[0148] While this disclosure has described several non-limiting exemplary embodiments, there are alterations, modifications, and various substitute equivalents that are within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]
[0149] 800 Decoder 810 Reconstruction Code 820 Signalling or Presumption Code 830 Segmentation Code 840 First Prediction Code 850 Second Prediction Code 948 System Bus 950 Graphics Adapter 954 Network Interface
Claims
1. A method executed by at least one processor implementing an encoder, comprising: receiving a picture; encoding the picture; and the encoding step includes signaling a current block of the picture based on a block vector of an intra block copy (IBC) mode and at least one of an intra prediction direction and a non-directional mode option; The step of encoding the picture includes: signaling the block vector of the IBC mode; selecting a block vector candidate list by signaling an index of an entry in the block vector candidate list; selecting a candidate from the block vector candidate list based on a default entry position of the block vector candidate list; signaling an intra mode as indicating intra prediction for at least one segment of the current block; and Establishing a set of intra-prediction mode candidate lists The method includes at least one of the following:
2. The method described in claim 1, wherein signaling the block vector of the IBC mode is based on vector prediction.
3. The method of claim 1, wherein the candidates in the block vector candidate list are the block vectors.
4. The method of claim 1, wherein the default entry position of the block vector candidate list is the first entry of the block vector candidate list.
5. The method of claim 1, wherein signaling the intra mode is based on a most probable mode list.
6. The method of claim 1, wherein a candidate is selected from the intra-prediction mode candidate list by signaling an index of an entry in the intra-prediction mode candidate list.
7. The method of claim 1, wherein a candidate is selected from the intra-prediction mode candidate list by selecting a default entry position in the intra-prediction mode candidate list.
8. A system comprising: at least one memory configured to store computer program code; at least one processor configured to access said computer program code and to act as directed by said computer program code; the computer program code comprising: receiving code configured to cause the at least one processor to receive a picture; encoding code configured to cause the at least one processor to encode the picture; and Including, The encoding includes signaling a current block of the picture based on a block vector of an intra block copy (IBC) mode and at least one of an intra prediction direction and a non-directional mode option; encoding the picture includes: signaling the block vectors of the IBC mode; selecting a block vector candidate list by signaling an index of an entry in the block vector candidate list; selecting a candidate from the block vector candidate list based on a default entry position of the block vector candidate list; signaling an intra mode as indicating intra prediction for at least one segment of the current block; and Establishing a set of intra-prediction mode candidate lists 1. A system comprising:
9. The system of claim 8, wherein signaling the block vector of the IBC mode is based on vector prediction.
10. The system of claim 8, wherein the candidate in the block vector candidate list is the block vector.
11. The system of claim 8, wherein the default entry position of the block vector candidate list is the first entry of the block vector candidate list.
12. The system of claim 8, wherein signaling the intra mode is based on a most probable mode list.
13. The system of claim 8, wherein a candidate is selected from the intra-prediction mode candidate list by signaling an index of an entry in the intra-prediction mode candidate list.
14. The system of claim 8, wherein a candidate is selected from the intra-prediction mode candidate list by selecting a default entry position in the intra-prediction mode candidate list.
15. A non-transitory computer-readable storage medium for storing computer instructions, comprising: The computer instructions, when executed by at least one processor, cause the at least one processor to: Receiving a picture; and encoding the picture; configured to cause The encoding includes signaling a current block of the picture based on a block vector of an intra block copy (IBC) mode and at least one of an intra prediction direction and a non-directional mode option; encoding the picture includes: signaling the block vectors of the IBC mode; selecting a block vector candidate list by signaling an index of an entry in the block vector candidate list; selecting a candidate from the block vector candidate list based on a default entry position of the block vector candidate list; signaling an intra mode as indicating intra prediction for at least one segment of the current block; and Establishing a set of intra-prediction mode candidate lists 16. A non-transitory computer-readable storage medium comprising at least one of:
16. A non-transitory computer-readable storage medium as described in claim 15, wherein signaling the block vector of the IBC mode is based on vector prediction.
17. A non-transitory computer-readable storage medium as described in claim 15, wherein the candidate in the block vector candidate list is the block vector.
18. A non-transitory computer-readable storage medium as described in claim 15, wherein the default entry position of the block vector candidate list is the first entry of the block vector candidate list.
19. The non-transitory computer-readable storage medium of claim 15, wherein signaling the intra mode is based on a most probable mode list.
20. The candidate is signaling an index of an entry of the intra-prediction mode candidate list; and Selecting a default entry position of the intra-prediction mode candidate list. The non-transitory computer-readable storage medium of claim 15 , wherein the intra-prediction mode candidate list is selected by at least one of:
