Method and apparatus for encoding and decoding video, and computer program

Abstract

To provide a method and apparatus for video decoding.SOLUTION: A decoding apparatus includes processing circuitry. The processing circuitry decodes prediction information of a current block from a coded video bitstream. The prediction information is indicative of an intra block copy mode. The processing circuitry determines, based on the intra block copy mode, a first portion of a resolution syntax. The resolution syntax is unified of the same semantic for block vectors in the intra block copy mode and motion vectors in an inter picture merge mode. Further, the processing circuitry decodes a second portion of the resolution syntax from the coded video bitstream, and determines a block vector based on a resolution that is indicated by a combination of the first portion and the second portion. The processing circuitry reconstructs at least one sample of the current block based on the block vector.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This disclosure claims priority to U.S. Provisional Application No. 62 / 663,171, entitled "Method for Improved Intra-Frame Block Copy," filed April 26, 2018, and U.S. Application No. 16 / 205,180, entitled "Method and Apparatus for Video Encoding / Decoding," filed November 29, 2018, each of which is incorporated by reference in its entirety.

[0002] [Technical field] This disclosure describes embodiments generally related to video encoding / decoding. [Background technology]

[0003] The discussion of the background art provided herein is intended to provide an overall context for the present disclosure. The extent of the work of the currently signed inventors described in this background section and in each aspect of this specification has not been presented as prior art at the time of the filing of this disclosure, and is not expressly or impliedly admitted as prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using interframe image prediction with motion compensation. Uncompressed digital video can include a sequence of images, each with spatial dimensions of, for example, 1920x1080 luma samples and associated chroma samples. The sequence of images can have a fixed or variable image rate (informally known as frame rate) of, for example, 60 images per second or 60 Hertz (Hz). Uncompressed video has very high bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires close to 1.5 Gbit / s of bandwidth. Such a video requires more than 600 GB of storage space per hour. One goal of video encoding and decoding is to reduce redundant information in the input video signal through compression. Compression can help reduce the bandwidth or storage space requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression, as well as a combination of both, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that it can be utilized for applications where a reconstructed signal is expected. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application, e.g., a user consuming a streaming application can tolerate higher distortion than a user of a television distribution application. The achievable compression ratio reflects that a higher allowed / tolerable distortion can produce a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from a number of broad categories, including, for example, motion compensation, transform, quantization, and entropy coding. Video encoding / decoding techniques can include a technique known as intraframe coding. In intraframe coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, an image is spatially subdivided into sample blocks. If all sample blocks are coded in intraframe mode, the image can be an intraframe image. Intraframe images and their derivatives, such as independent decoder refresh images, can be used to reset the state of the decoder and thus can be used as the first image or still image in the coded video bitstream and video session. Samples of intraframe blocks are used in the transform and the transform coefficients can be quantized before entropy coding. Intraframe prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.

[0006] Conventional intraframe coding, as known for example from MPEG-2 coding techniques, does not use intraframe prediction. However, some newer video compression techniques include techniques that attempt to derive a data block from surrounding sample data and / or metadata, which were obtained earlier during the coding / decoding period and in the decoding order of spatially adjacent blocks. Such techniques are hereinafter referred to as "intraframe prediction" techniques. It should be noted that, at least in some cases, intraframe prediction does not use reference data from a reference picture, but only reference data from the current picture being reconstructed. There can be many different forms of intra prediction. If a given video coding technique can use more than one of such techniques, the technique in use can perform coding in an intra prediction mode. In some cases, a mode may have sub-modes and / or parameters, which may be coded alone or included in a mode codeword. This may be the case for the entropy coding technique used to convert the codeword to a bitstream, since which codeword is used for a given mode / sub-mode / parameter combination affects the coding efficiency gain with intra prediction.

[0007] A specific mode of intraframe prediction was introduced in H.264, improved in H.265 and further improved in new encoding / decoding techniques such as the joint exploration model (JEM), versatile video coding (VVC) and benchmark set (BMS). A prediction block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied into the prediction block according to a certain direction. The reference to the direction in use may be coded in the bitstream or may itself be predicted.

[0008] Referring to FIG. 1, at the bottom right, a subset of 9 known prediction directions from the 35 possible prediction directions of H.265 is depicted. The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right, which are at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101), which are at an angle of 22.5 degrees from the horizontal.

[0009] Continuing with reference to FIG. 1, a square block of 4×4 samples (104) is depicted at the top left (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S" and its position in the X dimension (e.g., row index) and its position in the Y dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. S44 is at the bottom right because this block is a 4×4 sized sample. Additionally, reference samples are shown that follow a similar numbering scheme. The reference samples are labeled with "R" and their X (e.g., row index) and Y (e.g., column index) positions relative to the block (104). In both H.264 and H.265, since the predicted samples are adjacent to the block being reconstructed, it is not necessary to use negative values.

[0010] Intraframe image prediction can work by copying reference sample values ​​from adjacent samples according to a signaled prediction direction. For example, assume that the coded video bitstream contains signaling indicating for this block the prediction direction consistent with the arrow (102), i.e., the samples are predicted from one or more prediction samples in the upper right corner that are at a 45 degree angle with the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08. In some cases, to calculate a reference sample, particularly when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example through interpolation.

[0011] With the development of video coding technology, the number of possible directions has already been increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and JEM / VC / BMS can support up to 65 directions at the time of disclosure. Experiments were performed to identify the most likely directions, and some techniques in entropy coding are used to represent those possible directions with a small number of bits, with some compensation for less likely directions. Furthermore, the direction itself may be predicted from neighboring directions used in neighboring already decoded blocks.

[0012] FIG. 2 shows a schematic diagram (201) depicting 65 intra-frame prediction directions according to JEM to illustrate the increasing number of prediction directions over time.

[0013] The mapping from intra prediction directions to bits representing directions in the coded video bitstream can vary across video coding techniques and can range, for example, from a simple direct mapping to the prediction directions to complex adaptation schemes involving intra prediction modes, codewords, most likely modes, and similar techniques. In all cases, however, there may be certain directions that are statistically less likely to occur in the video content than other certain directions. Because the goal of video compression is to reduce redundancy, these less likely directions are represented with more bits than more likely directions in a well-functioning video coding technique. Summary of the Invention [Means for solving the problem]

[0014] Aspects of the present disclosure provide a method and apparatus for decoding moving images. In some examples, the apparatus includes a processing circuit for decoding moving images. The processing circuit decodes prediction information of a current block from an encoded video bitstream. The prediction information indicates an intra-frame block copy mode. And, the processing circuit determines a first portion of a resolution syntax element based on the intra-frame block copy mode. The resolution syntax element is integrated to have the same meaning for block vectors in the intra-frame block copy mode and motion vectors in the inter-frame merge mode. Further, the processing circuit decodes a second portion of the resolution syntax element from the encoded video bitstream and determines a block vector based on a target resolution indicated by a combination of the first portion and the second portion. And, the processing circuit reconstructs at least one sample of the current block based on the block vector.

[0015] According to one aspect of the disclosure, the processing circuit determines a first portion indicating that the selectable resolution is integer pixel based on an intra-frame block copy mode. For example, the processing circuit determines that the first portion is a binary 1 based on an intra-frame block copy mode, the binary 1 indicating an integer pixel resolution based on a meaning for a motion vector in an inter-frame image merge mode. Then, the processing circuit selects the target resolution from the selectable resolutions based on a second portion of the resolution syntax element.

[0016] According to one aspect of the disclosure, the processing circuit determines a first portion of a resolution syntax element based on an intra-frame block copy mode without decoding additional information from the encoded video bitstream. In one embodiment, the processing circuit recognizes a current image to which the current block belongs as a reference image for the current block, and decodes a value specifying a maximum number of candidates in a candidate list from a slice header of a slice that includes the current block. In one example, the processing circuit builds a merge candidate list for the current block in an intra-frame block copy mode, where the number of intra-frame merge candidates in the merge candidate list does not exceed the value. In another example, the processing circuit builds a merge candidate list for another block in an inter-frame prediction mode, where the number of inter-frame merge candidates in the merge candidate list does not exceed the value.

[0017] In one embodiment, no temporal reference pictures are used in the slice, hi another embodiment, the value is coded with a truncated unary code.

[0018] An aspect of the present disclosure provides a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a computer for decoding video, cause the computer to perform the video decoding method. [Brief description of the drawings]

[0019] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which: [Figure 1] 1 is a schematic diagram of a subset of intra-frame prediction modes according to some examples. [Diagram 2] 2 is a schematic diagram of intra-frame prediction direction according to some examples; FIG. [Diagram 3] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 4] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 5]FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] FIG. 2 shows a block diagram of an encoder according to another embodiment. [Figure 8] FIG. 4 shows a block diagram of a decoder according to another embodiment. [Figure 9] FIG. 2 illustrates an example of intra-frame block copying according to one embodiment of the present disclosure. [Figure 10] FIG. 2 illustrates an example of bilateral matching according to some embodiments. [Figure 11] FIG. 2 illustrates an example of template matching according to one embodiment of the present disclosure. [Figure 12] FIG. 13 is a diagram illustrating examples of spatial merge candidates. [Figure 13] FIG. 13 illustrates an example of parameter calculation for illumination compensation. [Figure 14] FIG. 13 is a diagram illustrating an example of overlap between a reference block and a current block. [Figure 15] FIG. 1 illustrates a flowchart outlining an example process according to some embodiments of the present disclosure. [Figure 16] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] FIG. 3 is a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a video image stream captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to reconstruct the video image, and display the video image based on the reconstructed video data. Unidirectional data transmission is common, such as in media serving applications.

[0021] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of encoded video data, such as may occur during a video conference. For bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a video image stream captured by the terminal device) for transmission over the network (350) to the other of the terminal devices (330) and (340). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can also decode the encoded video data to reconstruct the video image and display the video image on an accessible display device based on the reconstructed video data.

[0022] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (350) represents any number of networks that transmit encoded video data between the terminal devices (310), (320), (330), and (340), including wired and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of the network (350) may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0023] 4 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example application for the disclosed subject matter. The disclosed subject matter is equally applicable to other video supporting applications including, for example, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., video conferencing, digital TV, etc.

[0024] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, that creates, for example, an uncompressed video image stream (402). In one example, the video image stream (402) includes samples taken by the digital camera. The video image stream (402), depicted with thick lines to emphasize its high data volume compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronics device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or realize various aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted with thin lines to emphasize its low data volume compared to the video image stream (402), may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in an electronics device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) to generate an outgoing video image stream (411) that can be displayed on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video encoding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally referred to as Versatile Video Coding, or VVC. The disclosed subject matter can be used in the context of VVC.

[0025] It should be noted that the electronics devices (420) and (430) can include other components (not shown). For example, the electronics device (420) can include a video decoder (not shown), and the electronics device (430) can similarly include a video encoder (not shown).

[0026] 5 shows a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) can be included in an electronics device (530). The electronics device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0027] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510), and in the same or another embodiment, may receive one encoded video sequence at a time, where the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be transmitted to a respective using entity (not shown). The receiver (531) may separate the encoded video sequences from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be located outside the video decoder (510) (not shown). In still other cases, there may be a buffer memory (not shown) outside the video decoder (510), e.g., to prevent network jitter, and there may be another buffer memory (515) inside the video decoder (510), e.g., to handle playback timing. If the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (515) may not be necessary or may be small.For use in a best-effort packet network such as the Internet, the buffer memory (515) may be necessary and may be relatively large, advantageously of adaptive size, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0028] The video decoder (510) may include an analyzer (520) for reconstructing symbols (521) from the encoded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and potential information for controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronics device (530) but may be coupled to the electronics device (530) as shown in FIG. 5. The rendering device control information may be in the form of a supplemental enhancement information (SEI message) or a visual usability information (VUI) parameter set fragment (not shown). The analyzer (520) may perform analysis / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The analyzer (520) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups can include a group of pictures (GOP), an image, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The analyzer (520) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0029] The analyzer (520) can perform entropy decoding / analysis operations on the video sequence received from the buffer memory (515) to produce symbols (521).

[0030] The reconstruction of the symbols (521) may relate to a number of different units, depending on the type of coded video image or part thereof (e.g., inter-frame and intra-frame images, inter-frame and intra-frame blocks) and other factors. What units relate to and how they relate to may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not shown for clarity.

[0031] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In an actual embodiment operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0032] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients and control information from the analyzer (520) including what transform to use, block size, quantization factor, quantization scaling matrix, etc. as symbols (521). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0033] In some cases, the output samples of the scalar / inverse transform unit (551) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image, but may use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) uses surrounding already reconstructed information extracted from a current image buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current image buffer (558) buffers, for example, a partially reconstructed current image and / or a fully reconstructed current image. The aggregator (555) adds the prediction information generated by the intra-frame prediction unit (552) to the output sample information provided by the scalar / inverse transform unit (551), in some cases on a sample-by-sample basis.

[0034] In other cases, the output samples of the scalar / inverse transform unit (551) may belong to an inter-frame coded block and potentially a motion compensated block. In such cases, the motion compensated prediction unit (553) may access the reference image memory (557) to fetch samples used for prediction. After the fetched samples are motion compensated based on the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case referred to as residual samples or residual signals) to generate output sample information. The address in the reference image memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector available to the motion compensated prediction unit (553), for example in the form of a symbol (521) that may have X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557), motion vector prediction mechanisms, etc. when sub-sample accurate motion vectors are in use.

[0035] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the analyzer (520), may be responsive to meta-information obtained during decoding of a coded image or previous portion of a coded video sequence (in decoding order), and may be responsive to previously reconstructed loop filtered sample values.

[0036] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) and stored in a reference image memory (557) for use in future interframe image prediction.

[0037] Once a particular coded image has been fully reconstructed, it can be used as a reference image for future predictions. For example, once a coded image corresponding to a current image has been fully reconstructed and the coded image has been identified (e.g., by the analyzer (520)) as a reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before beginning reconstruction of a subsequent coded image.

[0038] The video decoder (510) may perform decoding operations according to a given video compression technique in a standard, such as ITU-T Rec. H.265. The encoded video sequence may conform to a syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence conforms to both the syntax of the video compression technique or standard and the profile as a document of the video compression technique or standard. In particular, the profile may select some tools from all tools available in the video compression technique or standard as the only tools available in the profile. It is also necessary for compliance that the complexity of the encoded video sequence is within a range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in mega samples per second), maximum reference picture size, etc. The limits set by the hierarchy may be further limited in some cases by a Hypthetical Reference Decoder (HRD) specification and HRD buffer management metadata signaled in the encoded video sequence.

[0039] In one embodiment, the receiver (531) can receive additional (redundant) data along with the encoded video. The additional data can be included as part of the encoded video sequence. The additional data can be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data can be in forms such as, for example, temporal, spatial, or signal noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0040] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronics device (620). The electronics device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0041] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronics device (620) in the example of FIG. 6) that captures video images to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronics device (620).

[0042] The video source (601) may provide a source video sequence encoded by the video encoder (603) in the form of a digital video sample stream, said digital video sample stream having any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...) and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual images that give motion when viewed in sequence. The images themselves may be organized as a spatial pixel array, where each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0043] According to one embodiment, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real-time or under any other time constraint required by the application. Enforcing an appropriate encoding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units as described below, which are not shown for clarity. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0044] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified explanation, in one example, the encoding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner to how the (remote) decoder creates the sample data (because in the video compression techniques contemplated in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). The decoding of the symbol stream provides bit-exact results regardless of the location of the decoder (local or remote), so that the contents of the reference image memory (634) correspond bit-exactly between the local and remote encoders. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same sample values ​​"seen" by the decoder when it uses the prediction during decoding. This basic principle of reference image synchrony (and the drift that occurs when synchrony is not maintained, e.g., due to channel errors) is also used in several related technologies.

[0045] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in connection with Figure 5. However, with further brief reference to Figure 5, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the analyzer (520), may not be fully implemented in the local decoder (633), because symbols are available and the encoding / decoding of symbols into the encoded video sequence by the entropy coder (645) and the analyzer (520) can be lossless. It should be noted at this point that any decoder technique other than analysis / entropy decoding present in the decoder must necessarily be present in a corresponding encoder in substantially identical functional form. For this reason, the disclosed subject matter focuses on the decoder operation. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas are more detailed descriptions necessary, and are provided below.

[0046] During operation, in some embodiments, the source coder (630) can perform motion-compensated predictive coding, which predictively codes an input image with reference to one or more previously coded images from the video sequence designated as “reference images.” In this manner, the coding engine (632) codes differences between pixel blocks of the input image and pixel blocks of reference images that can be selected as predictive references for the input image.

[0047] The local video decoder (633) can decode the encoded video data of the image that can be designated as a reference image based on the symbols generated by the source coder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data is decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence can be a replica of the source video sequence, usually with some errors. The local video decoder (633) can copy the decoding process that can be performed by the video decoder on the reference image and store the reconstructed reference image in the reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that has a common content with the reconstructed reference image obtained by the far-end video decoder (in the absence of transmission errors).

[0048] The predictor (635) can perform a prediction search for the coding engine (632). That is, for a new image to be coded, the predictor (635) can search the reference image memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference image motion vectors, block shapes, etc., that serve as suitable prediction references for the new image. The predictor (635) can operate on a pixel block by pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image can have prediction references drawn from multiple reference images stored in the reference image memory (634).

[0049] The controller (650) can manage the encoding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0050] The output of all the above mentioned functional units can be entropy coded in an entropy coder (645), which converts the symbols produced by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0051] The transmitter (640) can buffer the encoded video sequence generated by the entropy coder (645) in preparation for transmission over a communication channel (660), which can be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the encoded video data from the video coder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown). A controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign each encoded image a particular encoded image type, which can affect the encoding technique that can be applied to each image. For example, images are often assigned as one of the following image types: An intraframe picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intraframe pictures, such as Independent Decoder Refresh ("IDR") pictures. Those skilled in the art understand the variants of I-pictures and their applications and functions.

[0052] A predicted image (P-image) may be encoded and decoded using intra-frame or inter-frame prediction, which predicts the sample values ​​of each block using at most one motion vector and reference index.

[0053] Bidirectionally predicted images (B-pictures) may be encoded and decoded using intra-frame or inter-frame prediction, which predicts the sample values ​​of each block using at most two motion vectors and reference indices. Similarly, multi-predictive images may use more than two reference images and associated metadata to reconstruct a single block.

[0054] A source image is generally spatially subdivided into a number of sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and can be coded block by block. These blocks can be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective image of the block. For example, blocks of an I image may be non-predictively coded or they may be predictively coded with reference to already coded blocks of the same image (spatial prediction or intraframe prediction). Pixel blocks of a P image may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference image. Blocks of a B image may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference images.

[0055] The video encoder (603) may perform encoding operations according to a given video encoding technique or standard, such as ITU-T H.265, etc. In its operations, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0056] In one embodiment, the transmitter (640) can transmit additional data along with the encoded video. The source coder (630) can include such data as part of the encoded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images or slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0057] A video can be captured as multiple source images (video images) in a time sequence. Intraframe image prediction (often abbreviated as intraframe prediction) exploits spatial correlation in a given image, while interframe image prediction exploits correlation (temporal or other) between images. In one example, a particular image being coded / decoded, called a current image, is divided into blocks. If a block of the current image is similar to a reference block in a previously coded and still buffered reference image in the video, the block of the current image can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference image, and can have a third dimension that identifies the reference image if multiple reference images are used.

[0058] In some embodiments, bidirectional prediction techniques can be used for interframe image prediction. According to bidirectional prediction techniques, two reference images are used, such as a first and a second reference image, both of which are before the current image in the video in decoding order (but may be past and future, respectively, in display order). A block in the current image can be coded by a first motion vector that points to a first reference block in the first reference image and a second motion vector that points to a second reference block in the second reference image. A block can be predicted by a combination of the first and second reference blocks.

[0059] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-frame image prediction.

[0060] According to some embodiments of the present disclosure, prediction, such as inter-frame image prediction or intra-frame image prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, an image in a sequence of video images is divided into coding tree units (CTUs) for compression, and the CTUs in an image have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively divided into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. The CU is divided into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in encoding / decoding is performed in units of prediction blocks. Using luma prediction blocks as an example of prediction blocks, prediction blocks include matrices of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0061] 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video image in a video image sequence and to encode the processed block into an encoded image that is part of the encoded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) in the example of FIG. 4.

[0062] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (703) determines whether to encode the processing block using an intra mode, an inter mode, or a bidirectional prediction mode, for example using rate-distortion optimization. If the processing block is encoded in an intra mode, the video encoder (703) can encode the processing block into an encoded image using an intra prediction technique, and if the processing block is encoded in an inter mode or a bidirectional prediction mode, the video encoder (703) can encode the processing block into an encoded image using an inter prediction or bidirectional prediction technique, respectively. In certain video encoding techniques, the merge mode can be an inter image prediction submode in which a motion vector is derived from one or more motion vector predictors, without taking advantage of any encoded motion vector components other than the predictors. In certain other video encoding techniques, there may be motion vector components applicable to the subject block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0063] In the example of FIG. 7, the video encoder (703) includes an interframe encoder (730), an intraframe encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in FIG. 7.

[0064] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in reference images (e.g., blocks in previous and subsequent images), generate inter-frame prediction information (e.g., redundant information description from inter-frame coding techniques, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image that has been decoded based on the encoded video information.

[0065] The intraframe encoder (722) is configured to receive samples of a current block (e.g., a processing block), in some cases compare the block with previously encoded blocks in the same image, generate transformed and quantized coefficients, and in some cases generate intraframe prediction information (e.g., intraframe prediction direction information according to one or more intraframe coding techniques). In one example, the intraframe encoder (722) also calculates intraframe prediction results (e.g., predicted blocks) based on the intraframe prediction information and reference blocks in the same image.

[0066] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, if the mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the codestream. Also, if the mode is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the codestream.

[0067] The residual calculator (723) is configured to calculate a difference (residual data) between a received block and a prediction result selected from the intraframe encoder (722) or the interframe encoder (730). The residual encoder (724) is configured to operate on the residual data and generate transform coefficients by encoding the residual data. In one example, the residual encoder (724) is configured to transform the residual data in the frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intraframe encoder (722) and the interframe encoder (730) as appropriate. For example, the inter-frame encoder (730) may generate decoded blocks based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) may generate decoded blocks based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded image, and in some examples, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0068] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intraframe prediction information or interframe prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the disclosed subject matter, when encoding a block in an interframe mode or a merged sub-mode of a bi-predictive mode, there is no residual information.

[0069] 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded image that is part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0070] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an interframe decoder (880), a residual decoder (873), a reconstruction module (874), and an intraframe decoder (872), coupled together as shown in FIG. 8.

[0071] The entropy decoder (871) may be configured to reconstruct from the coded image certain symbols that represent syntax elements that make up the coded image. Such symbols may include, for example, a mode for coding the block (e.g., intra mode, inter mode, bi-predictive mode, merged submode of the latter two or another submode), prediction information (e.g., intra prediction information or inter prediction information, etc.) that may identify certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), respectively, and residual information, for example in the form of quantized transform coefficients. In one example, if the prediction mode is an inter prediction mode or a bi-predictive mode, the inter prediction information is provided to the inter decoder (880). And if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0072] The inter-frame decoder (880) is configured to receive the inter-frame prediction information and to generate an inter-frame prediction result based on the inter-frame prediction information.

[0073] The intraframe decoder (872) is configured to receive the intraframe prediction information and to generate a prediction result based on the intraframe prediction information.

[0074] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (e.g., including Quantizer Parameters (QP)), which may be provided by the entropy decoder (871) (data path not shown as this is only low volume control information).

[0075] The reconstruction module (874) is configured to combine, in the spatial domain, the residual as output by the residual decoder (873) and the prediction result (possibly as output by the inter-frame prediction module or the intra-frame prediction module) to form a reconstructed block, which may be part of a reconstructed image, which may then be part of a reconstructed video. It should be noted that it may perform other appropriate operations, such as a deblocking operation, to improve visual quality.

[0076] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be realized using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be realized using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0077] Aspects of the present disclosure provide a block-based compensation technique with the same image.

[0078] Block-based compensation can be used for inter-frame prediction and intra-frame prediction. For inter-frame prediction, block-based compensation from a different image is called motion compensation. Similarly, for intra-frame prediction, block-based compensation can also be done from a previously reconstructed region in the same image. Block compensation can also be performed from a previously reconstructed region in the same image. Block-based compensation from a reconstructed region in the same image is called intra-frame image block compensation or intra-frame block copy. The shift vector indicating the offset between the current block and a reference block in the same image is called a block vector (or BV for short). Unlike the motion vector in motion compensation, which can be any value (positive or negative, either in x or y direction), the block vector has some constraints, which ensure that the reference block is available and has already been reconstructed. Also, in some examples, some reference regions that are tile boundaries or wavefront ladder shape boundaries are excluded to allow for parallel processing.

[0079] The coding of block vectors may be either explicit or implicit. In explicit mode, the difference between a block vector and its predicted value is signaled, while in implicit mode, the block vector is reconstructed from a predicted value (called the block vector predictor) in a manner similar to the motion vectors in merge mode. In some implementations, the block vector resolution is limited to integer positions, while in other systems, the block vectors are allowed to point to fractional positions.

[0080] In some examples, the use of intra-frame block copying at the block level can be signaled using a reference index approach. The current picture being decoded is treated as a reference picture. In one example, such a reference picture is placed at the last position in the list of reference pictures. This special reference picture is also managed together with other temporal reference pictures in a buffer such as the decoded picture buffer (DPB).

[0081] There are also some variations on intra-frame block copying, such as inverted intra-frame block copying (where the reference block is flipped horizontally or vertically before being used to predict the current block), or line-based intra-frame block copying (where each compensation unit in an M×N coding block is an M×1 or 1×N line).

[0082] FIG. 9 illustrates an example of intra-frame block copying according to one embodiment of the present disclosure. A current image (900) is being decoded. The current image (900) includes a reconstructed region (910) (gray region) and a region to be decoded (920) (white region). A current block (930) is being reconstructed by the decoder. The current block (930) can be reconstructed from a reference block (940) that is located in the reconstructed region (910). The position offset between the reference block (940) and the current block (930) is called the block vector (950) (or BV (950)).

[0083] According to aspects of the present disclosure, the motion compensation based techniques can be appropriately modified for intra-frame block copying.

[0084] In one example, pattern matched motion vector derivation (PMMVD) mode is a technique in which the motion information of a block is not signaled but is derived at both the encoder and decoder sides. Generally, there are two pattern matched motion vector derivation methods: bilateral matching and template matching.

[0085] FIG. 10 is a diagram illustrating an example of bilateral matching according to some embodiments. As shown in FIG. 10, bilateral matching is used to derive motion information of a current CU (1010) by finding two best-matching blocks (1020) and (1030) in two different reference images (Ref0 and Ref1) along the motion trajectory of the current CU (1010) (in the current image). Under the assumption of continuous motion trajectory, the motion vectors MV0 and MV1 pointing to the two reference blocks (1020) and (1030) are proportional to the temporal distances between the current image and the two reference images (Ref0 and Ref1), i.e., TD0 and TD1. As a special case, if the current image is between the two reference images in time and the temporal distances from the current image (CurPic) to the two reference images (Ref0 and Ref1) are the same, the bilateral matching becomes mirror-based bidirectional MV.

[0086] 11 is a diagram illustrating an example of template matching according to an embodiment of the present disclosure. As shown in FIG 11, template matching is used to derive motion information of a current CU (1110) by finding the closest match between a template in a current image (including neighboring blocks (1120) and (1130) on the top and left of the current CU (1110)) and blocks (1140) and (1150) in a reference image (Ref0) (having the same shape and size as the template).

[0087] In some examples, motion compensation is performed at the block level, i.e., the current block is a processing unit for performing motion compensation using the same motion information. Given the size of a block, all pixels in the block use the same motion information to form their prediction block.

[0088] In another example, a technique using block-level merging candidates is used in motion compensation. Block-level merging candidates can include spatial merging candidates and temporally adjacent positions. In bi-prediction, block-level merging candidates can also include some combination of motion vectors from existing merging candidates.

[0089] FIG. 12 shows an example of a spatial merge candidate. In the example of FIG. 12, a current block (1201) contains samples that can be predicted from a spatially shifted previous block of the same size found by the encoder during the motion search process. Instead of directly encoding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent (in decoding order) reference picture using MVs associated with any one of five surrounding samples represented as A0, A1, and B0, B1, B2 (respectively 1202-1206). Then, the MV prediction can use predicted values ​​from the same reference picture that the neighboring blocks use. In the example of FIG. 12, samples at neighboring positions (1202)-(1206) of the current block are used for the spatial merge candidate.

[0090] In another example, illumination compensation (IC) is used for motion compensation.

[0091] For example, due to illumination changes from image to image or region to region, adjustments to reflect such changes, when applicable, can improve prediction accuracy. In some examples, illumination adjustments are performed at the block level for inter-frame coded blocks using a scaling factor a and an offset b. Illumination adjustments can be adaptively enabled or disabled for each coding unit (CU) coded in inter-frame mode. Illumination adjustments are also referred to as illumination compensation (IC). In one example, assume that X is an illumination value for a pixel of a prediction block A, and after illumination compensation, an adjusted illumination value for a corresponding pixel of a new prediction block B is calculated by y=ax+b, and this adjusted illumination value can be used for motion compensation. The parameters a and b can be signaled or calculated using the difference between neighboring pixels of the current CU and neighboring pixels of the reference block in the reference image. Or, they can be estimated from neighboring coded blocks (which already have parameters a and b).

[0092] FIG. 13 shows an example of parameter calculation for illumination compensation. In the example of FIG. 13, a number of adjacent samples (1320) of a current CU (1310) are selected, and the illumination values ​​of the selected adjacent samples (1320) are used as a representative value of y. Similarly, a number of adjacent samples (1340) of a reference block (1330) corresponding to the selected adjacent samples (1320) are selected, and the illumination values ​​of the selected adjacent samples (1340) are used as a representative value of x. The representative values ​​of y and x are used to calculate parameters a and b, assuming y=ax+b. Rec neig represents the illumination value of the adjacent samples of the current CU, and Rec refneig represents the illumination value of the corresponding adjacent sample of the reference block, and 2N is the Rec neig and Rec refneig Then, a and b can be calculated as shown in Equation 1 and Equation 2.

[0093]

number

[0094]

number

[0095] Traditionally, motion vector resolution is a fixed value, e.g., 1 / 4 pixel or 1 / 8 pixel accuracy in H.264 / AVC and HEVC Main Profile. In HEVC SCC, motion vector resolution can be selected at either 1 integer pixel or 1 / 4 pixel. The switch is done on a slice-by-slice basis. In other words, all motion vector resolutions within a slice are the same.

[0096] In some later developments, the motion vector resolution can be either 1 / 4 pixel, 1 integer pixel or 4 integer pixels. 4 integer pixels means that a vector difference of one unit represents 4 integer pixels. Thus, the distance between the symbols "0" and "1" is 4 integer pixels. Also, adaptivity is performed at the block level, i.e., the motion vector can be selected from different resolutions for each block. In some examples, such adaptivity is realized using an integer motion vector (IMV) flag with one bin (binary) or two bins (binary). The first bin indicates whether the MV of the current block is coded with integer pixel resolution. If not, this MV is coded with 1 / 4 pixel resolution. If the first bin indicates that the MV of the current block is coded with integer pixel resolution, the second bin indicates whether the MV of the current block is coded with 4 integer pixel resolution. If not, this MV is coded with 1 integer pixel resolution.

[0097] Techniques used in motion compensation, such as pattern match motion vector derivation, block level candidate techniques, illumination compensation techniques, etc., are appropriately applied in intra-frame block copying to improve efficiency.

[0098] According to an aspect of the present disclosure, block vector resolution is signaled, and adaptive block vector resolution and adaptive motion vector resolution are represented using unified signaling (e.g., the same flag). For example, intra-frame block copy mode and inter-frame mode share the same bitstream syntax structure for vector resolution adaptability. The same IMV signaling flag is used for both block vector (BV) in intra-frame block copy and motion vector (MV) in motion compensation.

[0099] In one example, the set of selectable resolutions for BV is a subset of the selectable resolutions for MV. For example, the selectable resolutions for MV include a subset of fractional pixel resolutions (e.g., 1 / 4 pixel) and integer pixel resolutions (e.g., 1 integer pixel and 4 integer pixel). The selectable resolutions for BV include a subset of integer pixel resolutions (e.g., 1 integer pixel and 4 integer pixel). In motion compensation, the resolution signaling includes a first signaling bin that signals whether the vector is coded with integer pixel resolution (integer pixel). If the first signaling bin indicates that the vector is coded with integer pixel resolution, the second signaling bin is used to indicate which integer pixel resolution is used to code the vector. If the current block is coded in intraframe block copy mode, the first signaling bin can be derived instead of being signaled, since the selectable resolutions are limited to integer pixel resolution. For the other bins in the IMV flag, the meaning of each bin for BV coding is the same as the meaning of the bin for MV coding. Tables 1 and 2 show examples of how to use the same IMV flags to match the BV resolution to the MV resolution.

[0100] [Table 1]

[0101] [Table 2] In another way, when a block is coded in intraframe block copy mode, the selectable resolution for BV can be different from the selectable resolution for MV. For example, BV can switch between 1 integer pixel, 2 integer pixels, or 4 integer pixels, and MV can switch between ¼ pixel, 1 integer pixel, and 4 integer pixels. If the number of selectable resolutions for BV is the same as the number of selectable resolutions for MV, the two types of vectors (BV and MV) can still share the IMV flag to signal the resolution. However, the meaning of the binarization of the IMV flag is different. Table 3 shows an example of binarizing the IMV flag using 1, 2, and 4 integer pixel resolutions for BV and ¼ pixel, 1, and 4 integer pixel resolutions for MV.

[0102] [Table 3] According to another aspect of the present disclosure, when the merge mode is enabled and no temporal reference picture is used, the intra-frame block copy mode is inferred. Thus, when no temporal reference picture is used, such as in the intra-frame block copy case, the maximum number of merge candidates in a slice can be signaled.

[0103] In conventional video coding methods, the merge mode is applied only when the current block is coded with inter-frame image compensation, and is referred to as the inter-frame image merge mode in one example. Thus, if the current slice has at least one temporal reference image, the maximum number of merge candidates to be used for the current slice is signaled.

[0104] In one embodiment, at the encoder side, if intra block copy is used in a slice, the encoder encodes the slice without using a temporal reference picture and enables a merge mode for this slice to indicate the use of intra block copy for this slice. Furthermore, the encoder signals the maximum number of merge candidates used for the current slice in the slice header. Since intra block copy is the only type of merge mode in such a slice, the maximum number of merge candidates is used to specify the maximum allowed merge candidates for intra block copy. Thus, at the decoder side, if enabling of merge mode for a slice without a temporal reference picture is detected, the decoder determines that the slice used intra block copy mode. Furthermore, at the decoder side, the decoder can decode the maximum number of merge candidates for intra block copy mode from the slice header.

[0105] In one method, the technique for signaling the maximum number of merge candidates in slices where no temporal reference picture is used (e.g., in intra-frame block copy mode) can be the same as the method for signaling the maximum number of merge candidates in slices where at least one temporal reference picture is used to specify the maximum number of merge candidates (e.g., in inter-frame picture merging mode). For example, in both cases, a truncated unary code is used to encode the syntax element for the maximum number of merge candidates.

[0106] In another approach, the technique for signaling the maximum number of merge candidates in slices where no temporal reference picture is used (e.g., in intra-frame block copy mode) can be different from the method for signaling the maximum number of merge candidates in slices where at least one temporal reference picture is used to specify the maximum number of inter-frame image merge candidates (e.g., in inter-frame image merge mode). For example, the maximum number of merge candidates in intra-frame block copy is signaled using a fixed-length code, and the maximum number of merge candidates in inter-frame image merge mode is signaled using a truncated unary code.

[0107] Alternatively, the maximum number of merge candidates may be signaled by the encoder in a sequence parameter set (SPS) or picture parameter set (PPS) or other parameter set. The decoder may then decode the maximum number of merge candidates. For slices with no temporal reference pictures, if the merge mode is enabled, the decoder may use the maximum number of merge candidates for the intra-frame block copy mode. For slices with at least one temporal reference picture, if the merge mode is enabled, the decoder may use the maximum number of merge candidates for the inter-frame picture merge mode.

[0108] According to another aspect of the present disclosure, to enable illumination compensation in intra-frame block copy mode, additional constraints, rules, or condition checks are required for the availability of neighboring pixels.

[0109] Usually, the top and / or left neighboring pixels are used to calculate the illumination compensation parameters. For the reference block and the current block, only if the neighboring pixels on the same side (left or top) are available in both the reference block and the current block, the pixels on this side can participate in the parameter calculation for illumination compensation. For example, for a left neighboring pixel, if this left neighboring pixel is not available from either the reference block or the current block, the left neighboring pixel cannot be used for the parameter calculation for illumination compensation.

[0110] In motion compensation, if the reference block is not located at the top / left border of the image, the neighboring pixels of the reference block are always available. Following the pseudo codes, an example of availability condition check for neighboring pixels of the reference block in motion compensation (interframe image merge mode) is shown. In the pseudo codes, LX indicates one of two prediction lists (e.g., L0 is the first list and L1 is the second list), refLX indicates a reference image (e.g., refL0 is the first reference image and refL1 is the second reference image), predFlagLX indicates a prediction flag for LX, (xR, yR) indicates the position of the top-left sample of the reference coding unit in LX relative to the top-left sample of the reference block, avaiAboveRowRefLX indicates the availability of the top neighboring pixels of the reference block, and avaiLeftColRefLX indicates the availability of the left neighboring pixels of the reference block.

[0111] [Pseudocode] if predFlagLX equals to 1 { derive (xR, yR) which is a vector between the top-left sample of the referenced coding unit in the reference picture refLX and the top left sample of the reference picture refLX; if yR is larger than 0, avaiAboveRowRefLX is set to 1; otherwise, if yR is equal to or smaller than 0, avaiAboveRowRefLX is set to 0; if xR is larger than 0, avaiLeftColRefLX is set to 1; otherwise, if xR1 is equal to or smaller than 0, avaiLeftColRefLX is set to 0; } However, for intra-frame block copying, the reference block comes from the same image as the current block. In addition to the requirement that the reference block has already been reconstructed for use in intra-frame block copying, the neighboring pixels of such a reference block must satisfy several constraints for availability in the parameter calculation of illumination compensation.

[0112] For motion compensation, the neighboring pixels of the current block come from the current image, while the neighboring pixels of the reference block and the reference block itself come from another image. There is no overlap between the two sets of pixels. However, in intra-frame block copying, the neighboring pixels of the current block can overlap with the reference block, because the two sets of pixels come from the same image.

[0113] Figure 14 shows an example where the reference block and the current block overlap. When overlap occurs, the illumination compensation mechanism may need to be adjusted.

[0114] When the illumination compensation parameter calculation is used, several methods are proposed in this disclosure to limit the use of neighboring pixels of the reference block for intra-frame block copying. The proposed methods can be applied individually or in combination.

[0115] In one example, if the first row of a reference block lies on an image, slice or tile boundary, the top neighboring pixels should not be used in the calculation of illumination compensation parameters since they are outside the boundary. One way to do this is to mark the neighboring pixels in the top row of the reference block as "unusable". This also applies to any partition boundaries, so that pixels can reference each other within the boundary but are not allowed to cross the boundary.

[0116] In another example, if the first column of a reference block lies on an image, slice or tile boundary, the neighboring pixels to the left should not be used in the calculation of the illumination compensation parameters since they are outside the boundary. One way to do this is to mark the neighboring pixels in the left column of the reference block as "unusable". This also applies to any partition boundaries, so that pixels can reference each other within the boundary but are not allowed to cross the boundary.

[0117] In another example, if the top neighboring pixels of the current block overlap with the reference block, the top neighboring pixels should not be used in the illumination compensation parameter calculation. One way to do this is to mark the top row neighboring pixels of the reference block as "unusable".

[0118] In another example, if the left neighboring pixel of the current block overlaps with the reference block, the left neighboring pixel should not be used in the illumination compensation parameter calculation. One way to do this is to mark the left column neighboring pixel of the reference block as "unusable".

[0119] In another example, if the left or top neighboring pixels of the current block overlap the reference block, the left and top neighboring pixels should not be used in the calculation of the illumination compensation parameters. One way to do this is to mark the left column neighboring pixel and the top row neighboring pixel of the reference block as "unusable". If both neighboring pixels are unusable, illumination compensation is effectively not used for this block.

[0120] FIG. 15 shows a flow chart outlining a process (1500) according to an embodiment of the present disclosure. The process (1500) can be used for reconstruction of a block encoded in intraframe mode and can generate a prediction block of the block being reconstructed. In various embodiments, the process (1500) is performed by processing circuits in the terminal devices (310), (320), (330) and (340), processing circuits performing the functions of a video encoder (403), processing circuits performing the functions of a video decoder (410), processing circuits performing the functions of a video decoder (510), processing circuits performing the functions of an intraframe prediction module (552), processing circuits performing the functions of a video encoder (603), processing circuits performing the functions of a predictor (635), processing circuits performing the functions of an intraframe encoder (722), processing circuits performing the functions of an intraframe decoder (872), etc. In some embodiments, the process (1500) is implemented by software instructions, and thus, when the processing circuits execute the software instructions, the processing circuits perform the process (1500). The process begins at (S1501) and proceeds to (S1510).

[0121] In step S1510, prediction information of the current block is decoded from the encoded video bitstream, where the prediction information indicates an intra-frame block copy mode.

[0122] In (S1520), a first portion of a resolution syntax element is estimated based on an intra-frame block copy mode. In one example, the resolution syntax elements of the intra-frame block copy mode and the inter-frame image merge mode are integrated to have the same meaning for block vectors in the intra-frame copy mode and for motion vectors in the inter-frame image merge mode. In one example, the selectable resolution for the intra-frame block copy mode is a subset of the selectable resolution for the inter-frame image merge mode. For example, the selectable resolution for the inter-frame image merge mode includes fractional pixel resolution and integer pixel resolution, and the selectable resolution for the intra-frame block copy mode is a subset of the integer pixel resolution. In one example, the first portion of the resolution syntax element is used to indicate whether the resolution is fractional pixel resolution or integer pixel resolution. In this way, when the intra-frame block copy mode is determined, the first portion of the resolution syntax element can be estimated to indicate the integer pixel resolution.

[0123] At (S1530), a second portion of the resolution syntax element is decoded from the encoded video bitstream. In one example, a second portion of the resolution syntax element indicating a particular integer pixel resolution is signaled in the encoded video bitstream. At the decoder side, the decoder can decode the second portion from the encoded video bitstream.

[0124] In (S1540), a block vector of the current block is determined based on the resolution indicated by the combination of the first and second parts, in the same sense as for the resolution of the motion vector in the inter-frame image merge mode.

[0125] In step S1550, samples of the current block are constructed based on the determined block vector, and the process then proceeds to step S1599 where it ends.

[0126] The above techniques can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 16 illustrates a computer system (1600) suitable for implementing certain embodiments of the disclosed subject matter.

[0127] Computer software may be encoded using any suitable machine code or computer language and may undergo assembly, compilation, linking, or similar mechanisms to create code containing instructions, which may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or may be executed by code interpretation, microcode execution, etc.

[0128] The instructions may be executed in various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, internet of things devices, and the like.

[0129] The components of the computer system (1600) illustrated in Figure 16 are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (1600).

[0130] The computer system (1600) may include several human interface input devices. Such human interface input devices may respond to input by one or more users through tactile input (e.g., keystrokes, swipes, data glove movements, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from still image cameras, etc.), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic vision, etc.).

[0131] The human interface input devices may include one or more of a keyboard (1601), a mouse (1602), a trackpad (1603), a touch screen (1610), a data glove (not shown), a joystick (1605), a microphone (1606), a scanner (1607), and a camera (1608) (only one of each is shown).

[0132] The computer system (1600) may also include a number of human interface output devices that may stimulate one or more of the user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may be haptic output devices (e.g., haptic feedback via a touch screen (1610), data gloves (not shown) or joystick (1605), but may also be haptic feedback devices that do not act as input devices), audio output devices (e.g., speakers (1609), headphones (not shown)), visual output devices (e.g., screens (1610), including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capability, each of which may or may not have haptic feedback capability, some of which may output two dimensional visual output or three or more dimensional visual output, such as, for example, via stereographic output, virtual reality glasses (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0133] The computer system (1600) may include human accessible storage devices and their associated media, such as optical media or similar media (1621), including CD / DVD ROM / RW (1620) with CDs / DVDs, thumb drives (1622), removable hard drives or solid state drives (1623), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0134] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitory signals.

[0135] The computer system (1600) may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local networks, wide area networks, metropolitan area networks, vehicular and industrial networks, real-time networks, delay tolerant networks, and the like. Examples of networks include LANs such as Ethernet, wireless LANs, cellular networks (GSM, 3G, 4G, 5G, LTE, and the like), television cable or wireless wide area digital networks (including cable television, satellite television, terrestrial broadcast television), vehicular and industrial networks (including CANBus), and the like. Some networks generally require an external network interface adapter connected to some general-purpose data port or peripheral bus (1649) (e.g., a USB port on the computer system (1600)), while other systems are typically integrated into the core of the computer system (1600) by connecting to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1600) can communicate with other entities. Such communication may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., from the Canbus to a specific Canbus device), or two-way, e.g., to other computer systems using local or wide area digital networks. As mentioned above, specific protocols and protocol stacks can be used for each of these networks and network interfaces.

[0136] The above-mentioned human interface devices, human accessible storage devices, and network interfaces can be connected to the core (1640) of the computer system (1600).

[0137] The cores (1640) may include one or more central processing units (CPUs) (1641), graphics processing units (GPUs) (1642), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1643), hardware accelerators for specific tasks (1644), and the like. These devices may be connected via a system bus (1648), along with read-only memory (ROM) (1645), random access memory (1646), internal mass storage (1647), such as an internal non-user-accessible hard disk drive, SSD, and the like. In some computer systems, the system bus (1648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the core's system bus (1648) or may be connected via a peripheral bus (1649). Peripheral bus architectures include Peripheral to External Controller Interface (PCI), Universal Serial Bus (USB), and the like.

[0138] The CPU (1641), GPU (1642), FPGA (1643), and accelerator (1644) can execute a number of instructions, which can be combined to constitute the above-mentioned computer code. The computer code can be stored in a ROM (1645) or a RAM (1646). Also, temporary data can be stored in the RAM (1646), while permanent data can be stored, for example, in an internal mass storage (1647). The use of a cache memory, which can be closely associated with one or more of the CPU (1641), GPU (1642), mass storage (1647), ROM (1645), RAM (1646), etc., allows for fast storage and retrieval from any memory device.

[0139] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be any medium and code known and available to those skilled in the computer software arts.

[0140] By way of example and not limitation, a computer system having the architecture (1600), and in particular the core (1640), can provide functionality as a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media may be media related to the user-accessible mass storage discussed above, and may be specific storage with the core (1640), such as the core internal mass storage (1647) or ROM (1645), which is non-volatile. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1640). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. This software may cause the core (1640), and in particular the processor therein (including CPU, GPU, FPGA, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM (1646) and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may provide functionality because of logic hardwired or otherwise incorporated into circuitry (e.g., accelerator (1644)) that may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as integrated circuits (ICs)) that store software to be executed, circuitry embodying logic to be executed, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0141] Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS:benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Visual Usability Information GOPs: Groups of Pictures TUs: Transform Units PUs: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD:Hypothetical Reference Decoder SNR: Signal Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT:Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE:Long-Term Evolution, long-term evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect, external controller interface FPGA: Field Programmable Gate Arrays SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit IMV: Integer Motion Vector

[0142] While this disclosure has described several exemplary embodiments, there are modifications, arrangements, and various equivalent substitutions which are within the scope of the disclosure. It should thus be understood that those skilled in the art will be able to design various systems and methods which, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within the spirit and scope of the disclosure.

Claims

[Claim 1] A video decoding method executed by a decoder, comprising: decoding prediction information of a current block from an encoded video bitstream, the prediction information indicating an intra-frame block copy mode; determining a first portion of resolution syntax elements based on the intra-frame block copy mode, where for a current block, the resolution syntax elements have the same meaning for block vectors in the intra-frame block copy mode and motion vectors in an inter-frame prediction mode; decoding a second portion of the resolution syntax element from the encoded video bitstream; determining the block vector based on a target resolution indicated by a combination of the first and second portions; reconstructing at least one sample of the current block based on the block vector; The method includes:

Images