Method and apparatus for video coding

Abstract

Embodiments of the present disclosure provide a method and apparatus for video coding. A method for video decoding includes receiving a syntax element indicating whether to disable a prediction refinement using optical flow (PROF) for affine prediction; determining that a minimum value of absolute values of affine parameters of an affine coded block is lower than or equal to a predetermined threshold when the syntax element indicates that the PROF is not applied for the affine prediction; and determining that the minimum value of the absolute values of the affine parameters of the affine coded block is higher than the predetermined threshold when the syntax element indicates that the PROF is applied for the affine prediction.

Description

[0001] By citation and inclusion in this article

[0002] This application claims priority to U.S. Patent Application No. 16 / 822,075, filed March 18, 2020, entitled "Method and Apparatus for Video Coding and Decoding," which in turn claims priority to U.S. Provisional Application No. 62 / 820,196, filed March 18, 2019, entitled "Affine Inter-Frame Prediction Correction Method"; U.S. Provisional Application No. 62 / 828,425, filed April 2, 2019, entitled "LIC Signaling and Affine Correction"; and U.S. Provisional Application No. 62 / 838,798, filed April 25, 2019, entitled "Inter-Frame Prediction Correction Method." The disclosure of the earlier applications is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to embodiments of video encoding and decoding. Background Technology

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can comprise a series of pictures, each with spatial dimensions, for example, 1920×1080 luma samples and correlated chroma samples. The series of pictures can have a fixed or variable picture rate (also informally referred to as the frame rate), such as 60 pictures per second or 60Hz. Uncompressed video has very high bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920x1080 luma sample resolution at 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0005] One goal of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce bandwidth or storage requirements by two or more orders of magnitude in some cases. Lossless compression, lossy compression, and combinations of both can be used. Lossless compression refers to the technique of reconstructing an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of distortion tolerated depends on the application. For example, users of some consumer streaming applications may tolerate higher distortion than users of television applications. The achievable compression ratio reflects that higher allowable / tolerable distortion results in a higher compression ratio.

[0006] Motion compensation can be a lossy compression technique and may involve using sample data blocks from a previously reconstructed image or a portion of a reconstructed image (reference image) that are spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) for prediction of a newly reconstructed image or image portion. In some cases, the reference image may be the same as the image currently being reconstructed. MV may have two dimensions, X and Y, or three dimensions, where the third dimension represents the reference image being used (the latter may indirectly be a temporal dimension).

[0007] In some video compression techniques, the motion vector (MV) applied to a region of sample data can be predicted based on other MVs, such as those MVs associated with another region of sample data adjacent to the region being reconstructed and preceding that MV in the decoding order. This significantly reduces the amount of data required to encode and decode MVs, eliminating redundancy and increasing compression. MV prediction can be performed efficiently, for example, because when encoding and decoding the input video signal derived from the camera (called natural video), there is a statistical probability that regions larger than the area applicable to a single MV will move in a similar direction. Therefore, in some cases, predictions can be made using similar motion vectors derived from MVs of adjacent regions. This results in MVs found for a given region being similar to or identical to MVs predicted based on surrounding MVs, and after entropy encoding and decoding, they can be represented with fewer bits than when directly encoding and decoding MVs. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors that arise when calculating predictions based on several surrounding MVs.

[0008] H.265 / HEVC (ITU-T H.265 Recommendation, “Efficient Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this disclosure describes a technique hereinafter referred to as “spatial combining”.

[0009] Reference Figure 1 The current block (101) includes samples discovered by the encoder during motion search, which can be predicted based on previous blocks of the same size that have been spatially shifted. Instead of directly encoding and decoding the MV, the MV is derived from metadata associated with one or more reference images, for example, from the most recent (in decoding order) reference image, using the MV associated with any one of the five surrounding samples. The five surrounding samples are denoted as A0, A1 and B0, B1, B2 (102 to 106 respectively). In H.265, MV prediction can use predictions from the same reference image being used by adjacent blocks. Summary of the Invention

[0010] This disclosure provides methods and apparatus for video encoding / decoding.

[0011] In some examples, a video decoding method includes: receiving a syntax element indicating whether to disable prediction correction for optical flow (PROF) for affine prediction; when the syntax element indicates that the PROF is not applied to the affine prediction, determining that the minimum absolute value of the affine parameters of the affine coded block is less than or equal to a predetermined threshold; and when the syntax element indicates that the PROF is applied to the affine prediction, determining that the minimum absolute value of the affine parameters of the affine coded block is greater than the predetermined threshold.

[0012] For example, during the PROF process, a predicted sample I(i,j) can be generated at the sample position (i,j) in the affine coding block. A spatial gradient g can be generated at the sample position (i,j) in the affine coding block. x (i,j) and g y (i,j). This can be based on the spatial gradient g. x (i,j) and g y (i,j) generates a prediction correction ΔI(i,j). The prediction correction ΔI(i,j) can be added to the prediction sample I(i,j) to generate a corrected prediction sample.

[0013] In the example, a syntax element is received at the decoder indicating whether PROF is enabled for affine prediction. In the example, PROF is enabled by default for affine prediction.

[0014] In the example, the syntax element signals at the sequence level, slice level, tile level, tile group level, or picture level. In the example, based on the values ​​of the affine parameters of the affine model of the affine-coded block, it is determined whether to apply the PROF to the affine-coded block. In the example, based on one or more first predicted samples of a first sub-block including the predicted sample I(i,j) and one or more second predicted samples of a second sub-block adjacent to the first sub-block, the spatial gradient g is generated at the sample position (i,j). x (i,j) and g y (i,j). The first sub-block and the second sub-block are partitioned from the affine coded block. In the example, inter-frame prediction is performed on the sub-blocks of the affine coded block; and the spatial gradient g is generated at the sample position (i,j) based on the prediction samples of the entire affine coded block. x (i,j) and g y (i,j).

[0015] In the example, the spatial gradient g is generated at the sample location (i,j) using a gradient filter generated on a reference sample in the reference image of the affine coding block. x (i,j) and g y (i,j). For example, the generated gradient filter can be generated by convolving a first gradient filter with an interpolation filter. The interpolation filter is applied to a reference sample in a reference image of the affine coding block to generate a predicted sample of the affine coding block. Subsequently, the first gradient filter is applied to the generated predicted sample of the affine coding block to generate the spatial gradient g. x (i,j) and g y (i,j).

[0016] In the example, based on the received indication of whether the PROF is enabled for affine prediction and the signaling indicating whether Local Luminance Compensation (LIC) is enabled, it is determined whether to apply the PROF to the affine coded block. In the example, the affine coded block is encoded in an affine merging mode, and the LIC flag value of the affine coded block is inherited from neighboring blocks, which serve as the source for affine model inheritance or affine model construction of the affine coded block.

[0017] In some examples, a video decoding apparatus includes: a receiving module for receiving a syntax element indicating whether to disable prediction correction for optical flow (PROF) for affine prediction; and a determining module for determining that, when the syntax element indicates that the PROF is not applied to the affine prediction, the minimum absolute value of the affine parameters of the affine coding block is less than or equal to a predetermined threshold; and when the syntax element indicates that the PROF is applied to the affine prediction, the minimum absolute value of the affine parameters of the affine coding block is greater than the predetermined threshold.

[0018] In some examples, another video decoding apparatus may include processing circuitry configured to perform the video decoding method.

[0019] In the example, the intermediate motion vector MV of the current block is determined. block The intermediate motion vector MV block It has high accuracy compared to the low accuracy supported by the interpolation filter of the current block; for the intermediate motion vector MV block Perform rounding to generate the rounded motion vector MV of the current block. round The rounding motion vector MV round Low precision supported by the interpolation filter of the current block; based on the rounded motion vector MV roundThe interpolation filter is used to generate the prediction block for the current block. The prediction correction offset ΔI(i,j) can be determined according to the following formula:

[0020] ΔI(i,j)=g x (i,j)*Δv x +g y (i,j)*Δv y ,

[0021] Among them, g x (i,j) and g y (i,j) are the gradients of the predicted sample of the prediction block at position (i,j) along the x and y directions, respectively, and Δv x and Δv y These are the intermediate motion vectors MV block With the rounded motion vector MV round The difference in motion vectors along the x and y directions between them. The predicted correction offset ΔI(i,j) is added to the predicted samples of the predicted block to generate corrected predicted samples.

[0022] In the example, the current block is a block or sub-block encoded using a Temporal Motion Vector Prediction Mode (TMVP), a Sub-Block Based TMVP Mode (SbTMVP), Merging with Motion Vector Difference (MMVD), Affine Advanced Motion Vector Prediction (AMVP), or a Spatial Motion Vector Prediction Mode, wherein the intermediate motion vector MV block It is generated during the motion vector scaling operation, or the current block is encoded using motion vector prediction values ​​with the high precision.

[0023] In the example, syntax elements indicating the high precision can be received in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice Header, and Tile Group Header. In the example, the intermediate motion vector MV with high precision can be... block Stored in memory. The intermediate motion vector MV used for spatial motion vector prediction can be used. block Encode another block in the current image that includes the current block.

[0024] This disclosure also provides a non-volatile computer-readable medium for storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method of video decoding. Attached Figure Description

[0025] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, wherein:

[0026] Figure 1This is a schematic diagram of the current block and its surrounding space merge candidates in an example.

[0027] Figure 2 This is a simplified block diagram of a communication system according to an embodiment.

[0028] Figure 3 This is a simplified block diagram of a communication system according to an embodiment.

[0029] Figure 4 This is a simplified block diagram of the decoder according to an embodiment.

[0030] Figure 5 This is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0031] Figure 6 A block diagram of an encoder according to another embodiment is shown.

[0032] Figure 7 A block diagram of a decoder according to another embodiment is shown.

[0033] Figure 8 The candidate locations according to an embodiment are shown, from which a set of spatial merging candidates can be selected to construct an expanded merging candidate list.

[0034] Figure 9 Candidate pairs on an expanded merge list for a redundancy check process, according to an embodiment, are shown.

[0035] Figure 10 An example is shown of exporting time merge candidates from an expanded merge list in the current image according to an embodiment.

[0036] Figure 11 The candidate position according to an embodiment is shown, from which a time merge candidate on an expanded merge list can be selected.

[0037] Figure 12 The predicted location is shown according to an embodiment, from which predictions can be selected in a merging mode with motion vector difference (MMVD).

[0038] Figure 13A The two control point motion vectors (CPMV) used to represent the 2-parameter affine model are shown.

[0039] Figure 13B Three CPMVs are shown to represent a 3-parameter affine model.

[0040] Figure 14 The diagram shows the motion vectors derived for a sub-block, which is partitioned from the current block encoded in an affine prediction mode.

[0041] Figure 15 The adjacent blocks of the current block are shown for deriving the affine merge candidate of the inheritance.

[0042] Figure 16 The candidate block positions for affine merging candidates used to derive the construction are shown.

[0043] Figure 17A This is a schematic diagram of a spatially adjacent block according to one embodiment, which can be used to determine the predicted motion information of the current block based on the motion information of the spatially adjacent blocks using the sub-block-based temporal motion vector prediction (SbTMVP) method.

[0044] Figure 17B This is a schematic diagram of a selected spatially adjacent block for the SbTMVP method according to one embodiment.

[0045] Figure 18 This is a schematic diagram of an example of the Predictive Correction (PROF) method using optical flow according to an embodiment.

[0046] Figure 19 An example is shown of neighboring samples 1901 of the current block 1900 and neighboring samples of the corresponding reference block 1910 for deriving local brightness compensation (LIC) parameters.

[0047] Figure 20 An example of using Local Luminance Compensation (LIC) with bidirectional prediction is shown.

[0048] Figure 21 An example of using LIC with multiple hypothetical intra-frame and inter-frame connections is shown.

[0049] Figure 22 An example of a first method for obtaining a reference sample is shown.

[0050] Figure 23 An example of a second method for obtaining a reference sample is shown.

[0051] Figure 24 An example of a third method for obtaining reference samples based on sub-blocks is shown.

[0052] Figure 25 An example of an extended coding unit (CU) region corresponding to the CU (2510) used in bidirectional optical flow (BDOF) is shown.

[0053] Figure 26 Examples of kernels for Sobel, Scharr, Prewitt, and Roberts filters are shown.

[0054] Figure 27 A flowchart of an overview process (2700) according to an embodiment of the present disclosure is shown.

[0055] Figure 28 Another flowchart of an overview process (2800) according to an embodiment of the present disclosure is shown.

[0056] Figure 29 This is a schematic diagram of a computer system according to an embodiment. Detailed Implementation

[0057] I. Encoders and decoders used for video encoding and decoding

[0058] Figure 2 A simplified block diagram of a communication system (200) according to an embodiment of the present disclosure is shown. The communication system (200) includes a plurality of terminal devices that can communicate with each other via, for example, a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected via the network (250). Figure 2 In this embodiment, the first pair of terminal devices (210) and (220) perform one-way data transmission. For example, terminal device (210) may encode video data (e.g., a video image stream captured by terminal device (210)) for transmission over a network (250) to another terminal device (220). The encoded video data is transmitted in the form of one or more encoded video streams. Terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to recover the video data, and display video images based on the recovered video data. One-way data transmission is common in applications such as media services.

[0059] In another embodiment, the communication system (200) includes a second pair of terminal devices (230) and (240) performing bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (230) and (240) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission over a network (250) to the other terminal device (230) and (240). Each of the terminal devices (230) and (240) may also receive encoded video data transmitted by the other terminal device (230) and (240), and may decode the encoded video data to recover the video data, and may display the video images on an accessible display device based on the recovered video data.

[0060] exist Figure 2In the embodiments described herein, the terminal devices (210), (220), (230), and (240) may be servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) refers to any number of networks that transmit encoded video data between the terminal devices (210), (220), (230), and (240), including, for example, wired (connected) and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this disclosure, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation of this disclosure.

[0061] As an example of the disclosed subject matter, Figure 3 The placement of a video encoder and video decoder in a streaming environment is illustrated. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0062] The streaming system may include an acquisition subsystem (313) that may include a video source (301) such as a digital camera, which creates an uncompressed video image stream (302). In an embodiment, the video image stream (302) includes samples captured by a digital camera. The video image stream (302) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (304) (or encoded video bitstream). The video image stream (302) may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (302), the encoded video data (304) (or the encoded video bitstream (304)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (304) (or the encoded video bitstream (304)), which can be stored on a streaming server (305) for future use. One or more streaming client subsystems, such as Figure 3Client subsystems (306) and (308) can access a streaming server (305) to retrieve copies (307) and (309) of encoded video data (304). Client subsystem (306) may include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and produces an output video picture stream (311) that can be displayed on a display (312) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (304), video data (307), and video data (309) (e.g., video streams) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T TH.265. In embodiments, the video codec standard under development is informally referred to as Versatile Video Coding (VVC), and the disclosed subject matter can be used in the context of the VVC standard.

[0063] It should be noted that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0064] Figure 4 A block diagram of a video decoder (410) according to an embodiment of the present disclosure is shown. The video decoder (410) may be disposed in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., receiving circuitry). The video decoder (410) may be used in place of Figure 3 The video decoder (310) in the embodiment.

[0065] The receiver (431) may receive one or more encoded video sequences to be decoded by the video decoder (410); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (431) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter referred to as "parser (420)"). In some applications, the buffer memory (415) is part of the video decoder (410). In other cases, the buffer memory (415) may be located external to the video decoder (410) (not indicated). In other cases, an external buffer (not shown) may be provided for the video decoder (410) to prevent network jitter, for example, and another buffer (415) may be configured internally for, for example, handling broadcast timing. When the receiver (431) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer (415) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, a buffer (415) may be required; this buffer may be relatively large and adaptive in size, and may be at least partially implemented in the operating system or a similar component (not shown) external to the video decoder (410).

[0066] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (410) and potential information for controlling a display device (412) (e.g., a display screen), which is not part of the electronic device (430) but may be coupled to it, such as... Figure 4As shown in the figure. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels in the encoded video sequence for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (420) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0067] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421).

[0068] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser (420) from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser (420) and the various units described below is not described.

[0069] In addition to the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0070] The first unit is the scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives quantization transform coefficients as symbols (421) and control information from the parser (420), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (451) can output a block containing sample values, which can be input into the aggregator (455).

[0071] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) generates blocks of the same size and shape as the blocks being reconstructed, using surrounding reconstructed information extracted from the current picture buffer (458). For example, the current picture buffer (458) buffers partially reconstructed and / or fully reconstructed current images. In some cases, the aggregator (455) adds the predictive information generated by the intra-picture prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) based on each sample.

[0072] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (453) can access the reference image memory (457) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (421), these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit (453) can obtain the prediction samples from the address in the reference image memory (457) under motion vector control, and the motion vector is available to the motion compensation prediction unit (453) in the form of the symbols (421), which, for example, include X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0073] The output samples of the aggregator (455) can be employed by various loop filtering techniques in the loop filter unit (456). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (421) from the parser (420) in the loop filter unit (456). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0074] The output of the loop filter unit (456) can be a sample stream, which can be output to a display device (412) and stored in a reference image memory (457) for subsequent inter-frame image prediction.

[0075] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (420)) is identified as the reference image, the current image buffer (458) can become part of the reference image memory (457), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0076] The video decoder (410) can perform decoding operations according to a predetermined video compression technique, such as that specified in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0077] In this embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0078] Figure 5 A block diagram of a video encoder (503) according to an embodiment of the present disclosure is shown. The video encoder (503) is disposed in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used in place of Figure 3 The video encoder (303) in the embodiment.

[0079] The video encoder (503) can obtain data from the video source (501) (not) Figure 5 In one embodiment, a portion of the electronic device (520) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (503). In another embodiment, the video source (501) is a portion of the electronic device (520).

[0080] A video source (501) can provide a sequence of source video samples encoded by a video encoder (503) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (501) can be a storage device storing previously prepared video. In a video conferencing system, the video source (501) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0081] According to an embodiment, the video encoder (503) can encode and compress images of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (550). In some embodiments, the controller (550) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (550) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (550) may be used with other suitable functions related to the video encoder (503) optimized for a particular system design.

[0082] In some embodiments, the video encoder (503) operates within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (530) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (533) embedded within the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (534). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (534) also correspond bit-precisely between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0083] The operation of the “local” decoder (533) can be combined with, for example, the above-described method. Figure 4 The video decoder (410) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 4 When symbols are available and the entropy encoder (545) and parser (420) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (410), including the buffer (415) and parser (420), may not be fully implemented in the local decoder (533).

[0084] It can be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. For this reason, the subject matter disclosed focuses on decoder operation. The description of encoder techniques can be simplified, as encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0085] During operation, in some embodiments, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0086] The local video decoder (533) can decode encoded video data that can be designated as a reference picture, based on symbols created by the source encoder (530). The operation of the encoding engine (532) can be a lossy process. When the encoded video data can be decoded by the video decoder (530) Figure 5 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0087] The predictor (535) can perform a prediction search against the encoding engine (532). That is, for a new image to be encoded, the predictor (535) can search in the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (535) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (535), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (534).

[0088] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0089] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (545). The entropy encoder (545) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0090] The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission via a communication channel (560), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (540) can combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0091] The controller (550) manages the operation of the video encoder (503). During encoding, the controller (550) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0092] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0093] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0094] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0095] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0096] The video encoder (503) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (503) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0097] In this embodiment, the transmitter (540) may transmit additional data while transmitting encoded video. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0098] The acquired video can serve as multiple source images (video images) presented in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0099] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0100] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0101] According to some embodiments of this disclosure, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU comprises three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In embodiments, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Taking a luma prediction block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0102] Figure 6 A schematic diagram of a video encoder (603) according to another embodiment of the present disclosure is shown. The video encoder (603) is used to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode said processing blocks into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (603) is used instead of Figure 3The video encoder (303) in the embodiment.

[0103] In the HEVC embodiment, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (603) uses, for example, rate-distortion optimization to determine whether to use intra-frame mode, inter-frame mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra-frame mode, the video encoder (603) can use intra-frame prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (603) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some video coding techniques, the merging mode can be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector prediction values ​​without relying on encoded motion vector components outside the prediction values. In some other video coding techniques, motion vector components applicable to the subject block may exist. In the embodiment, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0104] exist Figure 6 In one embodiment, the video encoder (603) includes, as shown below: Figure 6 The inter-frame encoder (630), intra-frame encoder (622), residual calculator (623), switch (626), residual encoder (624), general controller (621) and entropy encoder (625) are shown coupled together.

[0105] An inter-frame encoder (630) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and later images), generate inter-frame prediction information (e.g., redundancy information description, motion vectors, merging mode information based on inter-frame coding techniques), and calculate inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image based on encoded video information.

[0106] The intra encoder (622) is used to receive samples of the current block (e.g., the processing block), in some cases compare the block with previously encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases also (e.g., based on intra prediction direction information of one or more intra coding techniques) generate intra prediction information. In an embodiment, the intra encoder (622) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0107] A general-purpose controller (621) determines general-purpose control data and controls other components of the video encoder (603) based on the general-purpose control data. In an embodiment, the general-purpose controller (621) determines the mode of a block and provides control signals to a switch (626) based on the mode. For example, when the mode is an intra-frame mode, the general-purpose controller (621) controls the switch (626) to select an intra-frame mode result for use by the residual calculator (623) and controls the entropy encoder (625) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the mode is an inter-frame mode, the general-purpose controller (621) controls the switch (626) to select an inter-frame prediction result for use by the residual calculator (623) and controls the entropy encoder (625) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0108] A residual calculator (623) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (622) or the inter encoder (630). A residual encoder (624) is used to operate on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (624) is used to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (622) and the inter encoder (630). For example, the inter encoder (630) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (622) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some embodiments, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0109] An entropy encoder (625) is used to format the bitstream to produce encoded blocks. The entropy encoder (625) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (625) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0110] Figure 7A schematic diagram of a video decoder (710) according to another embodiment of the present disclosure is shown. The video decoder (710) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (710) is used instead of Figure 3 The video decoder (310) in the embodiment.

[0111] exist Figure 7 In this embodiment, the video decoder (710) includes, as follows: Figure 7 The entropy decoder (771), inter-frame decoder (780), residual decoder (773), reconstruction module (774), and intra-frame decoder (772) are shown coupled together.

[0112] An entropy decoder (771) can be used to reconstruct certain symbols from an encoded image, these symbols representing the syntax elements constituting the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can respectively identify certain samples or metadata used by the intra-frame decoder (772) or the inter-frame decoder (780) for prediction, residual information in the form of, for example, quantized transform coefficients, and so on. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (780); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (772). Residual information may be provided to the residual decoder (773) via inverse quantization.

[0113] The inter-frame decoder (780) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0114] The intra-frame decoder (772) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0115] The residual decoder (773) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require some control information (to obtain the quantizer parameters QP), and this information may be provided by the entropy decoder (771) (the data path is not indicated because this is only low-level control information).

[0116] The reconstruction module (774) is used to combine the residual output by the residual decoder (773) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be a part of a reconstructed image, which in turn may be a part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0117] It should be noted that any suitable technology can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In one embodiment, one or more integrated circuits can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (303), video encoder (503), and video encoder (603), as well as the video decoder (310), video decoder (410), and video decoder (710).

[0118] II. Inter-frame prediction techniques

[0119] In various embodiments, motion parameters, including motion vectors, reference image indices, reference image list usage indices, and possibly other additional information, can be used to generate inter-frame prediction samples for the CU (Cumulative Unit) of inter-frame prediction. The motion parameters can be signaled explicitly or implicitly. When a CU is encoded using a skip mode, the CU is associated with a PU (Programming Unit) and there are no explicit residual coefficients, encoded motion vector increments, or reference image indices associated with that CU.

[0120] When using merge mode, motion parameters for the current CU can be obtained from neighboring CUs, including spatial and temporal merge candidates and optionally other merge candidates. Merge mode can be applied to CUs for inter-frame prediction and can also be used in skip mode. An alternative to merge mode is explicit transmission of motion parameters. For example, motion vectors, individual reference image indices for each reference image list, reference image list usage flags, and other necessary information can be explicitly signaled by each CU.

[0121] In some embodiments, the following inter-frame prediction codec tools are used:

[0122] - Extended merge forecast

[0123] - Merging mode with motion vector difference (MMVD)

[0124] - Advanced Motion Vector Prediction (AMVP) Mode with Symmetric Motion Vector Difference (MVD) Signaling

[0125] -Affine Motion Compensation Prediction

[0126] - Sub-block-based temporal motion vector prediction (SbTMVP)

[0127] -Adaptive Motion Vector Resolution (AMVR)

[0128] - Two-way forecasting with weighted average (BWA)

[0129] - Bidirectional optical flow (BDOF)

[0130] -Decoder-side motion vector correction (DMVR)

[0131] - Triangle partitioning prediction

[0132] - Combined inter-frame and intra-frame prediction (CIIP)

[0133] 1. Extended merged forecast

[0134] In some embodiments, the merged candidate list is constructed by sequentially including the following five types of candidates:

[0135] 1) Spatial motion vector predictions (SMVP) from spatially adjacent CUs;

[0136] 2) Time motion vector prediction values ​​(TMVP) from the co-located CU;

[0137] 3) Historical motion vector predictions (HMVP) from a first-in, first-out (FIFO) table;

[0138] 4) Pairwise average motion vector predictions (MVP); and

[0139] 5) Zero MV.

[0140] In some embodiments, the size of the merge list is signaled in the slice header, and the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is encoded using context, while the other bins are encoded using bypass coding.

[0141] The following is an example describing the generation process for each type of merge candidate.

[0142] 1.1 Spatial Candidate Derivation

[0143] During the process of exporting space merging candidates, from those located in Figure 8 Up to four merge candidates are selected from the candidates at positions A1, B1, B0, A0, and B2 near the current block (810). The derivation order is A1, B1, B0, A0, and B2. Position B2 is considered when any CU at positions A1, B1, B0, or A0 is unavailable (e.g., because it belongs to another slice or tile) or is intra-coded. After adding the candidate at position A1, the addition of the remaining candidates undergoes redundancy checking, which ensures that candidates with the same motion information are excluded from the list. This improves encoding and decoding efficiency. To reduce computational complexity, not all possible candidate pairs are considered in the aforementioned redundancy check. Instead, a subset of candidates is considered. Figure 9 The pairs are linked by arrows. When the corresponding candidate used for redundancy checking does not have the same motion information, the candidate is added to the list.

[0144] 1.2 Derivation of Time Candidates

[0145] In this embodiment, a time candidate is added to the list. Specifically, during the process of deriving the time merging candidate for the current block (1011) in the current image (1001), a scaled motion vector (1031) is derived based on the co-located CU (1012) belonging to the co-located reference image (1002), such as... Figure 10 As shown. The list of reference images used to derive the co-located CU is explicitly signaled in the slice header. The motion vector (1031) used for scaling the time merging candidates is scaled from the motion vector (1032) of the co-located CU (1012) using the picture order count (POC) distances Tb and Td. Tb is defined as the POC difference between the current reference image (1003) and the current image (1001) of the current image (1001). Td is defined as the POC difference between the co-located reference image (1004) and the co-located image (1002) of the co-located image (1002). The reference image index of the time merging candidate is set to zero.

[0146] exist Figure 11 The position of the temporal candidate is selected between candidates C0 and C1. If the CU at position C0 is unavailable, intra-coded, or outside the current line of the CTU, position C1 is used. Otherwise, position C0 is used in the derivation of the temporal merging candidate.

[0147] 1.3 Derivation of Pairwise Average Merge Candidates

[0148] In some embodiments, pairwise averaging candidates are generated by averaging predefined candidate pairs from an existing merge candidate list. For example, a predefined pair is defined as {(0,1),(0,2),(1,2),(0,3),(1,3),(2,3)}, where the numbers represent the merge index of the merge candidate list. The average motion vector is calculated separately for each reference list. If two motion vectors are available in a list, they are averaged even if they point to different reference images. If only one motion vector is available, that motion vector is used directly. If no motion vector is available, the list is invalidated.

[0149] In one embodiment, if the merge list is not full after adding pairwise average merge candidates, a zero MVP is inserted at the end of the merge list until the maximum number of merge candidates is reached.

[0150] 2. Merging mode with Motion Vector Difference (MMVD)

[0151] In addition to the merge mode (where implicitly derived motion information is directly used for generating prediction samples for the current CU), some embodiments use MMVD. The MMVD flag is signaled immediately after the skip and merge flags are sent to specify whether the MMVD mode should be used for the CU.

[0152] In MMVD mode, after selecting a merging candidate, the candidate is further refined using motion vector difference (MVD) information signaled via a signal, to obtain corrected motion information. MVD information includes a merging candidate flag, a distance index specifying the motion amplitude, and an index indicating the motion direction.

[0153] Select one of the top two candidates in the merge list as the MV base (one or more starting MVs). Signal the merge candidate flag to specify which candidate to use. Figure 12 As shown, the MV base determines the starting point (1211) or (1221) of the reference image (1202) or (1203) in the reference image list L0 or L1 respectively.

[0154] The distance index specifies the motion amplitude information and indicates a predefined offset from the starting point (1211) or (1221). For example... Figure 12 As shown, this offset is added to the horizontal or vertical component of the starting MV (MV base) pointing to position (1211) or (1221). Table 1 specifies the mapping relationship between distance indices and predefined offsets.

[0155] Table 1

[0156]

[0157] The direction index indicates the direction of MVD relative to the starting point (1211) or (1221). The direction index can represent one of the four directions shown in Table 2.

[0158] Table 2

[0159] Directional IDX 00 01 10 11 x-axis + - Not applicable (N / A) Not applicable (N / A) y-axis Not applicable (N / A) Not applicable (N / A) + -

[0160] It should be noted that the meaning of the MVD symbol can vary depending on the information of the starting MV. When one or more starting MVs are unidirectional or bidirectional prediction MVs that both point to the same side of the current image (i.e., the POCs of both reference images are greater than or less than the POC of the current image), the symbols in Table 2 specify the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV and the two MVs point to different sides of the current image (i.e., the POC of one reference image is greater than the POC of the current image, while the POC of the other reference image is less than the POC of the current image), the symbols in Table 2 specify the sign of the MV offset added to the L0 MV component of the starting MV, while the signs of the L1 MV have the opposite value.

[0161] Based on the base MV, offset, and MVD symbols, one or more final MVs can be determined for the current CU.

[0162] 3. Affine Motion Compensation Prediction

[0163] In some examples, translational motion models are applied to motion compensation prediction (MCP). However, translational motion models may not be suitable for modeling other types of motion (e.g., zooming in / out, rotation, perspective motion, and other irregular motions). In some embodiments, block-based affine transformation motion compensation prediction is applied. Figure 13A In this context, when using a 4-parameter affine model, the affine motion field of a block is described by two control points (CP) CP0 and CP1, and two control point motion vectors (CPMV0 and CPMV1). Figure 13B In the 6-parameter affine model, the affine motion field of the block is described by the three control point motion vectors (CPMV) of three control points (CP0, CP1, and CP2) CPMV0, CPMV1, and CPMV2.

[0164] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as follows:

[0165]

[0166] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as follows:

[0167]

[0168] In expressions (2.1) and (2.2), (mv 0x ,mv 0y (mv) is the motion vector of the top-left control point. 1x ,mv 1y (mv) is the motion vector of the upper right control point. 2x ,mv 2y ) is the motion vector of the lower left control point. Additionally, the coordinates (x, y) are about the upper left corner of each block, and W and H represent the width and height of each block.

[0169] To simplify motion compensation prediction, some embodiments employ sub-block-based affine transformation prediction. For example, in Figure 14 In this study, a 4-parameter affine motion model was used, and two CPMVs were determined. and To derive the motion vector for each 4×4 (sample) luma sub-block (1402) divided from the current block (1410), the motion vector (1401) of the center sample of each sub-block (1402) is calculated according to the above expression (2.1) and rounded to 1 / 16 fractional precision. Then, a motion-compensated interpolation filter is applied to generate a prediction for each sub-block (1402) using the derived motion vector (1401). The sub-block size for the chroma component is set to 4×4. The MV of the 4×4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4×4 luma sub-blocks.

[0170] Similar to translational motion inter-frame prediction, some embodiments employ two affine motion inter-frame prediction modes: affine merging mode and affine AMVP mode.

[0171] 3.1 Affine Merging Prediction

[0172] In some embodiments, the affine merging pattern can be applied to CUs with a width and height greater than or equal to 8. Affine merging candidates for the current CU are generated based on the motion information of spatially adjacent CUs. There can be a maximum of five affine merging candidates, and an index is signaled to indicate which affine merging candidate should be used for the current CU. For example, the following three types of affine merging candidates are used to form the affine merging candidate list:

[0173] (i) Inherited affine merge candidates inferred from the CPMV of adjacent CUs;

[0174] (ii) Affine merging candidates for the construction derived using the translation MV of adjacent CUs; ​​and

[0175] (iii) Zero MV

[0176] In some embodiments, there can be up to two inherited affine candidates derived from the affine motion models of neighboring blocks, one derived from the left neighboring CU and the other from the top neighboring CU. The candidate block, for example, can be located in... Figure 8 The positions are shown. For the left predictor, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first inherited candidate is selected from each side. No pruning check is performed between candidates from two inherited sides.

[0177] When neighboring affine CUs are identified, their CPMVPs are used to derive CPMVP candidates from the affine merging list of the current CU. For example... Figure 15 As shown, the adjacent lower left block A of the current CU (1510) is encoded in affine mode to obtain the motion vectors of the upper left, upper right, and lower left corners of the CU (1520) containing block A. and When encoding block A using a 4-parameter affine model, according to and Calculate the two CPMVs of the current CU(1510). and When encoding block A using a 6-parameter affine model, according to and Calculate the three CPMVs (not shown) of the current CU.

[0178] The constructed affine candidates are built by combining the translational motion information of adjacent control points. The motion information of the control points comes from... Figure 16 The specified spatial and temporal adjacent blocks are derived as shown. CPMVk (k = 1, 2, 3, 4) represents the k-th control point. For CPMV1, blocks B2->B3->A2 are checked sequentially, and the MV of the first available block is used. For CPMV2, blocks B1->B0 are checked, and for CPMV3, blocks A1->A0 are checked. If the TMVP at block T is available, it can be used as CPMV4.

[0179] After obtaining the motion MVs of the four control points, affine merging candidates are constructed based on this motion information. The following combinations of control point MVs are used to construct the candidates sequentially: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0180] Combining three CPMVs constructs a 6-parameter affine merge candidate, while combining two CPMVs constructs a 4-parameter affine merge candidate. To avoid issues during motion scaling, combinations of control point MVs are discarded if the reference indices of the control points are different.

[0181] After checking the affine merge candidates for inheritance and the affine merge candidates for construction, if the list is still not full, zero MV is inserted at the end of the merge candidate list.

[0182] 3.2 Affine AMVP Prediction

[0183] In some embodiments, the affine AMVP mode can be used for CUs with a width and height greater than or equal to 16. A flag at the CU level is signaled in the bitstream to indicate whether the affine AMVP mode is used, followed by a flag indicating whether a 4-parameter or 6-parameter affine mode is used. The difference between the current CU's CPMV and its predicted CPMVP is signaled in the bitstream. The affine AMVP candidate list is 2^n in size and can be generated by sequentially using the following four types of CPMV candidates:

[0184] (i) Inherited affine AMVP candidates inferred from the CPMV of adjacent CUs;

[0185] (ii) Affine AMVP candidates derived using translation MV of adjacent CUs;

[0186] (iii) Translation MV from adjacent CU; and

[0187] (iv) Zero MV.

[0188] In the example, the checking order of inherited affine AMVP candidates is similar to the checking order of inherited affine merge candidates. According to the embodiment, the difference is that, for AMVP candidates, affine CUs with the same reference image as the reference image in the current block are considered. No pruning process is applied when the inherited affine motion prediction values ​​are inserted into the candidate list.

[0189] Constructed AMVP candidates from Figure 16The specified spatially adjacent blocks are exported. The same checking order as in the affine merge candidate construction is used. Additionally, the reference picture indices of adjacent blocks are checked. The first block in the checking order, which is inter-frame coded and has the same reference picture as in the current CU, is used. When the current CU is coded using a 4-parameter affine model and both CPMV0 and CPMV1 are available, these available CPMVs are added as candidates to the affine AMVP candidate list. When the current CU is coded using a 6-parameter affine model and all three CPMVs (CPMV0, CPMV1, and CPMV2) are available, these available CPMVs are added as candidates to the affine AMVP candidate list. Otherwise, the constructed AMVP candidate is set to unavailable.

[0190] If the affine AMVP candidate list is still less than 2 after checking the inherited and constructed affine AMVP candidates, the translational motion vector adjacent to the control point (if available) is added to predict the MV of all control points in the current CU. Finally, if the affine AMVP candidate list is still not full, it is filled with zero MV.

[0191] 4. Sub-block-based temporal motion vector prediction (SbTMVP)

[0192] Figure 17A This is a schematic diagram of a spatially adjacent block according to one embodiment, which can be used to determine the predicted motion information of the current block (1711) using the sub-block-based temporal MV prediction (SbTMVP) method. Figure 17A The current block (1711) and its spatially adjacent blocks, represented as A0, A1, B0, and B1 (represented as 1712, 1713, 1714, and 1715, respectively), are shown. In some examples, the spatially adjacent blocks A0, A1, B0, and B1 belong to the same picture as the current block (1711).

[0193] Figure 17B This is a schematic diagram illustrating the use of the SbTMVP method according to an embodiment to determine motion information of the sub-blocks of the current block (1711) based on selected spatially adjacent blocks (e.g., block A1 in this non-limiting example). In this example, the current block (1711) is located in the current image (1710), the reference block (1761) is located in the reference image (1760), and can be identified based on the motion shift (or displacement) between the current block (1711) and the reference block (1761) indicated by the motion vector (1722).

[0194] In some embodiments, similar to Temporal Motion Vector Prediction (TMVP) in HEVC, SbTMVP uses motion information from various reference sub-blocks in a reference image for the current block in the current image. In some embodiments, the same reference image used by TMVP can be used for SbTMVP. In some embodiments, TMVP predicts motion information at the CU level, while SbTMVP predicts motion information at the subCU level. In some embodiments, TMVP uses temporal motion vectors from co-located blocks in the reference image, having a corresponding position adjacent to the lower right corner or center of the current block, and SbTMVP uses temporal motion vectors from reference blocks, which can be identified by performing motion shifts based on the motion vectors of one of the spatially neighboring blocks of the current block.

[0195] For example, such as Figure 17A As shown, adjacent blocks A1, B1, B0, and A0 can be checked sequentially during the SbTMVP process. Once a first spatially adjacent block (whose motion vector uses the reference picture (1760) as a reference picture) is identified, such as block A1, which has a motion vector (1722) pointing to, for example, reference block AR1 in the reference picture (1760), this motion vector (1722) can be used to perform a motion shift. If no such motion vector is available for spatially adjacent blocks A1, B1, B0, and A0, the motion shift is set to (0, 0).

[0196] After determining the motion displacement, the reference block (1761) can be identified based on the position of the current block (1711) and the determined motion displacement. Figure 17B In this context, the reference block (1761) can be further divided into 16 sub-blocks with reference motion information MRa to MRp. In some examples, the reference motion information for each sub-block in the reference block (1761) can be determined based on the minimum motion grid covering the center sample of that sub-block. The motion information may include motion vectors and corresponding reference indices. The current block (1711) can be further divided into 16 sub-blocks, and in some examples, the motion information MVa to MVp of the sub-blocks in the current block (1711) can be derived from the reference motion information MRa to MRp through time scaling in a manner similar to a TMVP process.

[0197] The sub-block size used in the SbTMVP process can be fixed (or otherwise predetermined) or signaled. In some examples, the sub-block size used in the SbTMVP process can be 8×8 samples. In some examples, the SbTMVP process applies only to blocks with a width and height equal to or greater than the fixed size or the signaled size (e.g., 8 pixels).

[0198] In the example, a sub-block-based merge list containing a combination of SbTMVP candidates and affine merge candidates is used for signaling in the sub-block-based merge mode. The SbTMVP mode can be enabled or disabled via the Sequence Parameter Set (SPS) flag. In some examples, if the SbTMVP mode is enabled, the SbTMVP candidate is added as the first entry in the list of sub-block-based merge candidates, followed by affine merge candidates. In some embodiments, the maximum allowed size of the sub-block-based merge list is set to 5. However, other sizes can be used in other embodiments.

[0199] In some embodiments, the encoding logic for additional SbTMVP merge candidates is the same as that for other merge candidates. That is, for each block in a P-slice or B-slice, an additional rate distortion check may be performed to determine whether to use an SbTMVP candidate.

[0200] 5. Prediction Correction Using Optical Flow (PROF)

[0201] Affine motion model parameters can be used to derive the motion vector for each pixel in the CU. However, due to the high complexity of performing pixel-based affine prediction and the memory access bandwidth requirements, sub-block-based affine motion compensation methods are implemented in some embodiments. In some embodiments, the current block (e.g., CU) can be divided into multiple sub-blocks, each of size 4×4, and assigned a sub-block MV derived from the CPMV of the current block. Sub-block-based affine motion compensation is a trade-off between improving encoding / decoding efficiency, complexity, and memory access bandwidth, with the trade-off being reduced prediction accuracy.

[0202] In some embodiments, a prediction correction using optical flow (PROF) method can be implemented to improve sub-block-based affine motion compensation to have a finer granularity of motion compensation. According to the PROF method, after performing sub-block-based affine motion compensation, the predicted sample (e.g., luminance or chromaticity) can be corrected by adding a set of adjustment values ​​derived from the optical flow equation.

[0203] Figure 18 This is a schematic diagram illustrating an example of the PROF method according to an embodiment. Figure 18 In the example shown, the current block (1810) is divided into four sub-blocks (1812, 1814, 1816, and 1818). Each of the sub-blocks (1812, 1814, 1816, and 1818) has a size of 4×4 pixels. The sub-block MV (1820) of the sub-block (1812) located at the center of the sub-block (1812) can be derived from the CPMV (representing the affine model) of the current block 1810 and points to the reference sub-block (1832). The initial sub-block prediction sample can be determined based on the reference sub-block (1832).

[0204] The correction value to be applied to the initial sub-block prediction samples can be calculated as if each prediction sample were located at the position indicated by pixel MV (e.g., pixel MV(1840)) (e.g., position (1832a) of sample (1812a)). For example, pixel MV(1840) can be calculated based on an affine model represented by CPMV. The difference (called the adjustment vector) ΔV(1842) between pixel MV(1840) and sub-block MV(1820) can be determined accordingly. Subsequently, using an optical flow method, the correction value corresponding to pixel (1812a) can be determined based on the adjustment vector ΔV(1842).

[0205] In the example, the PROF method can begin by performing sub-block-based affine motion compensation to generate initial sub-block predictions (also known as initial prediction sub-blocks) (e.g., block (1832)), which consist of samples denoted by I(i,j), where i and j are the coordinates of a specific sample corresponding to position (i,j) in the current sub-block. Next, the spatial gradient g of the initial sub-block prediction at each sample position (i,j) can be computed using, for example, a 3-tap filter [-1, 0, 1] according to the following formula. x (i,j) and g y (i,j),

[0206] g x (i,j)=I(i+1,j)-I(i-1,j) (Equation 3)

[0207] g y (i,j)=I(i,j+1)-I(i,j-1) (Equation 4)

[0208] Sub-block prediction can be extended by one pixel on each side for gradient calculation. In some embodiments, to reduce memory bandwidth and complexity, pixels on the extended boundary can be copied from the nearest integer pixel position in the reference image (e.g., the reference image including block (1832)). This avoids additional interpolation of the filled region.

[0209] The prediction correction can then be calculated using the following optical flow equation:

[0210] ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j), (Equation 5)

[0211] Here, Δv(i,j) (e.g., ΔV(1842)) is the difference between the pixel MV (e.g., pixel MV(1840)) of the sample position (i,j) represented by v(i,j) and the sub-block MV (e.g., sub-block MV 1820) of the sub-block to which pixel position (i,j) belongs. Since the affine model parameters and the pixel position relative to the sub-block center do not change between sub-blocks, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. In some examples, let x and y be the horizontal and vertical positions of Δv(i,j) relative to the sub-block center, Δv(i,j) can be derived by the following equation:

[0212]

[0213] Where, Δv x (x,y) is the x-component of Δv(i,j), αv y (x,y) is the y component of Δv(i,j).

[0214] For a 4-parameter affine model

[0215]

[0216] For a 6-parameter affine model

[0217]

[0218] Among them, (v 0x ,v 0y ), (v 1x ,v 1y ) and (v 2x ,v 2y ) are the motion vectors of the top left, top right, and bottom left control points, and w and h are the width and height of the current block, including the current child block.

[0219] Finally, the prediction correction ΔI(i,j) can be added to the initial sub-block prediction sample I(i,j). The final corrected prediction sample I' according to the PROF method can be generated using the following equation:

[0220] I′(i,j)=I(i,j)+ΔI(i,j). (Equation 9)

[0221] 6. Prediction Correction Based on Local Luminance Compensation (LIC)

[0222] 6.1 Local brightness compensation

[0223] Local Luminance Compensation (LIC) is a block-level coding mode used in some embodiments. In some examples, LIC can be turned on and off at the coding unit (CU) level.

[0224] LIC is based on a linear model of brightness variation, using a scaling factor 'a' and an offset 'b'. LIC predictions based on traditional inter-frame prediction can be generated according to the following formula:

[0225] Pred LIC = a·Pred+b, (Equation 10)

[0226] Among them Pred LIC denoted as LIC prediction, Pred as conventional inter-frame prediction represented by motion vectors, and a and b as scaling factor and offset, respectively.

[0227] Final Reconstruction of LIC Mode LIC The following can be exported:

[0228] Rec LIC =Pred LIC +Res=a·Pred+b+Res, (Equation 11)

[0229] Where Res is the residual of the current block (after dequantization and inverse transformation).

[0230] When the LIC mode is applied to the CU, the least square error method is used. Parameters a and b are derived using the neighboring samples of the current CU and their corresponding reference samples according to the following formula.

[0231] {a,b}=argmin{(Rec x,y -Ref x′,y′ ) 2 (Equation 12)

[0232] Among them, Rec x,y Represents the adjacent reconstructed samples of the current block, Ref x′,y′ The Rec in the reference image to which the motion vector of the current block points. x,y The corresponding sample.

[0233] Figure 19 An example is shown of neighboring samples (1901) of the current block (1900) and neighboring samples of the corresponding reference block (1910) used to derive LIC parameters a and b. As shown, the neighboring samples (1901) of the CU (1900) and the corresponding sample (1911) in the reference image are obtained using a 2:1 secondary sampling of the CU (1900) (identified by motion information from the current CU (1900) (or sub-CU)). In this example, the LIC parameters can be derived and applied to each prediction direction separately.

[0234] When the CU encodes in merge mode, the LIC flag can be copied from one of the adjacent blocks in a manner similar to motion information copying in merge mode; otherwise, the CU can be signaled to the LIC flag to indicate whether the LIC should be applied.

[0235] When LIC mode is enabled for an image, an additional CU-level rate-distortion (RD) check can be performed to determine whether LIC mode is applied to the CU. In the example, when LIC is enabled for the CU, for integer pixel motion search and fractional pixel motion search, the mean-removed sum of absolute difference (MR-SAD) and the mean-removed sum of absolute Hadamard-transformed difference (MR-SATD) are used instead of the sum of absolute difference (SAD) and the sum of absolute Hadamard-transformed difference (SATD), respectively.

[0236] 6.2 Unidirectional brightness compensation

[0237] In some embodiments, an improved LIC method, called unidirectional lumen compensation, is employed. For example, in unidirectional lumen compensation, the derivation of the linear model parameters can remain unchanged. The improved LIC is applied based on CU. The improved LIC is not applied to sub-block-based inter-frame prediction (e.g., Advanced Temporal Motion Vector Prediction (ATMVP) or affine prediction), triangular partitioning, multi-hypothesis intra-inter-frame, and bidirectional prediction.

[0238] Figure 20 An example of using LIC with bidirectional prediction is shown. First, unidirectional prediction blocks (2001)-(2002) in directions L0 and L1 can be obtained separately. Then, modified blocks (2003)-(2004) can be obtained by performing LIC using reconstructed neighboring samples (2005). Finally, a weighted average can be performed based on the modified blocks (2003)-(2004) to generate a bidirectional prediction block (2006).

[0239] and Figure 20 Conversely, in the improved approach, in some examples, LIC is not applied to bidirectional prediction because the reconstructed neighboring samples of the current block (2005) do not require inter-frame prediction to be performed in the inter-frame pipeline, and therefore cannot be used for every unidirectional inter-frame prediction; otherwise, LIC would need to be applied to bidirectional prediction. Furthermore, applying LIC to bidirectional prediction introduces an additional stage to perform the LIC process before the weighting operation.

[0240] Figure 21An example of using LIC in conjunction with multi-hypothesis intra- and inter-frame mapping is shown. First, inter-frame prediction blocks (2101) and intra-frame prediction blocks (2102) can be generated. LIC can be performed to obtain a correction block (2103) based on the inter-frame prediction blocks (2101). Finally, an inter- and intra-frame hybrid block (2104) can be generated by performing a weighted average on the intra-frame coded blocks (2102) and the correction blocks (2103).

[0241] and Figure 21 Conversely, in some examples, because LIC processing is applied after inter-frame prediction and before weighted averaging, which delays the weighting between intra-frame and inter-frame prediction blocks, LIC processing is not applied to multi-hypothesis intra-inter-frame.

[0242] In merge mode, the LIC flag can be included as part of the motion information, in addition to the MV and reference index. For example, when constructing the merge candidate list, the LIC flag can be inherited from neighboring blocks of the merge candidate. In some examples, for simplicity, the LIC flag is not used for motion vector pruning. In one example, the LIC flag is not stored in the motion vector buffer of the reference image; therefore, for TMVP mode, the LIC flag can be set to false. In another example, the LIC flag is set to false for bidirectional merge candidates (e.g., pairwise averaging candidates and zero-motion candidates). In yet another example, the LIC flag is not signaled when the LIC tool is not applied.

[0243] 6.3 Applying LIC processing for affine prediction

[0244] In some embodiments, the LIC can be extended to affine-coded CUs. For example, the derivation of the linear model parameters remains unchanged, and several methods can be used to obtain reference samples of the affine-coded CUs.

[0245] Figure 22 An example of a first method for obtaining a reference sample is shown. The current CU (2201) in the current picture (2210) can be an affine coded block. The co-occurring CU (2202) and reference CU (2203) of the current CU (2201) are shown in the reference picture (2220). The motion vector (MV) (2231) of the top-left sub-block of the affine coded CU (2201) can be used to obtain a reference sample for the entire CU (2201).

[0246] Figure 23An example of a second method for extracting reference samples is shown. The current CU (2301) in the current image (2310) can be an affine-coded block. The co-occurring CU (2302) and reference CU (2303) of the current CU (2301) are shown in the reference image (2320). The motion vector (MV) (2331) of the central sub-block of the affine-coded CU (2301) can be used to obtain a reference sample of the entire CU (2301).

[0247] Figure 24 An example of a third method for obtaining reference samples based on sub-blocks is shown. The current CU (2401) in the current image (2410) is an affine coded block. The co-occurring CU (2402) of the current CU (2401) is shown in the reference image (2420). As shown, the corresponding reference blocks from A Ref to G Ref in the reference image (2420) can be determined using the sub-blocks MV from MVa to MVg in the left column and top row of the current CU (2401). The reference samples adjacent to those reference blocks can be used to derive the linear model parameters.

[0248] 7. Prediction Correction Based on Bidirectional Optical Flow (BDOF)

[0249] In some embodiments, a bidirectional optical flow (BDOF) tool is employed. Compared to another method used in JEM (called BIO), BDOF is a simpler method that requires less computation, particularly in terms of the number of multiplications and the size of the multipliers.

[0250] BDOF can be used to correct the bidirectional prediction signal of the CU at the 4×4 sub-block level. In the example, BDOF is applied to the CU when the following conditions are met: (1) the height of the CU is not 4 and the size of the CU is not 4×8; (2) the CU is not encoded using affine mode or ATMVP merging mode; (3) the CU is encoded using "true" bidirectional prediction mode, that is, one of the two reference images is displayed before the current image in the order of display, and the other is displayed after the current image in the order of display. In the example, BDOF is applied only to the luma component.

[0251] The BDOF model is based on the concept of optical flow, which assumes that the motion of the object is smooth. For each 4×4 sub-block, motion correction (v) is calculated by minimizing the difference between the L0 and L1 prediction samples. x ,v y Then, motion correction is used to adjust the bidirectional prediction sample values ​​in the 4×4 sub-blocks. The following steps are applied in the example BDOF process.

[0252] First, the horizontal gradient of the two predicted signals is calculated by directly calculating the difference between two adjacent samples. and vertical gradient Right now

[0253]

[0254] Among them, I (k) (i,j) is the sample value at coordinate (i,j) of the predicted signal in list k (k=0,1). shift1 is calculated based on the luminance bit depth bitDepth, and is shift1=max(2,14-bitDepth).

[0255] Then, the autocorrelation and cross-correlation of the gradients S1, S2, S3, S5, and S6 are calculated as follows:

[0256]

[0257] in,

[0258]

[0259] Where Ω is the 6×6 window surrounding the 4×4 sub-block, n a and n b The values ​​are set to min(5, bitDepth-7) and min(8, bitDepth-4), respectively.

[0260] Then, using cross-correlation and autocorrelation terms, the motion correction (v) is derived through the following equation. x ,v y ):

[0261]

[0262] in, th′ BIO =2 13-BD ,and It is the floor function.

[0263] Based on motion correction and gradient, the following adjustments are calculated for each sample in the 4×4 sub-block:

[0264]

[0265] Finally, the BDOF samples of CU are calculated as follows by adjusting the bidirectional prediction samples:

[0266] pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset >> shift (Equation 18)

[0267] In the example, these values ​​are chosen so that the multiplier in the BDOF process does not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits.

[0268] In some examples, to derive gradient values, it is necessary to generate some predicted samples I from a list k (k = 0, 1) outside the current CU boundary. (k) (i,j). Figure 25 An example of an extended CU region corresponding to the CU(2510) used in BDOF is shown. As shown, BDOF uses an extended row / column around the boundary of CU(2510). To control the computational complexity of generating prediction samples outside the boundary, prediction samples in the extended region (non-shaded location) are generated by directly taking reference samples at nearby integer positions (using the floor() operation on the coordinates) without interpolation, and a normal 8-tap motion-compensated interpolation filter is used to generate prediction samples within CU(2510) (shaded location). These extended sample values ​​are used only for gradient calculation. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, they are padded (i.e., repeated) based on their nearest neighbors.

[0269] III. PROF for Affine Prediction

[0270] 1. Conditional application of PROF in affine prediction

[0271] In some embodiments, PROF can be enabled always for affine prediction, which increases the complexity of the encoder or decoder. Conversely, in some other embodiments, PROF can be conditionally enabled or disabled for affine prediction to improve performance and reduce the complexity of the encoder or decoder. For example, PROF can be conditionally applied to affine coded blocks based on affine parameter values.

[0272] As mentioned above, the affine parameters can be calculated using CPMV in the following way. Let x and y be the horizontal and vertical offsets from the pixel position to the center of the sub-block in the current CU, and the adjusted MVΔv(x,y) can be derived using the following equation.

[0273]

[0274] For a 4-parameter affine model

[0275]

[0276] For a 6-parameter affine model

[0277]

[0278] Among them, (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) is the motion vector of the upper left, upper right, and lower left control points of the current CU, and w and h are the width and height of the current CU.

[0279] In one embodiment, for an affine-coded block, if the maximum absolute value of the affine parameters a, b, c, or d (denoted as max_parameter = max{|a|,|b|,|c|,|d|}) is greater than or equal to a predetermined threshold, PROF for affine coding is not applied to the affine-coded block. Otherwise, if max_parameter is less than the threshold, PROF can be applied to the affine-coded block.

[0280] In another embodiment, for an affine-coded block, if the minimum absolute value of the affine parameters a, b, c, or d (denoted as min_parameter = min{|a|,|b|,|c|,|d|}) is less than or equal to a predetermined threshold, PROF for affine coding is not applied to the affine-coded block. Otherwise, if min_parameter is greater than the threshold, PROF can be applied to the affine-coded block.

[0281] In another embodiment, for an affine coded block, a PROF for affine mapping can be applied to the affine coded block when the absolute value of any of the affine parameters a, b, c, or d is within a predetermined range. Otherwise, if any absolute value of the affine parameter is not within the predetermined range, no PROF is applied to the affine coded block. The value range can be represented as [M, N], where M and N are positive numbers. In one example, M and N are predefined or signaled in the bitstream, for example, in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), tile group header, tile header, slice header, etc.

[0282] In another embodiment, the use (enabling or disabling) of PROF for affine-coded blocks can be determined, for example, by a flag (represented by prof_flag). This flag can be signaled at the block level. Therefore, PROF can be enabled when the flag is true.

[0283] In another embodiment, the syntax element can be used to determine whether PROF is enabled for affine prediction. In the example, this syntax element can be a high-level syntax element. For example, the syntax element can signal at the sequence level, strip level, tile level, tile group level, picture level, etc.

[0284] 2. Gradient calculation of PROF

[0285] In some embodiments, gradient computation can be performed on a sub-block basis during the PROF process. For example, inter-frame prediction (including interpolation operations during inter-frame prediction) can be performed for each affine sub-block (e.g., for each 4x4 sub-block used for affine prediction). The gradient is then computed using a 3-tap filter [1 0 -1] based on the prediction output of the affine sub-block. Subsequently, the sub-block is also corrected. To compute the gradient of each affine sub-block, the affine sub-block can be padded to a larger size, for example, by extending it by 1 pixel on each side.

[0286] 2.1 Gradient calculation for the entire block

[0287] In one embodiment, gradient computation in PROF can be performed on the entire coded block once inter-frame prediction (interpolation) of all affine sub-blocks is complete. This avoids expanding each sub-block for gradient computation. Instead, the entire coded block can be expanded (padded) by one pixel on each side used for gradient computation, which is simpler than expanding each sub-block. Each sub-block can be corrected based on the gradients of its individual sub-blocks, which are obtained from computations based on the entire block.

[0288] For filling the entire block, the following methods can be used in various embodiments: (1) generating pixels on the extended boundary by interpolating the motion vectors of the corresponding sub-blocks. (2) copying pixels on the extended boundary from the nearest integer pixel position in the reference image. (3) copying pixels on the extended boundary from the nearest sample of the inter-frame prediction output of the current block.

[0289] 2.2 Generating a gradient filter from an interpolation filter

[0290] In some embodiments, the interpolation filter used for affine prediction and the gradient filter used for PROF gradient computation can be combined to generate a combined gradient filter. This combined gradient filter can be directly applied to samples in the reference image of the current coding block to generate the corresponding PROF gradient. In this way, the gradient computation process can be executed in parallel with the corresponding interpolation process to accelerate PROF processing.

[0291] In one embodiment, a 10-tap gradient filter can be generated by convolving a PROF gradient filter (e.g., taps of [1 0 -1]) with an 8-tap interpolation filter for inter-frame prediction. Therefore, the generated 10-tap gradient filter can be used to directly perform gradient calculations on reference samples.

[0292] In another embodiment, a 10-tap gradient filter can be generated by convolving the PROF gradient filter with an 8-tap interpolation filter for inter-frame prediction, and then truncated to an 8-tap filter by removing one coefficient from each side. Therefore, the generated 8-tap gradient filter can be used to directly perform gradient calculations on the reference sample.

[0293] In another embodiment, an 8-tap gradient filter can be generated by convolving the PROF gradient filter with a 6-tap interpolation filter used for inter-frame prediction. Therefore, the generated 8-tap gradient filter can be used to directly perform gradient calculations on the reference samples.

[0294] In another embodiment, an N-tap gradient filter can be generated by convolving an M-tap PROF gradient filter with an L-tap interpolation filter for inter-frame prediction, and then truncated to an N-tap filter, where N is less than or equal to M+L. Therefore, the generated N-tap gradient filter can be used to directly perform gradient calculations on reference samples.

[0295] IV. Interaction between PROF and LIC

[0296] 1. Conditional application of PROF based on LIC flags in affine prediction

[0297] When both PROF and LIC are applied to an affine-coded CU, the correction performance of the affine-coded CU may not improve further compared to applying either PROF or LIC, and may even become worse. Therefore, in some embodiments, when PROF is enabled for affine prediction, the decision to enable or disable LIC may be considered to determine whether PROF can be applied. PROF can be enabled for affine prediction by default, or it can be enabled through advanced syntax, such as signaling at the sequence level (e.g., SPS), picture level (e.g., PPS), slice level, tile level, tile group level, CTU level, etc.

[0298] In one embodiment, PROF can be enabled for an affine-coded block when LIC is signaled to be disabled at a given high level (e.g., sequence level (e.g., SPS), picture level (e.g., PPS), slice level, tile / tile group level, etc.).

[0299] In another embodiment, PROF can be enabled for affine-coded blocks regardless of whether LIC is applied. In one example, sample adjustments from the LIC are applied before PROF. In another example, sample adjustments from the LIC are applied after PROF.

[0300] In another embodiment, some blocks have a signaled LIC flag indicating whether LIC is applied to individual blocks when the LIC tool is enabled by default or signaled at a higher level (e.g., sequence level (e.g., SPS), picture level (e.g., PPS), slice level, tile / tile group level, etc.). In such a scenario, in the example, if LIC is signaled at the block level for a block with affine inter-frame prediction, it is inferred that PROF is disabled for that block. Otherwise, if LIC is disabled for an affine-coded block, it is inferred that PROF is enabled for that block.

[0301] 2. Inheritance of the LIC flag in affine merge

[0302] When encoding the current block in affine merge mode, the LIC flag value corresponding to the current block can be determined in various ways.

[0303] In one embodiment, affine merging by inheritance is used on the current block. Therefore, the LIC flag of the current block can be inherited from the LIC flag of the block used as the source of affine model inheritance.

[0304] In another embodiment, a constructed affine merge is used on the current block. The LIC flag can be inherited from the first available source candidate of a neighboring block used to construct the affine control point motion vector (CPMV) of the current block. k (k = 1, 2, 3, 4). In the example, when the current block uses... Figure 16 In the example described, when constructing an affine model with 3 control point motion vectors, the combination of CPMVs is {CPMV1, CPMV2, CPMV3}. The adjacent blocks used to construct CPMV1 can be used as sources for LIC flag inheritance. In another example, when the current block uses... Figure 16 When constructing an affine model with two control point motion vectors, the combination of CPMVs is {CPMV1, CPMV3}. The adjacent blocks used to construct CPMV1 can be used as sources for LIC flag inheritance.

[0305] In another embodiment, a constructed affine merge is used on the current block. If at least two source candidates for the constructed affine CPMV have the same LIC flag value, that LIC flag value can be applied to the current block. Otherwise, if only two control points are used for the constructed affine merge and their LIC flags are different, the LIC can be set to be disabled for the current block.

[0306] In another embodiment, for affine merge mode, it can be inferred that the LIC flag of the current block is disabled (e.g., has a value of 0).

[0307] In another embodiment, when using an inherited affine merge, the LIC flag of the current block can be inherited from the LIC flags of adjacent blocks. Otherwise, if the selected merge candidate is not an inherited affine merge candidate, it can be inferred that the LIC flag of the current block is disabled.

[0308] V. Perform PROF-like correction on blocks with high-precision intermediate motion vectors.

[0309] In TMVP mode or sub-block-based TMVP (SbTMVP) mode, scaling operations can be performed on the temporally predicted prediction MV to generate intermediate MVs. The intermediate MVs can have higher accuracy than the accuracy supported by the interpolation filters used to generate the prediction blocks (e.g., 1 / 16 of a sample). The intermediate MVs can be cropped or rounded to generate final MV predictions with the supported accuracy. Therefore, the final MV predictions may lose accuracy due to rounding operations compared to the intermediate MVs.

[0310] In some embodiments, to improve encoding and decoding efficiency, intermediate high-precision MVs are used as the final MV predictions for TMVP or SbTMVP. However, this comes at the cost of more interpolation stages and potential additional memory requirements for storing high-precision MVs.

[0311] To address this issue, in some embodiments, higher accuracy can be achieved for the intermediate MV (composed of MV) compared to the accuracy supported by existing interpolation filters. block Blocks (represented by) use PROF-like corrections. Examples of coding modes to which PROF-like corrections can be applied include TMVP, SbTMVP, Spatial MV Prediction, MMVD, AMVP, etc. In those coding modes, scaling operations can be performed to generate MV predictions, or candidate MVs with higher precision can be used as MV predictions. High-precision intermediate MVs in those coding modes can be rounded to the precision supported by the interpolation filter and subsequently used in motion compensation to generate prediction blocks. The difference between the high-precision intermediate MV and the precision MV supported by interpolation can be used to correct prediction blocks generated by motion compensation.

[0312] In the example, a correction process similar to PROFP may include the following steps:

[0313] Step 1: Forming a MV round The prediction block Pred, the MV round This represents a rounding to the highest precision supported by existing interpolation filters.

[0314] Step 2: Calculate the PROF-like prediction correction offset ΔI(i,j) for each sample location in the current block. An example of ΔI(i,j) is derived by the following formula:

[0315] ΔI(i,j)=g x (i,j)*Δv x +g y (i,j)*Δv y (Equation 22)

[0316] Among them, g x (i,j) and g y (i,j) represent the gradients of the predicted Pred along the x and y directions at position (i,j), respectively, and Δv x and Δv y It's a music video. block With MV round The difference in MV between them.

[0317] Step 3: Add ΔI(i,j) to the inter-frame prediction Pred. In some examples, the resulting pixels will be cropped to the target range if necessary.

[0318] In an embodiment, when the current block has the same motion vector as the entire block (e.g., sub-blocks with the same motion vector), the aforementioned PROC-like correction process is applied to the entire block. When the current block has different motion vectors at the sub-block level, such as blocks encoded using SbTMVP, the aforementioned PROC-like correction process is applied to each individual sub-block.

[0319] In this embodiment, MV can be predefined. block The precision, or the signaling of MV within the bitstream (e.g., in the sequence parameter set (e.g., SPS), picture parameter set (e.g., PPS), slice header, tile group header, etc.). block The precision. Alternatively, when using BDOF, the same precision used for BDOF can be used. For example, it can be MV. block Predefined 1 / 32, 1 / 24, or 1 / 64 of the sample.

[0320] In some embodiments, the gradient of Pred can be predicted using a 3-tap filter [-1, 0, 1] as follows:

[0321] g x (i,j)=I(i+1,j)-I(i-1,j)

[0322] g y (i,j)=I(i,j+1)-I(i,j-1), (Equation 23)

[0323] Where I(i,j) indicates the block prediction Pred at position (i,j).

[0324] To compute the gradient, in one embodiment, to reduce memory bandwidth and complexity, pixels on the extended boundary can be copied from the nearest integer pixel location in the reference image. This avoids additional interpolation for filling the region. In another embodiment, the extended prediction sample can be generated by additional interpolation from the corresponding reference sample. In one embodiment, the gradient accuracy is based on the bit depth of the source video. In another embodiment, a Sobel filter can be used to compute the gradients in the x and y directions. In another embodiment, a Scharr filter can be used to compute the gradients in the x and y directions. In another embodiment, a Prewitt filter can be used to compute the gradients in the x and y directions. In another embodiment, a Roberts filter can be used to compute the gradients in the x and y directions. Figure 26 Examples of kernels for Sobel, Scharr, Prewitt, and Roberts filters are shown.

[0325] In one embodiment, when storing the MV of the current image, a higher-precision MV (higher than the highest precision supported by the interpolation filter) can be stored and used for spatial MV prediction. In another embodiment, when storing the MV of the current image into a frame buffer (which will be used as the TMVP for other images), the higher-precision MV can be rounded to the highest precision supported by the interpolation filter before MV compression.

[0326] In some embodiments, a PROF-like correction process may not be used with certain coding modes (e.g., BDOF, DMVR, triangle merging, intra-frame / inter-frame mode, etc.).

[0327] VI. Examples of PROF procedures or similar PROF-like correction procedures

[0328] 1. Example of the PROF procedure

[0329] Figure 27 A flowchart of an overview process (2700) according to an embodiment of the present disclosure is shown. Process (2700) can be used to reconstruct blocks encoded in an affine prediction mode (e.g., affine merge mode, affine AMVP, etc.), thereby generating prediction blocks for the blocks being reconstructed. In various embodiments, process (2700) is executed by processing circuitry, such as processing circuitry in terminal devices (210), (220), (230), and (240), processing circuitry performing the functions of a video decoder (310), processing circuitry performing the functions of a video decoder (410), etc. In some embodiments, process (2700) is implemented as software instructions, so that when the processing circuitry executes these software instructions, the processing circuitry executes process (2700). The process begins at (S2701) and proceeds to (S2710).

[0330] In S2710, a syntax element is received indicating whether PROF is enabled for affine prediction. In the example, this syntax element signals at the sequence level, slice level, tile level, tile group level, or picture level. In another example, PROF can be enabled for affine prediction by default without signaling this syntax element.

[0331] In S2720, it is determined whether to apply PROF to the affine-coded block. In the example, the application of PROF to the affine-coded block is determined based on the values ​​of the affine parameters of the affine model of the affine-coded block. In the example, the application of PROF to the affine-coded block is determined based on the received syntax element indicating whether PROF is enabled for affine prediction and the signaling indicating whether LIC is enabled.

[0332] In S2730, when it is determined that PROF will be applied to the affine coding block, the PROF procedure is executed. In the example, the PROF procedure may include generating a predicted sample I(i,j) at sample position (i,j) in the affine coding block, and generating a spatial gradient g at sample position (i,j) in the affine coding block. x (i,j) and g y (i,j), based on spatial gradient g x (i,j) and g y (i,j) generates the prediction correction ΔI(i,j) and adds the prediction correction ΔI(i,j) to the prediction sample I(i,j) to generate the corrected prediction sample.

[0333] In the example, a spatial gradient g is generated at sample location (i,j) based on one or more first predicted samples from a first sub-block including predicted sample I(i,j) and one or more second predicted samples from a second sub-block adjacent to the first sub-block. x (i,j) and g y (i,j). The first and second sub-blocks can be sub-blocks divided from affine-coded blocks.

[0334] In the example, a gradient filter generated on a reference sample in the reference image of the affine coded block is used to generate a spatial gradient g at sample location (i,j). x (i,j) and g y (i,j). For example, the generated gradient filter is generated by convolving the first gradient filter with the interpolation filter. For example, assuming that the generated gradient filter is not used, the interpolation filter can be applied to the reference sample in the reference image of the affine coding block to generate the predicted sample of the affine coding block. Subsequently, the first gradient filter can be applied to the generated predicted sample of the affine coding block to generate the spatial gradient g. x (i,j) and g y(i,j). The generated gradient filter makes the interpolation (or prediction) of the affine coded block and the spatial gradient g x (i,j) and g y The generation of (i,j) can be performed in parallel.

[0335] In the example, the affine coding block is encoded in an affine merging mode. The LIC flag value of the affine coding block can be inherited from neighboring blocks, which serve as the source for affine model inheritance or affine model construction. Process 2700 can be executed up to S2799 and ends at S2799.

[0336] 2. An example of a correction process similar to PROF

[0337] Figure 28 A flowchart of an overview process (2800) according to an embodiment of this disclosure is shown. Process (2800) can be used to reconstruct a block with an intermediate MV having a higher accuracy than that supported by the interpolation filters used in motion compensation, to generate a prediction of the block. In various embodiments, process (2800) is executed by processing circuitry, such as processing circuitry in terminal devices (210), (220), (230), and (240), processing circuitry performing the functions of a video decoder (310), processing circuitry performing the functions of a video decoder (410), etc. In some embodiments, process (2800) is implemented as software instructions, so that when the processing circuitry executes these software instructions, the processing circuitry executes process (2800). The process begins at (S2801) and proceeds to (S2810).

[0338] In S2810, the intermediate motion vector MV of the current block can be determined. block Intermediate motion vector MV block It offers high precision compared to the low precision supported by the interpolation filter of the current block. The current block can be a block or sub-block encoded using TMVP, SbTMVP, MMVD, affine AMVP, spatial motion vector prediction mode, etc. Therefore, the intermediate motion vector MV... block It can be generated during the motion vector scaling operation. Alternatively, the current block can be encoded using high-precision motion vector predictions.

[0339] In the example, syntax elements indicating high precision can be received in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), slice header, tile group header, etc. Intermediate motion vector MV block It may have a higher precision when initially generated than the high precision notified by the signal, and is rounded to the high precision notified by the signal.

[0340] In S2820, the intermediate motion vector MV blockPerform rounding to generate the rounded motion vector MV of the current block. round Rounding motion vector MV round It features low precision supported by the interpolation filter of the current block.

[0341] In S2830, it is possible to base the rounded motion vector MV round And interpolation filters, to generate the prediction block for the current block.

[0342] In S2840, the predicted correction offset ΔI(i,j) can be determined according to the following formula:

[0343] ΔI(i,j)=g x (i,j)*Δv x +g y (i,j)*Δv y (Equation 24)

[0344] Among them, g x (i,j) and g y (i,j) represent the gradients of the predicted sample of the prediction block at position (i,j) along the x and y directions, respectively, and Δv x and Δv y These are the intermediate motion vectors MV block With rounded motion vector MV round The difference in motion vectors along the x and y directions between them.

[0345] In S2850, the predicted correction offset ΔI(i,j) is added to the predicted samples of the prediction block to generate corrected predicted samples. In the example, the intermediate motion vector MV with high accuracy is used. block Stored in memory. The intermediate motion vector MV used for spatial motion vector prediction can be used. block Encode another block in the current image that includes the current block. Process 2800 can be executed up to S2899 and ends at S2899.

[0346] VII. Computer System

[0347] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 29 A computer system (2900) is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0348] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, etc.

[0349] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0350] Figure 29 The components shown for the computer system (2900) are exemplary in nature and are not intended to limit the scope or functionality of the computer software implementing embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer system (2900).

[0351] The computer system (2900) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0352] Human-machine interface input devices may include one or more of the following (only one is shown): keyboard (2901), mouse (2902), touchpad (2903), touch screen (2910), data glove (not shown), joystick (2905), microphone (2906), scanner (2907), camera (2908).

[0353] The computer system (2900) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2910), data gloves (not shown), or joystick (2905), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2909), headphones (not shown)), visual output devices (e.g., screens (2910) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which may output two-dimensional or more three-dimensional visual outputs by means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0354] The computer system (2900) may also include human-accessible storage devices and related media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (2920) or similar media (2921), thumb drives (2922), removable hard disk drives or solid-state drives (2923), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0355] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0356] The computer system (2900) may also include interfaces to one or more communication networks. For example, the network may be wireless, wired, or optical. The network may also be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a vehicle-mounted network and an industrial network, a real-time network, a latency-tolerant network, and so on. Networks also include Ethernet, wireless LANs, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle-mounted and industrial networks (including CANBus), and so on. Some networks typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses (2949) (e.g., the USB port of the computer system (2900); other systems are typically integrated into the core of the computer system (2900) via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (2900) can communicate with other entities. The communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as through a local area or wide area digital network to other computer systems. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0357] The aforementioned human-computer interface device, human-accessible storage device, and network interface can be connected to the core (2940) of the computer system (2900).

[0358] The core (2940) may include one or more central processing units (CPU) (2941), graphics processing units (GPUs) (2942), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (2943), task-specific hardware accelerators (2944), etc. These devices, along with read-only memory (ROM) (2945), random access memory (2946), and internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives, etc.) (2947), can be connected via a system bus (2948). In some computer systems, the system bus (2948) may be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (2948) or connected via a peripheral bus (2949). Peripheral bus architectures include External Peripheral Component Interconnect (PCI), Universal Serial Bus (USB), etc.

[0359] The CPU (2941), GPU (2942), FPGA (2943), and accelerator (2944) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (2945) or RAM (2946). Transient data can also be stored in RAM (2946), while permanent data can be stored, for example, in internal mass storage (2947). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2941), GPUs (2942), mass storage (2947), ROM (2945), RAM (2946), etc.

[0360] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0361] By way of example and not limitation, a computer system having an architecture (2900), particularly a core (2940), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the non-volatile core (2940), such as internal mass storage (2947) or ROM (2945). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core (2940). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2940), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2946) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide logic hardwired or otherwise incorporated into circuitry (e.g., an accelerator (2944)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing the execution logic, or both. This disclosure includes any suitable combination of hardware and software.

[0362] Appendix A: Acronyms

[0363] AMVP: Advanced MVP

[0364] ASIC: Application-Specific Integrated Circuit

[0365] BDOF: Bi-directional Optical Flow

[0366] BMS: Benchmark set

[0367] CANBus: Controller Area Network Bus

[0368] CD: Compact Disc

[0369] CPUs: Central Processing Units

[0370] CRT: Cathode Ray Tube

[0371] CTBs: Coding Tree Blocks

[0372] CTUs: Coding Tree Units

[0373] CU: Coding Unit

[0374] DVD: Digital Video Disc

[0375] FPGA: Field Programmable Gate Array

[0376] GOPs: Groups of Pictures

[0377] GPUs: Graphics Processing Units

[0378] GSM: Global System for Mobile Communications

[0379] HEVC: High Efficiency Video Coding

[0380] HMVP: History-Based MVP (Motion Vector Prediction)

[0381] HRD: Hypothetical Reference Decoder

[0382] IC: Integrated Circuit

[0383] JEM: Joint Exploration Model

[0384] LAN: Local Area Network

[0385] LCD: Liquid Crystal Display

[0386] LIC: Local illumination compensation

[0387] LTE: Long-Term Evolution

[0388] MMVD: Merge with MVD

[0389] MV: Motion Vector

[0390] MVD: Motion Vector Difference

[0391] MVP: Motion vector predictor

[0392] OLED: Organic Light-Emitting Diode

[0393] PBs: Prediction Blocks

[0394] PCI: Peripheral Component Interconnect

[0395] PLD: Programmable Logic Device

[0396] PROF: Prediction refinement with optical flow

[0397] PUs: Prediction Units

[0398] RAM: Random Access Memory

[0399] ROM: Read-Only Memory

[0400] SbTMVP: Sub-block-based TMVP

[0401] SEI: Supplementary Enhancement Information

[0402] SNR: Signal-to-Noise Ratio

[0403] SSD: Solid-state drive

[0404] TMVP: Temporal Motion Vector Prediction

[0405] Tus: Transform Units

[0406] USB: Universal Serial Bus

[0407] VTM: Versatile test model

[0408] VUI: Video Usability Information

[0409] VVC: Versatile Video Coding

[0410] While this disclosure has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are within the scope of this disclosure. Therefore, it should be understood that those skilled in the art can design various systems and methods that, while not explicitly shown or described herein, embody the principles of this disclosure and are thus within its spirit and scope.

Claims

1. A method for video decoding, characterized in that, include: Receive syntax elements that indicate whether to disable the prediction correction PROF using optical flow for affine prediction; When the syntax element indicates that the PROF is not applied to the affine prediction, it is determined that the minimum of the absolute values ​​of the affine parameters of the affine coding block is less than or equal to a predetermined threshold. When the syntax element indicates that the PROF is applied to the affine prediction, it is determined that the minimum of the absolute values ​​of the affine parameters of the affine coding block is higher than the predetermined threshold.

2. The method according to claim 1, characterized in that, PROF is enabled by default for the affine prediction.

3. The method according to claim 1, characterized in that, The syntax elements signal notifications at the sequence level, slice level, tile level, tile group level, or picture level.

4. The method according to claim 1, characterized in that, Further includes: When the syntax element indicates that the PROF is applied to the affine prediction. In the prediction sample Sample positions in the affine coding block Generate spatial gradient and , Based on the spatial gradient and Generate predicted correction values ,as well as The predicted correction value Added to the prediction sample This is used to generate corrected prediction samples.

5. The method according to claim 4, characterized in that, At the sample location The spatial gradient is generated at the location. and include: Based on the predicted samples The first predicted sample of the first sub-block and the second predicted sample of the second sub-block adjacent to the first sub-block, at the sample location The spatial gradient is generated at the location. and The first sub-block and the second sub-block are divided from the affine coding block.

6. The method according to claim 4, characterized in that, At the sample location The spatial gradient is generated at the location. include: Perform inter-frame prediction on sub-blocks of the affine coded block; and The spatial gradient is generated at the sample location based on the predicted sample of the affine coding block. .

7. The method according to claim 4, characterized in that, At the sample location The spatial gradient is generated at the location. and include: Using a gradient filter generated on a reference sample in the reference image of the affine coded block, at the sample location The spatial gradient is generated at the location. and .

8. The method according to claim 7, characterized in that, The generated gradient filter is produced by convolving the first gradient filter with the interpolation filter. Specifically, the interpolation filter is applied to the reference samples in the reference image of the affine coding block to generate predicted samples of the affine coding block. Then, the first gradient filter is applied to the generated predicted samples of the affine coding block to generate the spatial gradient. and .

9. The method according to claim 7, characterized in that, The gradient filter is generated by convolution of a PROF gradient filter with taps of [1 0 -1] and an 8-tap interpolation filter for inter-frame prediction.

10. The method according to claim 7, characterized in that, The gradient filter is obtained by convolving a PROF gradient filter with taps of [1 0 -1] and an 8-tap interpolation filter for inter-frame prediction, and then truncating it into an 8-tap filter by removing one coefficient from each side.

11. The method according to claim 7, characterized in that, An N-tap gradient filter is generated by convolving an M-tap PROF gradient filter and an L-tap interpolation filter for inter-frame prediction, and then truncated to an N-tap filter, where N is less than or equal to M+L, thus obtaining the gradient filter.

12. The method according to claim 1, characterized in that, The step of determining whether to apply the PROF to the affine coding block includes: Based on the received indication of whether the syntax element of the PROF is disabled for affine prediction and the signaling indicating whether the Local Luminance Compensation (LIC) is enabled, it is determined whether to apply the PROF to the affine coding block.

13. The method according to claim 1, characterized in that, The affine coding block is encoded in an affine merging mode, and the LIC flag value of the affine coding block is inherited from the neighboring blocks of the affine coding block. The neighboring blocks are used as the source for the affine model inheritance or affine model construction of the affine coding block.

14. A video decoding apparatus, characterized in that, include: A receiving module is configured to receive a syntax element indicating whether to disable the prediction correction PROF using optical flow for affine prediction. A determining module is configured to determine that when the syntax element indicates that the PROF is not applied to the affine prediction, the minimum absolute value of the affine parameters of the affine coding block is lower than or equal to a predetermined threshold; and when the syntax element indicates that the PROF is applied to the affine prediction, the minimum absolute value of the affine parameters of the affine coding block is higher than the predetermined threshold.

15. The apparatus according to claim 14, characterized in that, It further includes a generation module for: When the syntax element indicates that the PROF is applied to the affine prediction. In the prediction sample Sample positions in the affine coding block Generate spatial gradient and , Based on the spatial gradient and Generate predicted correction values ,as well as The predicted correction value Added to the prediction sample This is used to generate corrected prediction samples.

16. The apparatus according to claim 15, characterized in that, The generation module is used for: Based on the predicted samples The first predicted sample of the first sub-block and the second predicted sample of the second sub-block adjacent to the first sub-block, at the sample location The spatial gradient is generated at the location. and The first sub-block and the second sub-block are divided from the affine coding block.

17. The apparatus according to claim 15, characterized in that, The generation module is used for: Perform inter-frame prediction on sub-blocks of the affine coded block; as well as The spatial gradient is generated at the sample location based on the predicted sample of the affine coding block. .

18. The apparatus according to claim 15, characterized in that, The generation module is used for: Using a gradient filter generated on a reference sample in the reference image of the affine coding block, at the sample location The spatial gradient is generated at the location. and .

19. A video decoding device, characterized in that, Includes circuitry configured to perform the video decoding method according to any one of claims 1-13.

20. A video encoding method, characterized in that, include: Determine the syntax element that indicates whether to disable the prediction correction PROF using optical flow for affine prediction; Determine the minimum value among the absolute values ​​of the affine parameters of the affine-coded block; When the minimum value of the absolute values ​​of the affine parameters of the affine coding block is less than or equal to a predetermined threshold, the syntax element indicates that the PROF should not be applied to the affine prediction; When the minimum value of the absolute values ​​of the affine parameters of the affine coding block is higher than the predetermined threshold, the syntax element indicates that the PROF is applied to the affine prediction; The syntax elements are encoded into a bitstream.

21. A method for storing a bit stream, characterized in that, The method of video encoding according to claim 20 is used to generate a bitstream; and the bitstream is stored.

22. A method for transmitting a bit stream, characterized in that, The method of video encoding according to claim 20 is used to generate a bitstream; and the bitstream is transmitted.

23. A non-volatile computer-readable medium, characterized in that, Used to store instructions that, when executed by a computer for video encoding and decoding, cause the computer to perform the method according to any one of claims 1-13 and 20.

Images