Motion compensated picture boundary filling
By using motion compensation filling blocks and inter-frame prediction technology in video encoding, the boundary filling process of video encoding is optimized, and the problem of low encoding efficiency in the prior art is solved, and more efficient data compression and transmission is achieved.
Patent Information
- Application Number
- CN202380015105.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-30
- Filing Date
- 2023-10-31
- Publication Date
- 2024-11-22
AI Technical Summary
The existing video encoding technology is difficult to effectively handle picture boundary filling during motion compensation, resulting in low encoding efficiency and increased data transmission bandwidth.
By introducing a motion compensation fill (MCP) block into video encoding, the pixel values in the fill area are determined using the motion vectors in multiple candidate blocks, and encoding using inter-prediction reference frames, the boundary fill process is optimized.
It improves the encoding efficiency of video encoding, reduces data transmission bandwidth, and enhances the compression performance of video signals.
Smart Images

Figure CN119013983A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims the benefit of priority of U.S. non-provisional patent application No. 18 / 497,748, filed on October 30, 2023, entitled “Motion Compensated Picture Boundary Filling”, which is based on and claims the benefit of priority of U.S. Provisional Application No. 63 / 452,651, filed on March 16, 2023, entitled “About Motion Compensated Picture Boundary Filling”. The entire disclosure of the above prior application is incorporated herein by reference. Technical Field
[0003] The present disclosure relates generally to video coding, and more particularly to methods and systems for motion compensated boundary filling. Background Art
[0004] Uncompressed digital video may include a series of pictures and may impose specific bit rate requirements on storage, data processing, and transmission bandwidth in streaming applications. One purpose of video encoding and decoding may be to reduce redundancy in the uncompressed input video signal through various compression techniques. Summary of the invention
[0005] The present disclosure generally relates to video coding, and more particularly to a method and system for border filling for motion compensation. For example, a motion vector of a motion compensation filling block in a motion compensation filling area of a picture in a video can be determined by scanning a set of adjacent border blocks of the picture and / or a set of co-located time blocks of the border block. The motion compensation filling block can be filled according to the motion vector and used for motion compensation of other pictures in the video.
[0006] In an exemplary implementation, a method for encoding a first picture of a video based on a second picture of the video in a bitstream is disclosed. The method may include: determining a motion compensation padding (MCP) block in a MCP area around a picture boundary of the second picture; selecting a candidate block from a plurality of candidate blocks, the plurality of candidate blocks being temporal blocks or spatial neighboring blocks of the MCP block at the picture boundary of the second picture; deriving a motion vector from the selected candidate block of the MCP block, the motion vector pointing to a reference block in a reference frame of the second picture; filling the MCP block with pixel values according to the reference block; and using the second picture as an inter-frame prediction reference frame of the first picture, and using the MCP block in the second picture for motion compensation to encode the first picture in the bitstream.
[0007] In another exemplary implementation, a method for encoding a first picture of a video based on a second picture of the video in a bitstream is disclosed. The method may include: determining a motion compensation padding (MCP) region around a picture boundary of the second picture; identifying an MCP block within the MCP region; determining motion information of a set of inter-frame minimum blocks at the picture boundary of the second picture; determining a motion vector of the MCP block based on the motion information; filling the MCP block with pixel values based on the motion vector; and using the second picture as an inter-frame prediction reference frame of the first picture and using the MCP block for motion compensation to inter-frame encode the first picture in the bitstream.
[0008] In any of the above exemplary implementations, the multiple candidate blocks include N1 spatial neighboring blocks and N2 time blocks of the MCP block, where N1 and N2 are positive integers.
[0009] In any of the above exemplary implementations, the N2 temporal blocks include N2 temporal sub-blocks, and the N2 temporal sub-blocks are located at N2 predetermined positions in a reference frame relative to the MCP block. In some exemplary implementations, N2=2, and the N2 temporal sub-blocks in the reference frame are co-located with the center sub-block and the lower right sub-block of the MCP block of the second picture. In some exemplary implementations, N1 spatial neighboring blocks are scanned before scanning the N2 temporal blocks to derive the motion vector of the MCP block of the second picture.
[0010] In any of the above exemplary implementations, deriving the motion vector of the MCP block includes: identifying a first candidate block having an available motion vector from a plurality of candidate blocks.
[0011] In any of the above exemplary implementations, the MCP area corresponds to a plurality of rectangles, each rectangle extending outward from one of the picture boundaries of the second picture in units of pixels to an MCP range consisting of a predetermined number of pixels.
[0012] In any of the above exemplary implementations, all or a subset of the plurality of candidate blocks include a set of neighboring blocks of the MCP block in the second picture at a corresponding boundary of the second picture, wherein each neighboring block in the set of neighboring blocks has a predetermined block size, and the set of neighboring blocks corresponds to a predetermined relative position relative to the MCP block in the second picture. In some exemplary implementations, deriving a motion vector for the MCP block includes: identifying a first candidate block having an available motion vector from the plurality of candidate blocks according to a predetermined scanning order. In some exemplary implementations, all or a subset of the plurality of candidate blocks also include one or more blocks in a reference frame of the second picture that are temporally co-located with a boundary block in the second picture corresponding to the MCP block. In some exemplary implementations, the one or more blocks in the reference frame of the second picture that are temporally co-located with a boundary block in the second picture corresponding to the MCP block include one or more sub-blocks of the boundary block, and the one or more sub-blocks are located at a set of predetermined positions within the boundary block relative to a middle sub-block of the boundary block.
[0013] In any of the above exemplary implementations, deriving the motion vector of the MCP block includes: identifying a first candidate block with an available motion vector from multiple candidate blocks according to a predetermined scanning order, wherein the group of adjacent blocks is first scanned, and then one or more blocks that are temporally co-located with the boundary blocks in the second picture corresponding to the MCP block are scanned until the first candidate block is identified.
[0014] In any of the above exemplary implementations, the method may further include signaling in a bitstream that the candidate block is selected from a plurality of candidate blocks.
[0015] In any of the above exemplary implementations, a binary file of the candidate index is used to signal the selection of a candidate block from first two candidate blocks of the plurality of candidate blocks.
[0016] In any of the above exemplary implementations, variable length coding is used to signal one candidate block among three available candidate blocks among multiple candidate blocks.
[0017] In any of the above exemplary implementations, when only one candidate block among multiple candidate blocks has an available motion vector, the selection of the candidate block is implicit and is not signaled in the bitstream.
[0018] In any of the above exemplary implementations, the signaling of candidate block selection for all MCP blocks of the second picture is aggregated in a predetermined MCP block scanning order.
[0019] In any of the above exemplary implementations, the method may further include determining an offset based on a first DC value of the selected candidate block and a second DC value of a reference block in a reference frame of the candidate block; and applying the offset when filling the MCP block.
[0020] Aspects of the present disclosure also provide an electronic device or apparatus, including a circuit or a processor, wherein the circuit or the processor is configured to execute any of the above method implementations.
[0021] Aspects of the present disclosure also provide a non-transitory computer-readable medium, which stores instructions. When an electronic device executes the instructions, the electronic device executes any one of the above-mentioned method implementations. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:
[0023] Figure 1 A schematic diagram showing a simplified block diagram of a communication system (100) according to an example embodiment;
[0024] Figure 2 A schematic diagram showing a simplified block diagram of a communication system (200) according to an example embodiment;
[0025] Figure 3 A schematic diagram showing a simplified block diagram of a video decoder according to an example embodiment;
[0026] Figure 4 A schematic diagram showing a simplified block diagram of a video encoder according to an example embodiment;
[0027] Figure 5 shows a block diagram of a video encoder according to another example embodiment;
[0028] Figure 6 shows a block diagram of a video decoder according to another example embodiment;
[0029] Figure 7 The figure shows the positions of spatial merging candidates according to one embodiment of the present disclosure.
[0030] Figure 8 Candidate pairs considered for redundancy check of spatial merging candidates according to one embodiment of the present disclosure are shown.
[0031] Fig. 9 Exemplary motion vector scaling of temporal merging candidates is shown.
[0032] Fig.10 Exemplary candidate positions (eg, C0 and C1) of temporal merging candidates for the current coding unit are shown.
[0033] Figures 11A to 11B Affine motion fields in some examples are shown.
[0034] Fig.12 Block-based affine transform prediction in some examples is shown.
[0035] Fig.13 Control point motion vectors are shown in some examples.
[0036] Fig.14 Spatial and temporal neighbors are shown in some examples.
[0037] Fig.15 An exemplary diagram showing the difference between a sample motion vector and a sub-block motion vector.
[0038] Figures 16 to 17 An exemplary subblock-based temporal motion vector prediction (SbTMVP) process used in SbTMVP mode is shown.
[0039] Fig.18 Extended coding unit areas in some examples are shown.
[0040] Fig.19 An exemplary schematic diagram illustrating decoder-side motion vector correction in some examples is shown.
[0041] Fig. 20 Examples of dividing lines of the geometric partitioning pattern in some examples are shown.
[0042] Fig.21 Schematic diagram showing unidirectional prediction motion vector selection for geometric partitioning patterns in some examples.
[0043] Fig. 22 Schematic diagrams showing the generation of blending weights in geometric partitioning mode in some examples.
[0044] Fig.23 A schematic diagram illustrating sub-block level bi-prediction constraints in some examples is shown.
[0045] Figures 24 to 25 Schematic diagram showing padded picture extension areas in some examples.
[0046] Fig.26 Schematic diagram showing motion compensated border filling in some examples.
[0047] Fig. 27 Example diagrams showing deriving motion information for a motion compensated padding block in some examples.
[0048] Fig.28 A schematic diagram showing the encoding order in some examples.
[0049] Fig.29 Schematic diagram showing time co-located blocks in some examples.
[0050] Fig.30 A flow chart of an overview process according to some embodiments of the present disclosure is shown.
[0051] Fig.31 Another flow chart of an overview process according to some embodiments of the present disclosure is shown.
[0052] Fig.32 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION
[0053] In this specification and claims, terms may have nuanced meanings suggested or implied by the context rather than just the explicitly stated meanings. The phrases "in one embodiment / implementation" or "in some embodiments / implementations" as used herein do not necessarily refer to the same embodiment / implementation, and the phrases "in another embodiment / implementation" or "in other embodiments" as used herein do not necessarily refer to different embodiments. For example, the claimed subject matter is intended to include a combination of all or part of the exemplary embodiments / implementations.
[0054] In general, terms can be understood at least in part from their use in context. For example, terms such as "and", "or", or "and / or" as used herein can include a variety of context-dependent meanings. Typically, "or", if used to relate to a list, such as A, B, or C, means A, B, and C in an inclusive sense and A, B, or C in an exclusive sense. In addition, the terms "one or more", "at least one", "one", "each", or "the" used herein depend at least in part on the context and can be in singular or plural form. In addition, the terms "based on" or "determined by..." also depend at least in part on the context and can be understood to not necessarily be intended to convey a set of exclusive factors, but can allow for the presence of other factors that are not necessarily explicitly described.
[0055] Figure 1 A simplified block diagram of a communication system (100) according to an embodiment of the present disclosure is shown. The communication system (100) includes a plurality of terminal devices, such as a first terminal device 110, a second terminal device 120, a third terminal device 130, and a fourth terminal device 140, which can communicate with each other via, for example, a network (150). Figure 1In the example of , the first terminal device (110) and the second terminal device (120) can perform unidirectional data transmission. For example, the first terminal device (110) can encode video data in the form of one or more encoded code streams (for example, video picture streams collected by the first terminal device (110)) for transmission through the network (150). The second terminal device (120) can receive the encoded video data from the network (150), decode the encoded video data to restore the video data, and display the video picture based on the restored video data. Unidirectional data transmission is more common in applications such as media services.
[0056] In another example, the third terminal device (130) and the fourth terminal device (140) can perform bidirectional transmission of encoded video data, for example, during a video conferencing application. In one example, for bidirectional data transmission, each of the third terminal device (130) and the fourth terminal device (140) can encode video data (e.g., a video picture stream collected by the terminal device) for transmission to the other terminal device of the third terminal device (130) and the fourth terminal device (140). Each of the third terminal device (130) and the fourth terminal device (140) can also receive encoded video data from the other terminal device of the third terminal device (130) and the fourth terminal device (140) to restore and display the video picture.
[0057] exist Figure 1 In the example of , multiple terminal devices can be implemented as servers, personal computers and smart phones, but the applicability of the basic principles of the present disclosure may not be limited to this. Embodiments of the present disclosure can be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing devices, etc. Network (150) represents any number or type of network that transmits encoded video data between multiple terminal devices, including, for example, wired (wired) and / or wireless communication networks. Communication network (150) can exchange data in circuit switching, packet switching and / or other types of channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet.
[0058] As an example of an application of the disclosed subject matter, Figure 2 The placement of the video encoder and video decoder in a video streaming environment is shown. The disclosed subject matter can be equally applicable to other video applications, including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., and the like.
[0059] like Figure 2As shown, a video streaming system may include a video acquisition subsystem (213), which may include a video source (201) such as a digital camera, for creating uncompressed video pictures or picture streams (202). In one example, the video picture stream (202) includes samples captured by the digital camera of the video source 201. Compared to the encoded video data (204) (or the encoded video code stream), the video picture stream (202) is depicted as a thick line to emphasize the high data volume. The video picture stream (202) can be processed by an electronic device (220), which includes a video encoder (203) coupled to the video source (201). The video encoder (203) can include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video picture stream (202), the encoded video data (204) (or the encoded video code stream (204)) is depicted as a thin line to emphasize the low data volume. The encoded video data (204) (or the encoded video code stream (204)) can be stored on the streaming server (205) for future use or directly stored to a downstream video device (not shown). One or more streaming client subsystems, such as Figure 2 The client subsystem (206) and the client subsystem (208) in the embodiment of the present invention can access the streaming server (205) to retrieve the copy (207) and the copy (209) of the encoded video data (204). The client subsystem (206) can include a video decoder (210) such as in the electronic device (230). The video decoder (210) decodes the input copy (207) of the encoded video data and creates an output stream (211) of video pictures, which is uncompressed and can be presented on a display (212) (e.g., a display screen) or other presentation device (not shown).
[0060] Figure 3 A block diagram of a video decoder (310) of an electronic device (330) according to any of the following embodiments of the present disclosure is shown. The electronic device (330) may include a receiver (331) (e.g., a receiving circuit). The video decoder (310) may be used to replace Figure 2 The video decoder (210) of the example.
[0061] like Figure 3As shown, a receiver (331) can receive one or more encoded video sequences from a channel (301). To prevent network jitter and / or process playback timing, a buffer memory (315) can be set between the receiver (331) and an entropy decoder / parser (320) (hereinafter referred to as "parser (320)"). The parser (320) can reconstruct symbols (321) based on the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (310) and potential information for controlling a rendering device such as a display device (312) (e.g., a display screen). The parser (320) can parse / entropy decode the encoded video sequence. The parser (320) can extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the encoded video sequence. The subgroup can include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (320) may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. The reconstruction of the symbols (321) may involve a number of different processing or functional units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (320) from the coded video sequence.
[0062] The first unit may include a sealer / inverse transform unit (351). The sealer / inverse transform unit (351) may receive quantized transform coefficients as symbols (321) and control information from the parser (320), including information indicating which type of inverse transform to use, block size, quantization factor / parameter, quantization scaling matrix and position. The sealer / inverse transform unit (351) may output a block including sample values, which may be input into an aggregator (355).
[0063] In some cases, the output samples of the scaler / inverse transform unit (351) may belong to an intra-coded block, i.e., a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) may generate a block of the same size and shape as the block being reconstructed, using surrounding block information that has already been reconstructed and stored in a current picture buffer (358). For example, the current picture buffer (358) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (355) adds the prediction information generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.
[0064] In other cases, the output samples of the scaler / inverse transform unit (351) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (353) may access the reference picture memory (357) based on the motion vector to obtain samples for inter-frame picture prediction. After the extracted reference samples are motion compensated according to the block-related symbols (321), these samples may be added to the output of the scaler / inverse transform unit (351) (the output of the scaler / inverse transform unit (351) may be referred to as residual samples or residual signals) by the aggregator (355) to generate output sample information.
[0065] The output samples of the aggregator (355) can be processed by various loop filtering techniques in a loop filter unit (356) including several types of loop filters. The output of the loop filter unit (356) can be a sample stream that can be output to a rendering device (312) and stored in a reference picture memory (357) for subsequent inter-frame picture prediction.
[0066] Figure 4 A block diagram of a video encoder (403) according to an exemplary embodiment of the present disclosure is shown. The video encoder (403) may be provided in an electronic device (420). The electronic device (420) may further include a transmitter (440) (e.g., a transmission circuit). The video encoder (403) may be used to replace Figure 4 The video encoder (403) in the example.
[0067] The video encoder (403) may receive video samples from a video source (401). According to some exemplary embodiments, the video encoder (403) may encode and compress pictures of a source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (450). In some embodiments, the controller (450) is functionally coupled to and controls other functional units as described below. The parameters set by the controller (450) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc.
[0068] In some exemplary embodiments, the video encoder (403) may be configured to operate in an encoding loop. The encoding loop may include a source encoder (430) and a (local) decoder (433) embedded in the video encoder (403). Even if the embedded decoder (433) processes an encoded video stream that is not entropy encoded by the source encoder (430), the decoder (433) reconstructs symbols to create sample data in a manner similar to that of the (remote) decoder to create sample data (because in the video compression technology considered by the disclosed subject matter, any compression between symbols and the encoded video code stream in entropy encoding can be lossless). At this point, it can be observed that any decoder technology other than parsing / entropy decoding that may only exist in the decoder must also exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may sometimes focus on the decoder operation, which is associated with the decoding portion of the encoder. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the fully described decoder technology. A more detailed description is only required in some places and is provided below.
[0069] In some example implementations, during operation, the source encoder (430) may perform motion compensated predictive coding, which predictively encodes an input picture with reference to one or more previously encoded pictures in a video sequence designated as "reference pictures."
[0070] The local video decoder (433) can decode the encoded video data of a picture that can be designated as a reference picture. The local video decoder (433) replicates the decoding process that can be performed by the video decoder on the reference picture and can cause the reconstructed reference picture to be stored in the reference picture cache (434). In this way, the video encoder (403) can store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the far-end (remote) video decoder.
[0071] The predictor (435) may perform a prediction search for the encoding engine (432). That is, for a new picture to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as a suitable prediction reference for the new picture.
[0072] The controller (450) can manage the encoding operations of the source encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.
[0073] The outputs of all of the above functional units may be entropy encoded in an entropy encoder (445). The transmitter (440) may buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission over a communication channel (460), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (440) may combine the encoded video data from the video encoder (403) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).
[0074] The controller (450) may manage the operation of the video encoder (403). During encoding, the controller (450) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types: intra picture (I picture), predictive picture (P picture), bidirectional predictive picture (B picture), multi-predictive picture. A source picture may generally be spatially subdivided into a plurality of sample coding blocks as described in further detail below.
[0075] Figure 5 A schematic diagram of a video encoder (503) according to another exemplary embodiment of the present disclosure is shown. The video encoder (503) is configured to receive a processed block (e.g., a prediction block) of sample values within a current video picture in a video picture sequence, and encode the processed block into an encoded picture as part of an encoded video sequence. The exemplary video encoder (503) may be used instead of Figure 4 The video encoder (403) in the example.
[0076] For example, the video encoder (503) receives a matrix of sample values for a processing block. The video encoder (503) then uses, for example, rate-distortion optimization (RDO) to determine whether to use intra mode, inter mode, or bi-predictive mode to best encode the processing block.
[0077] exist Figure 5 In the example of FIG. 5 , the video encoder ( 503 ) includes: Figure 5 The example arrangement shows an inter-frame encoder (530), an intra-frame encoder (522), a residual calculator (523), a switch (526), a residual encoder (524), a common controller (521), and an entropy encoder (525) coupled together.
[0078] The inter-frame encoder (530) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a later picture in display order), generate inter-frame prediction information (e.g., redundant information description, motion vectors, merge mode information according to an inter-frame coding technique), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique.
[0079] The intra encoder (522) is configured to receive samples of a current block (e.g., a processing block), compare the block with encoded blocks in the same picture, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques).
[0080] The general controller (521) may be configured to determine general control data and control other components of the video encoder (503) based on the general control data, such as determining a prediction mode for a block and providing a control signal to a switch (526) based on the prediction mode.
[0081] The residual calculator (523) may be configured to calculate the difference (residual data) between the received block and the prediction result of the block selected from the intra encoder (522) or the inter encoder (530). The residual encoder (524) may be configured to encode the residual data to generate a transform coefficient. Then, the transform coefficient is quantized to obtain a quantized transform coefficient. In various exemplary embodiments, the video encoder (503) further includes a residual decoder (528). The residual decoder (528) is configured to perform an inverse transform and generate decoded residual data. The entropy encoder (525) may be configured to format the code stream to produce an encoded block and perform entropy encoding.
[0082] Figure 6 A schematic diagram of an exemplary video decoder (610) according to another embodiment of the present disclosure is shown. The video decoder (610) is configured to receive an encoded picture as part of an encoded video sequence, and decode the encoded picture to generate a reconstructed picture. In one example, the video decoder (610) can be used instead of Figure 4 A video decoder (410) is shown in an example.
[0083] exist Figure 6 In the example of FIG. 6 , the video decoder ( 610 ) includes Figure 6 The example arrangement shows an entropy decoder (671), an inter-frame decoder (680), a residual decoder (673), a reconstruction module (674), and an intra-frame decoder (672) coupled together.
[0084] The entropy decoder (671) may be configured to reconstruct certain symbols from the encoded picture, the symbols representing the syntax elements constituting the encoded picture. The inter-frame decoder (680) may be configured to receive inter-frame prediction information, and to generate an inter-frame prediction result based on the inter-frame prediction information. The intra-frame decoder (672) may be configured to receive intra-frame prediction information, and to generate a prediction result based on the intra-frame prediction information. The residual decoder (673) may be configured to perform inverse quantization to extract dequantized transform coefficients, and to process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The reconstruction module (674) may be configured to combine the residual output by the residual decoder (673) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which forms a part of the reconstructed picture, which in turn is a part of the reconstructed video.
[0085] It should be noted that the video encoder (203), video encoder (403) and video encoder (503) and video decoder (210), video decoder (310) and video decoder (610) may be implemented using any suitable technology. In some exemplary embodiments, the video encoder (203), video encoder (403) and video encoder (503) and video decoder (210), video decoder (310) and video decoder (610) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (203), video encoder (403) and video encoder (503) and video decoder (210), video decoder (310) and video decoder (610) may be implemented using one or more processors that execute software instructions.
[0086] For block partitioning for encoding and decoding, the usual partitioning can start from the basic block and can follow a predetermined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning can be hierarchical and recursive. After the basic block is segmented or divided according to any exemplary partitioning process described below or other processes or combinations thereof, a set of final partitions or coding blocks can be obtained. Each of these partitions can be at one of the various partitioning levels in the partitioning hierarchy and can have various shapes. Each of these partitions can be called a coding block (CB). For various exemplary partitioning implementations further described below, each resulting CB can have any allowed size and partitioning level. These partitions are called coding blocks because these partitions can form units to which some basic encoding / decoding decisions can be made, and encoding / decoding parameters can be optimized, determined, and written into the encoded video code stream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. The coding block can be a luminance coding block or a chrominance coding block. The CB tree structure for each color can be called a coding block tree (CBT). The coding blocks of all color channels can be collectively referred to as coding units (CUs). The hierarchical structures of all color channels can be collectively referred to as coding tree units (CTUs). The division modes or structures of various color channels in a CTU can be the same or different.
[0087] In some implementations, the partitioning tree schemes or structures for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or modes. In addition, whether the luma channel and the chroma channel use the same or different coding partitioning tree structures and the actual coding partitioning tree structures to be used may depend on whether the slice being encoded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partitioning tree structures or coding partitioning tree structure modes, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding partitioning tree scheme. When a separate coding partitioning tree structure or mode is used, the luma channel may be divided into multiple CBs by one coding partitioning tree structure, and the chroma channel may be divided into multiple chroma CBs by another coding partitioning tree structure.
[0088] In inter prediction, for each inter prediction CU, motion parameters consisting of a motion vector, a reference picture index, and a reference picture list usage index, as well as other additional information, can be used for inter prediction sample generation. The motion parameters can be signaled explicitly or implicitly. When a CU is encoded in skip mode, the CU can be associated with one PU, may have no significant residual coefficients and no encoded motion vector changes or reference picture indexes. A merge mode can be specified, whereby the motion parameters of the current CU are obtained from neighboring CUs, including obtaining the motion parameters of the current CU from spatial and / or temporal candidates of neighboring CUs. The merge mode can be applied to any inter prediction CU, not just for skip mode. An alternative example of a merge mode for a CU may be explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other information are explicitly signaled to the CU. Various coding tools for prediction are further described in detail below, including but not limited to extended merge prediction, merge motion vector difference (MMVD), advanced motion vector predictor (AMVP) mode with symmetric MVD signaling, affine motion compensated prediction, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8×8 motion field compression), bi-prediction with CU-level weights (BCW), bi-directional optical flow (BDOF), decoder side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), and geometric partitioning mode (GPM).
[0089] In some examples, extended merge prediction may be implemented. For example, the merge candidate list may be constructed by sequentially including the following five types of candidates:
[0090] The spatial motion vector predictor (MVP) from the spatially neighboring CUs;
[0091] ·Time MVP from the same CU;
[0092] History-based MVP from a first-in-first-out (FIFO) table;
[0093] Average MVP in pairs;
[0094] Zero MV.
[0095] The size of the merge list can be signaled in the slice header, and the maximum allowed size of the merge list can be predetermined (e.g., the maximum allowed size is 6). For each CU encoded in merge mode, the index of the best merge candidate can be encoded using truncated unary (TU) binarization. For example, the first bin of the merge index can be encoded with context, and the other bins can be encoded using bypass.
[0096] The merged candidates of each category described above can be generated by various processes as described below. In some exemplary implementations, for spatial MVP candidate derivation, the candidate can be obtained from the location Figure 7 Up to four merge candidates are selected from the candidates at the positions shown. The derived order can be B1, A1, B0, A0 and B2. For example, position B2 may be considered only when any CU at position A0, B0, B1, A1 is not available (for example, because the CU belongs to another slice or another tile) or is intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subject to a redundancy check, which ensures that candidates with the same motion information are excluded from the list, thereby improving coding efficiency. In some exemplary implementations, in order to reduce computational complexity, not all possible candidate pairs are considered in the redundancy check. Instead, only Figure 8 , and if the corresponding candidate for redundancy check does not have the same motion information, the candidate can only be added to the list.
[0097] In some exemplary implementations, the temporal MVP for MVP candidate derivation may be considered next. For example, a temporal candidate may be added to the list. In particular, in the derivation of the temporal merge candidate, the scaled motion vector may be derived based on the co-located CU belonging to the co-located reference picture. The reference picture list for deriving the co-located CU may be explicitly signaled in the slice header. Fig. 9As shown by the dotted line in , the scaled motion vector of the temporal merge candidate can be obtained, which is scaled according to the motion vector of the co-located CU ("col_CU") using the picture order count (POC) distances tb and td, where tb is defined as the POC difference between the reference picture of the current picture ("curr_ref") and the current picture, and td is defined as the POC difference between the reference picture of the co-located picture ("col_ref") and the co-located picture ("col_pic"). The reference picture index of the temporal merge candidate can be set to zero.
[0098] In some exemplary implementations, Fig.10 As shown, the position of the temporal candidate can be selected between candidates C0 and C1. If the CU at position C0 is not available, is intra-coded, or is outside the current row of the CTU, position C1 can be used. Otherwise, position C0 can be used to derive the temporal merge candidate.
[0099] In some exemplary implementations, only the translation motion model is applied to motion compensation prediction (MCP), and the translation motion model can be called a traditional motion compensation model. However, in real-world images or video frames, in addition to translation in a predetermined direction, there may be other types of motion, such as zooming in / out, rotation, perspective change, distortion, and other irregular motions. In some exemplary implementations, in order to simplify motion compensation prediction, block-based affine transformation prediction is applied. Block-based affine transformation motion compensation prediction can be applied. In such an implementation, the traditional translation-only motion would be a special case of the affine motion model. As Fig.11A and Fig. 11B As shown, the affine motion field of the block “Cur” can be obtained by two control points (4 parameters, such as Fig.11A as shown) or three control point motion vectors (6 parameters, such as Fig. 11B The motion information of the control point is described by the motion information of the control point (as shown in FIG. 1 ). The motion vector of the control point can be called the control point motion vector (CPMV).
[0100] In one example, for a 4-parameter affine motion model, such as Fig.11A As shown, the motion vector at the sample position (x, y) in the block "Cur" can be derived as a function of the sample position:
[0101]
[0102] The function of the sample position can be parameterized as:
[0103]
[0104] In another example, for a 6-parameter affine motion model, such as Fig.11A and Fig. 11B As shown, the motion vector at the sample position (x, y) in the block can be derived as:
[0105]
[0106] The motion vector at this sample position can be parameterized as:
[0107]
[0108] In the above formula, (mv 0x , mv 0y ) represents the motion vector of the upper left control point, (mv 1x , mv 1y ) represents the motion vector of the upper right corner control point, (mv 2x , mv 2y ) represents the motion vector of the lower left control point.
[0109] In some further exemplary implementations of block-based affine transform prediction, sub-block-based affine transform prediction may be applied. A block may be divided into sub-blocks, such as 4×4 sub-blocks. In order to derive, for example, a motion vector for each 4×4 luminance sub-block, such as Fig.12 The motion vector of the center sample of each sub-block shown (where each small square represents a sub-block) can be calculated according to the above formula (depending on whether the 4-parameter model or the 6-parameter model is used, Fig.12 An example of a 4-parameter model is shown) and rounded to a predetermined precision, such as fractional precision of 1 / 16. A motion compensated interpolation filter can then be applied to generate a prediction for each sub-block with a derived motion vector. The sub-block sizes of the chrominance components can also be set to the same (e.g., 4×4). The MV of a 4×4 chrominance sub-block can be calculated as the average of the MVs of the four corresponding 4×4 luminance sub-blocks (assuming that luminance samples are sampled twice as much as chrominance samples). Therefore, this affine model takes into account warping in sub-block units, but for each sub-block, only translational motion compensation is considered in order to reduce the computational complexity involved in full-pixel level warping compensation.
[0110] In some further exemplary implementations, and as implemented for translation-only motion inter prediction, the affine motion inter prediction mode may be implemented in an affine merge mode and / or an affine advanced MVP (AMVP) mode.
[0111] In one example, an affine merge mode called AF_MERGE mode can be applied to CUs whose width and height are both greater than or equal to a predetermined size (e.g., 8). In this mode, the CPMV of the current CU can be generated based on the motion information of the spatially adjacent CUs. The CPMV predictor (CPMVP) candidates can be up to a predetermined number (e.g., five), and an index can be signaled to indicate the CPMVP candidate to be used for the current CU. The following three types of CPVM candidates can be used to form an affine merge candidate list:
[0112] Inherited affine merge candidates inferred from the CPMV of neighboring CUs;
[0113] Affine merged candidate CPMVP constructed using the translation MV of the adjacent CU;
[0114] Zero MV.
[0115] In some exemplary implementations, there may be a predetermined maximum number (e.g., 2) of inherited affine candidates, which may be derived from the affine motion models of neighboring blocks, e.g., one from the left neighboring CU and one from the upper neighboring CU. Figure 7 As shown in . For the prediction value on the left, the candidate scanning order can be A0→A1, and for the prediction value on the top, the scanning order is B0→B1→B2. In some implementations, only the first inherited candidate from each side is selected. No pruning check is performed between two inherited candidates. When an adjacent affine CU is identified, the control point motion vectors of the adjacent affine CU can be used to derive the CPMVP candidate in the affine merge list of the current CU. Fig.13 As shown in the figure, if the adjacent lower left block A is encoded in affine mode, the motion vectors V2, V3 and V4 of the upper left corner, upper right corner and lower left corner of the CU containing block A can be obtained. When block A is encoded in a 4-parameter affine model, two CPMVs of the current CU can be calculated based on V2 and V3. In the case where block A is encoded in a 6-parameter affine model, three CPMVs of the current CU are calculated based on V2, V3 and V4.
[0116] The above-mentioned affine candidate constructed may refer to a candidate constructed by combining adjacent translation motion information of each control point. The motion information of the control point can be obtained from Fig.14 The specified spatial neighbors and temporal neighbors are exported as shown. CPMV k(k=1,2,3,4) represents the kth control point. For CPMV1, check the B2→B3→A2 block, and the MV of the first available block can be used. For CPMV2, check the B1→B0 block, and for CPMV3, check the A1→A0 block. If temporal MVP (TMVP) is available, TMVP is used as CPMV4.
[0117] After obtaining the MVs of the four control points, an affine merge candidate can be constructed based on the motion information. The following example combinations of control point MVs can be used to construct affine merge candidates in the order indicated: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.
[0118] Specifically, a combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and a combination of 2 CPMVs constructs a 4-parameter affine merge candidate. In some exemplary implementations, to avoid the motion scaling process, if the reference indexes of the control points are different, the relevant combination of control point MVs may be discarded.
[0119] In some exemplary implementations, as described above, after checking the inherited affine merge candidates and the constructed affine merge candidates, if the list is still not full (according to a predetermined maximum number of candidates in the list), a zero MV may be inserted at the end of the list.
[0120] In another example, the affine AMVP mode can be applied to CUs whose width and height are both greater than or equal to a predetermined size (e.g., 16). The affine flag at the CU level can be signaled in the bitstream to indicate whether the affine AMVP mode is used. If the affine AMVP mode is used, another flag is signaled to indicate whether a 4-parameter affine model or a 6-parameter affine model is used. In this mode, the difference between the CPMV of the current CU and the predicted value CPMVP of these CPMVs can be signaled in the bitstream. The affine AVMP candidate list size can be predetermined, such as 2, and the affine AVMP candidate list size can be generated by using four types of CPVM candidates in the order shown in the following example:
[0121] Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs;
[0122] Affine AMVP candidate CPMVP constructed using the translation MV of the neighboring CU;
[0123] · Translational MV from neighboring CU;
[0124] Zero MV.
[0125] The order of checking inherited affine AMVP candidates can be the same as the order of checking inherited affine merge candidates. The difference is that, for example, for AMVP candidates, only affine CUs with the same reference picture as in the current block are considered. When the inherited affine motion prediction value is inserted into the candidate list, the pruning process cannot be applied.
[0126] The AMVP candidates can be constructed from Fig.14 The specified spatial neighbors shown are exported. The same check order as in the above-mentioned affine merge mode can be used for candidate construction. In addition, the reference picture index of the adjacent block can also be checked. The first block in the check order can be used, which is inter-coded and has the same reference picture as in the current CU. When the current CU is encoded using a 4-parameter affine mode and both mv0 and mv1 are available, there may be only one candidate, and mv0 and mv1 are added to the affine AMVP list as a candidate. When the current CU is encoded using a 6-parameter affine mode and all three CPMVs are available, the three CPMVs can be added to the affine AMVP list as a candidate. Otherwise, the constructed AMVP candidate may be set to unavailable.
[0127] If after checking the inherited affine AMVP candidates and the constructed AMVP candidates, the affine AMVP list candidates are still less than a predetermined number, such as 2, then when mv0, mv1, and mv2 are available, mv0, mv1, and mv2 are added in order as translation MVs to predict all control point MVs of the current CU. Finally, if the affine AMVP list is still not full, the affine AMVP list is filled with, for example, zero MVs.
[0128] In some exemplary implementations, the above-described sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation, at the expense of prediction accuracy loss (as described above, warping is considered only at the sub-block level, not within each sub-block). In some other exemplary implementations, in order to achieve finer granularity of warped motion compensation, prediction refinement with optical flow (PROF) can be used to refine sub-block-based affine motion compensation predictions without significantly increasing the memory access bandwidth for motion compensation. For example, the luma prediction samples can be modified by adding differences derived from one or more optical flow equations. An example PROF may include the following four steps:
[0129] Step 1): First, sub-block based affine motion compensation may be performed to generate a sub-block prediction I(i, j).
[0130] Step 2): The spatial gradient g of the sub-block prediction can then be calculated at each sample position using, for example, a 3-tap filter [1,0,1]. x (i,j) and g y (i, j). The gradient calculation may be exactly the same as the gradient calculation in bidirectional optical flow (BDOF). For example, the spatial gradient g x (i,j) and g y (i,j) can be calculated as:
[0131] g x (i,j)=(I(i+1,j)>>shift1)-(I(i-1,j)>>shift1) (5)
[0132] g y (i,j)=(I(i,j+1)>>shift1)-(I(i,j-1)>>shift1) (6)
[0133] Among them, shift1 is used to control the accuracy of the gradient. Sub-block (i.e., size 4×4) prediction can be extended by one sample on each side of the gradient calculation. In order to avoid additional memory bandwidth and additional interpolation calculations, those extended samples on the extended boundary can be copied from the nearest integer pixel position in the reference picture.
[0134] Step 3): The brightness prediction correction can be calculated by the following exemplary optical flow equation:
[0135] ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) (7)
[0136] Among them, Fig.15 As shown, Δv(i,j) represents the difference between the sample MV (denoted as v(i,j)) calculated for the sample position (i,j) and the sub-block MV of the sub-block to which the sample (i,j) belongs. Δv(i,j) is quantized in units of a predetermined or signaled luma sample precision (e.g., 1 / 32 luma sample precision).
[0137] Since the affine model parameters and the sample positions relative to the sub-block center do not change across sub-blocks, we can compute Δv(i,j) for the first sub-block and reuse it for other sub-blocks in the same CU. Let dx(i,j) and dy(i,j) be the distance from the sample position (i,j) to the sub-block center (x SB ,y SB), Δv(i,j) can be derived by the following example equations:
[0138]
[0139]
[0140] To maintain accuracy, the sub-block center (x SB ,y SB ) can be calculated as ((W SB -1) / 2, (H SB -1) / 2), where W SB and H SB are the width and height of the sub-block respectively.
[0141] For the 4-parameter affine model,
[0142]
[0143] For the 6-parameter affine model,
[0144]
[0145] Among them, (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the upper left, upper right and lower left control point motion vectors, w and h are the width and height of the CU.
[0146] Step 4): Finally, the brightness prediction correction ΔI(i,j) can be added to the sub-block prediction I(i,j). Then the final prediction I' can be generated. For example, the final prediction I' is:
[0147] I'(i,j)=I(i,j)+ΔI(i,j) (12)
[0148] In some exemplary implementations, for an affine-coded CU, PROF may not be applied in two cases: 1) when all control point MVs are the same, indicating that the CU has only translational motion; or 2) the affine motion parameters are greater than a specified limit, because sub-block based affine motion compensation is degraded to CU based motion compensation to avoid large memory access bandwidth requirements.
[0149] In some examples, a sub-block based temporal motion vector prediction (SbTMVP) method may be implemented. For example, the example SbTMVP may use the motion field in the co-located picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same co-located picture used by the temporal motion vector prediction (TMVP) may be used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects:
[0150] TMVP predicts motion at CU level, while SbTMVP predicts motion at sub-CU level;
[0151] TMVP obtains the temporal motion vector from the co-located block in the co-located picture (the co-located block is the block relative to the lower right corner or the center block of the current CU), while SbTMVP applies a motion offset before obtaining the temporal motion information from the co-located picture, where the motion offset can be obtained from the motion vector of one of the spatial neighboring blocks from the current CU.
[0152] Fig.16 and Fig.17 An exemplary SbTMVP process is shown in FIG. SbTMVP predicts the motion vector of a sub-CU within the current CU in two exemplary steps. In the first step, it is possible to check Fig.16 If A1 has a motion vector that uses the co-located picture as its reference picture, then that motion vector can be selected as the motion offset to be applied. If no such motion vector is identified, then the motion offset can be set to (0, 0).
[0153] In the second step, if Fig.17 As shown, the motion offset identified in the first step can be applied (ie, added to the coordinates of the current block) to obtain sub-CU level motion information (motion vector and reference index) from the co-located picture. Fig.17 The example in assumes that the motion offset is set to the motion of block A1. Then, for each sub-CU, the motion information of the sub-CU can be derived using the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample). After identifying the motion information of the co-located sub-CU, the motion information of the co-located sub-CU can then be converted to the motion vector and reference index of the current sub-CU in a manner similar to the conventional TMVP process, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.
[0154] In some exemplary implementations, bidirectional prediction with CU-level weights (BCW) may be used, where a bidirectional prediction signal may be generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In addition, the bidirectional prediction mode may be extended beyond simple averaging to allow weighted averaging of two prediction signals.
[0155] P bi-pred =((8-w)*P0+w*P1+4)>>3 (13)
[0156] Among them, multiple (e.g., five) weights may be allowed in weighted average bidirectional prediction, such as w∈{-2,3,4,5,10}. For each bidirectionally predicted CU, the weight w may be determined in one of two ways: 1) For non-merged CUs, the weight index may be signaled after the motion vector difference; 2) For merged CUs, the weight index may be inferred from neighboring blocks based on the merge candidate index. BCW may be applied only to CUs with a minimum number (e.g., 256) or more luma samples (i.e., the CU width multiplied by the CU height is greater than or equal to the minimum number, e.g., 256). For low-latency pictures, all 5 weights may be used. For non-low-latency pictures, only a subset thereof may be used, such as 3 weights (w∈{3,4,5}).
[0157] At the encoder, a fast search algorithm can be applied to find the weight index without significantly increasing the complexity of the encoder. In some exemplary implementations, when combined with AMVR, unequal weights may only be conditionally checked for 1-pixel and 4-pixel motion vector precision if the current picture is a low-latency picture.
[0158] In some example implementations, when combined with affine, affine ME may be performed on unequal weights if and only if the affine mode is selected as the current best mode.
[0159] In some example implementations, unequal weights are only checked conditionally when the two reference pictures in the bi-directional prediction are the same.
[0160] In some exemplary implementations, when certain conditions are met, depending on the POC distance between the current picture and the reference picture of the current picture, the encoding quantization parameter (QP), and the temporal level, unequal weights may not be searched.
[0161] In some exemplary implementations, the BCW weight index may be encoded using one context-encoded binary followed by a bypass-encoded binary. The first context-encoded binary indicates whether equal weights are used; and if unequal weights are used, an additional binary may be signaled using bypass encoding to indicate which unequal weights are used.
[0162] In some exemplary implementations, weighted prediction (WP) may be supported to efficiently encode fading video content. WP may allow weighting parameters (weights and offsets) to be signaled for each reference picture in each reference picture list in reference picture list L0 and reference picture list L1. Then, during motion compensation, one or more weights and one or more offsets of the corresponding reference picture may be applied. WP and BCW may be designed for different types of video content. To avoid interaction between WP and BCW, if a CU uses WP, the BCW weight index may not be signaled, and w may be inferred to be 4 (e.g., equal weights are applied). For a merged CU, a weight index may be inferred from neighboring blocks based on a merge candidate index. This may be applied to both conventional merge mode and inherited affine merge mode. For a constructed affine merge mode, affine motion information may be constructed based on, for example, motion information of up to 3 blocks. The BCW index of a CU using a constructed affine merge mode may be simply set equal to the BCW index of the first control point MV.
[0163] In some implementations, combined intra and inter prediction (CIIP) and BCW may not be jointly applied to a CU. When a CU is encoded in CIIP mode, the BCW index of the current CU may be set to, for example, 2, such as equal weight.
[0164] In some further examples, a modified bi-directional optical flow (BDOF) may be implemented. Compared to the traditional bi-directional optical flow (BIO), the example BDOF may be a simpler version that requires less computation, especially in terms of the number of multiplications and the size of the multipliers.
[0165] For example, BDOF can be used to modify the bidirectional prediction signal of a CU at a 4×4 sub-block level. In some exemplary implementations, BDOF can be applied to a CU if all of the following conditions are met:
[0166] The CU is encoded using the "true" bidirectional prediction mode, that is, one of the two reference pictures precedes the current picture in display order, and the other reference picture follows the current picture in display order;
[0167] The distances from the two reference pictures to the current picture (i.e., POC differences) are the same;
[0168] Both reference images are short-term reference images;
[0169] The CU is not encoded using affine mode or SbTMVP merge mode;
[0170] The CU has more than a certain number (e.g., 64) of luma samples;
[0171] CU height and CU width are both greater than or equal to a certain number (e.g., 8) of luma samples;
[0172] BCW weight index indicates equal weight;
[0173] The WP in the current CU is unavailable;
[0174] CIIP mode is not used for the current CU.
[0175] In some exemplary implementations, BDOF may be applied only to the luma component. As the name implies, the BDOF mode may be based on the concept of optical flow, which assumes that the motion of objects is smooth. For each, for example, 4×4 sub-block, the motion correction (v x ,v y ). The motion correction can then be used to adjust the bi-directional prediction sample values in the 4×4 sub-block. The following example steps can be applied to the BDOF process.
[0176] First, the horizontal and vertical gradients of the two prediction signals can be calculated by directly calculating the difference between two adjacent samples, and k=0,1, that is,
[0177]
[0178]
[0179] Among them, I (k) (i, j) is the sample value of the prediction signal in list k at coordinate (i, j), k = 0, 1, and shift1 can be calculated based on the brightness bit depth (bitDepth) as shift1 = max(6, bitDepth-6).
[0180] Then, the autocorrelation and cross-correlation of gradients S1, S2, S3, S5 and S6 can be calculated as:
[0181] S1=∑ (i,j)∈Ω Abs(ψ x (i,j)),S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j))
[0182] S2=∑ (i,j)∈Ω ψ x (i,j)·Sign(ψ y (i,j)) (15)
[0183] S5=∑ (i,j)∈Ω Abs(ψ y (i,j)),S6=∑ (i,j)∈Ω θ(i,j)·Sign(ψ y (i,j))
[0184] in,
[0185]
[0186]
[0187] θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b )
[0188] where Ω is a 6×6 window around the 4×4 sub-block, n a and n b The values are set to min(1, bitDepth-11) and min(4, bitDepth-8) respectively.
[0189] The motion correction (v) can then be derived using the cross-correlation and autocorrelation terms using the following formula: x ,v y ):
[0190]
[0191]
[0192] in, th′ BIO =2 max(5,BD-7) , is the floor function, and
[0193] Based on the motion correction and the gradient, the following adjustment can be calculated for each sample in the 4×4 sub-block:
[0194]
[0195] Finally, the BDOF samples of the CU can be calculated by adjusting the bidirectional prediction samples as follows:
[0196] pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift (20)
[0197] These values may be chosen such that the multiplier in the BDOF process does not exceed, for example, 15 bits, and the maximum bit width of the intermediate parameters in the BDOF process is kept within, for example, 32 bits.
[0198] In some exemplary implementations, in order to derive the gradient value, it may be necessary to generate some prediction samples I in the list k (k=0, 1) outside the current CU boundary. (k) (i,j). Fig.18 As shown, the example BDOF may use an extended row / column around the CU boundary 1802. In order to control the computational complexity of generating prediction samples outside the boundary, the prediction samples in the extended area (outermost rows and columns) may be generated by directly obtaining reference samples at nearby integer positions (using floor() operations on the coordinates) without interpolation, and the usual example 8-tap motion compensated interpolation filter may be used to generate prediction samples within the CU (within the boundary 1802). These extended sample values may only be used for gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are needed, the samples and gradient values outside the CU boundary may be filled (i.e., repeated) from the samples and gradient values of its nearest neighbors.
[0199] When the width and / or height of the CU is greater than a predetermined number (e.g., 16 luma samples), the CU may be partitioned into sub-blocks having a width and / or height equal to a predetermined number (e.g., 16 luma samples), and the sub-block boundaries may be considered as CU boundaries in the BDOF process. The maximum unit size of the BDOF process may be limited to, for example, 16×16. The BDOF process may be skipped for each sub-block. The BDOF process may not be applied to the sub-block when the sum of absolute difference (SAD) between the initial L0 and L1 prediction samples is less than a threshold. The threshold may be set equal to (8*W*(H>>1), where W represents the sub-block width and H represents the sub-block height. In order to avoid the additional complexity of the SAD calculation, the SAD between the initial L0 and L1 prediction samples calculated in the DVMR process may be reused here.
[0200] In some exemplary implementations, if BCW is available in the current block, i.e., the BCW weight index indicates unequal weights, then bidirectional optical flow is not available. Similarly, if WP is available in the current block, i.e., luma_weight_lx_flag is 1 for either of the two reference pictures, then BDOF is also not available. In some exemplary implementations, BDOF is also not available when the CU is encoded in symmetric MVD mode or CIIP mode.
[0201] In some examples, decoder-side motion vector correction (DMVR) can be implemented. For example, to increase the accuracy of the MV in merge mode, decoder-side motion vector correction based on bilateral-matching (BM) can be applied. In a bidirectional prediction operation, a corrected MV can be searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method can calculate the distortion between two candidate blocks in reference picture list L0 and list L1. Fig.19 As shown, the SAD between the reference blocks of each MV candidate based on the surrounding of the initial MV can be calculated. The MV candidate with the lowest SAD can become the modified MV and be used to generate the bidirectional prediction signal.
[0202] In some example implementations, the application of DMVR may be limited and may be applied only to CUs encoded using the following modes and features:
[0203] CU-level merge mode with bi-directional prediction MV;
[0204] Relative to the current picture, one reference picture is before and the other reference picture is after;
[0205] The distances from the two reference pictures to the current picture (i.e., POC differences) are the same;
[0206] Both reference images are short-term reference images;
[0207] The CU has more than a predetermined number (e.g., 64) of luma samples;
[0208] CU height and CU width are both greater than or equal to a predetermined number (e.g., 8) of luma samples;
[0209] BCW weight index indicates equal weight;
[0210] WP in the current block is unavailable;
[0211] CIIP mode is not used for the current block.
[0212] In some exemplary implementations, the modified MV derived from the DMVR process can be used to generate inter-frame prediction samples and can also be used for temporal motion vector prediction in subsequent picture encoding, while the original MV is used in the deblocking process and also for spatial motion vector prediction in subsequent CU encoding.
[0213] In some exemplary implementations of DVMR, the search points around the initial MV and MV offset can follow the MV difference mirror rule. In other words, any point examined by DMVR represented by a candidate MV pair (MV0, MV1) can follow the following two equations:
[0214] MV0′=MV0+MV_offset (21)
[0215] MV1′=MV1-MV_offset (22)
[0216] Wherein, MV_offset represents the correction offset between the initial MV and the modified MV in one of the reference pictures. The correction search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search phase and a fractional sample correction phase.
[0217] In some exemplary implementations, a full search of, for example, 25 points may be applied to perform an integer sample offset search. First, the SAD of the initial MV pair may be calculated. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of DMVR may be terminated. Otherwise, the SAD of the remaining 24 points may be calculated and checked in raster scan order. The point with the smallest SAD may be selected as the output of the integer sample offset search stage. In order to reduce the penalty for uncertainty in DMVR correction, the original MV may be biased in the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates may be reduced by 1 / 4 of the SAD value.
[0218] In some exemplary implementations, the integer sample search may be followed by a fractional sample correction. To save computational complexity, the fractional sample correction may be derived using a parametric error surface equation rather than an additional search through SAD comparison. The fractional sample correction may be conditionally called based on the output of the integer sample search phase. The fractional sample correction may be further applied when the integer sample search phase ends with a center having a minimum SAD in either the first iteration or the second iteration search.
[0219] In some exemplary implementations, in the sub-pixel offset estimation based on the parameter error surface, the center position cost and the costs at four neighboring positions from the center may be used to fit a 2D parabolic error surface equation of the following form:
[0220] E(x,y)=A(x min ) 2 +B(yy min ) 2 +C (23)
[0221] Among them, (x min ,y min ) corresponds to the fractional position with the minimum cost, and C corresponds to the minimum cost value. By solving the above equation using the cost values of the five search points, (x min ,y min ) can be calculated as:
[0222] x min =(E(-1,0)-E(1,0)) / (2(E(-1,0)+E(1,0)-2E(0,0))) (24)
[0223] y min =(E(0,-1)-E(0,1)) / (2((E(0,-1)+E(0,1)-2E(0,0))) (25)
[0224] Since all cost values are positive and the minimum is E(0,0), the above x min and min The value of can be automatically constrained to be between, for example, -8 and 8. This corresponds, for example, to a half-pixel offset with 1 / 16 pixel MV precision. The calculated fraction (x min ,y min ) can be added to the integer distance correction variable MV to obtain a sub-pixel accurate correction variable MV.
[0225] In some further examples, bilinear interpolation and sample filling may be implemented. The resolution of the MV may be, for example, 1 / 16 brightness samples. Samples at fractional positions may be interpolated using, for example, an 8-tap interpolation filter. In DMVR, the search point may be around the initial fractional pixel MV with an integer sample offset. Therefore, for the DMVR search process, it may be necessary to interpolate the samples at these fractional positions. In order to reduce computational complexity, a bilinear interpolation filter may be used to generate fractional samples for use in the search process in DMVR. Another important effect of using a bilinear filter may be that DVMR does not access more reference samples within a two-sample search range than a conventional motion compensation process. After obtaining the modified MV through the DMVR search process, a conventional 8-tap interpolation filter may be applied to generate the final prediction. In order not to access more reference samples than the conventional MC process, samples that are not required by the interpolation process based on the original MV but may be required by the interpolation process based on the modified MV will be filled from those available samples.
[0226] In some exemplary implementations, when the width and / or height of a CU is greater than a predetermined number (e.g., 16) of luma samples, the CU may be further partitioned into sub-blocks having a width and / or height equal to 16 luma samples. The maximum unit size of the DMVR search process may be limited to, for example, 16×16.
[0227] In some further examples, a geometric partitioning mode may be implemented for inter prediction. The geometric partitioning mode may be signaled using a CU level flag as a merge mode, other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. For each possible CU size w×h=2 m ×2 n , where m,n∈{3…6} (excluding sizes such as 8×64 and 64×8), the geometric partitioning mode can support a total of, for example, 64 partitions.
[0228] In some exemplary implementations, when this mode is used, the CU may be split into two parts by a geometrically positioned straight line, such as Fig. 20 As shown. The position of the partition line can be mathematically derived from the angle and offset parameters of the specific partition. Each part of the geometric partition in the CU can use its own motion for inter-frame prediction. For example, each partition may only allow unidirectional prediction. In other words, each part can have one motion vector and one reference index. Unidirectional prediction motion constraints can be applied to ensure that only two motion compensated predictions are required for each CU (similar to traditional bidirectional prediction). The unidirectional prediction motion of each partition can be derived using the process described further below.
[0229] In some exemplary implementations, if a geometry partitioning mode (GPM) is used for the current CU, a geometry partition index indicating the partitioning mode (angle and offset) of the geometry partition and two merge indices (one for each partition) may be further signaled. The number of maximum GPM candidate sizes may be explicitly signaled in the SPS and the syntax binarization may be specified for the GPM merge indices. After predicting each part of the geometry partition, a hybrid process with adaptive weights may be used to adjust the sample values along the edges of the geometry partitions, as further described below. This may be a prediction signal for the entire CU, and the transform and quantization processes may be applied to the entire CU as in other prediction modes. Finally, as described in further detail below, the motion field of the CU predicted using the geometry partitioning mode may be stored.
[0230] Returning to the exemplary implementation of the unidirectional prediction candidate list construction, such a unidirectional prediction candidate list can be directly derived from the merge candidate list constructed according to the extended merge prediction process. The index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list can be represented by n. The LX motion vector of the nth extended merge candidate (where X is equal to the parity check of n) can be used as the nth unidirectional prediction motion vector of the geometric partitioning mode. These motion vectors are in Fig.21 It can be marked with "x". If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate can be used as the unidirectional prediction motion vector of the geometric partition mode.
[0231] Returning to the example blending along the geometric partition edge above, in some example implementations, blending can be applied to the two prediction signals to derive samples around the geometric partition edge after each portion of the geometric partition is predicted using the motion of the geometric partition itself. The blending weight for each position of the CU can be derived based on the distance between the single position and the partition edge.
[0232] For example, the distance from a position (x,y) to the edge of a partition can be derived as:
[0233]
[0234]
[0235]
[0236]
[0237] where i,j are the indices of the angle and offset of the geometry partition, which may depend on the signaled geometry partition index. x,j and ρ y,j The sign of may depend on the angle index i.
[0238] The weights of each part of the above geometric partition can be derived as follows:
[0239] wIdxL(x,y)=partIdx? 32+d(x,y):32-d(x,y) (30)
[0240]
[0241] w1(x,y)=1-w0(x,y) (32)
[0242] In some exemplary implementations, the above partIdx may depend on the angle index i. An example of weight w0 is Fig. 22 Shown in.
[0243] Returning to the motion field storage of the geometry partitioning mode, in some exemplary implementations, Mv1 from the first part of the geometry partition, Mv2 from the second part of the geometry partition, and a combination Mv of Mv1 and Mv2 may be stored in the motion field of the CU encoded in the geometry partitioning mode.
[0244] The type of motion vector stored for each individual position in the motion field can be determined as:
[0245] sType=abs(motionIdx)<32?2:(motionIdx≤0?(1-partIdx):partIdx) (33)
[0246] Where motionId is equal to d(4x+2,4y+2), motionId can be recalculated. partIdx may depend on the angle index i.
[0247] In some exemplary implementations, if sType is equal to 0 or 1, Mv0 or Mv1 may be stored in the corresponding motion field, otherwise, if sType is equal to 2, a combined Mv from Mv0 and Mv2 may be stored. The combined Mv may be generated using the following process:
[0248] • If Mv1 and Mv2 are from different reference picture lists (one from L0 and the other from L1), then Mv1 and Mv2 are simply combined to form a bi-directional prediction motion vector.
[0249] Otherwise, if Mv1 and Mv2 are from the same list, only the unidirectional predicted motion Mv2 is stored.
[0250] In some further examples, adaptive motion vector resolution (AMVR) may be implemented. For example, a fixed motion vector resolution of one quarter luma sample may be used. However, because it may be necessary to select the best balance between the displacement vector rate and the prediction error rate to achieve overall rate-distortion optimality, in some exemplary implementations, it may be possible to allow the motion vector resolution to be selected at the coding block level to balance bit rate and fidelity when signaling motion parameters. This can be achieved through the AMVR mode. If at least one component of the MVD is not equal to zero, the AMVR mode may be signaled at the coding block level. The motion vector prediction value may be rounded to a given resolution so that the resulting motion vector is guaranteed to fall on a grid of a given resolution. For each given resolution, the corresponding AmvrShift value in Table 1 below may also be defined to specify the resolution of the motion vector difference with the left shift operation of the AmvrShift-bit. When the AMVR mode is available, the given motion vector difference (denoted as MvdL0 and MvdL1 in the AMVP mode and as MvdCpL0 and MvdCpL1 in the affine AMVP mode) may be modified as follows.
[0251] Table 1-Description of AmvrShift in VVC
[0252]
[0253]
[0254] In some inter-prediction cases, reference samples used in motion compensation may be outside a picture boundary for which there is no corresponding reference sample. Therefore, in some exemplary implementations, some constraints may be applied when some reference samples used in motion compensation are outside a picture boundary, especially in bidirectional prediction (where reference samples in either of the reference pictures may be outside a picture boundary).
[0255] In some exemplary implementations, when a current pixel with a bidirectionally predicted motion vector has a motion vector pointing to a position beyond a distance threshold outside the picture boundary in one of the two reference lists, the motion vector of the reference list can be considered to be outside the boundary, and the inter-frame prediction can be changed to unidirectional prediction. Only the motion vector of the other reference list that is not outside the boundary can be used for unidirectional prediction. In some exemplary implementations, when the MVs of both reference lists are outside the boundary, bidirectional prediction may not be constrained.
[0256] In some other exemplary implementations, the constraints of bidirectional prediction may be applied at the sub-block level. Fig.23As shown, for each of the N×N sub-blocks in a coding block with an inter-frame bidirectional prediction MV, if a motion vector on one of the reference lists points to a threshold of M pixels outside the boundary of the reference picture, the sub-block can be changed to a unidirectional prediction mode, where only the MV on the reference list does not point outside the boundary threshold of the corresponding reference picture.
[0257] In some exemplary implementations, when bidirectional prediction is changed to unidirectional prediction due to out-of-boundary conditions, bidirectional prediction related methods may be unavailable or modified. For example, when bidirectional prediction restrictions are applied and unidirectional prediction is used, BDOF may not be available.
[0258] In some example implementations of border filling for motion compensation purposes, the extended picture area may be defined as an area around the picture having a predetermined or signaled size (e.g., (maximum CU width + 16)) in each direction of the picture boundary, such as Fig.24 For example, the pixel values in the extended area can be derived by repeated border filling. When the reference block is partially or completely outside the picture boundary (OOB), the repeated filling pixels in the extended area can be used for motion compensation (MC), such as Fig.24 shown.
[0259] In some further exemplary implementations, motion compensation may be used to derive samples outside of picture boundaries instead of just using repetitive padding. Fig.24 In contrast, the total padding area size can be further increased by a specific predetermined or configurable amount, such as 64 pixels, to maintain MV cropping and achieve non-standard repeated padding. Fig.25 As shown, the 64 pixels added to the boundary area are shown as the "MC filling area" (or motion compensation filling area) around the "reference picture". Outside the increased boundary area, an extended area of a predetermined or signaled size (e.g., (maximum CU width+16)) can be further configured.
[0260] In some exemplary implementations, for motion compensated padding, the MV of, for example, a 4×4 boundary block may be used to derive an M×4 or 4×M padding block. The value of M may be derived as the distance from the reference block to the picture boundary, such as Fig.26 If the boundary block is intra-coded, then MV may be considered unavailable and M may be correspondingly set equal to 0. If M is smaller than the example size 64 described above, the rest of the padding region is filled with repeated padding samples.
[0261] In some exemplary implementations, in the case of bidirectional inter prediction, only one prediction direction may be used in MC boundary filling, and the motion vector of this prediction direction points to the pixel position in the reference picture that is farther from the picture boundary relative to the filling direction.
[0262] In some exemplary implementations, pixels in the MC filling block may be corrected with an offset, which may be equal to the difference between the DC values of the reconstructed boundary block and the reference block to which the reconstructed boundary block corresponds.
[0263] In the exemplary boundary filling implementation described above, only the MV of the corresponding boundary block is used for filling. In some cases, repeated filling may be used unnecessarily instead of motion compensation filling. The following additional disclosure describes various example methods for enhancing the use of boundary block MVs for boundary filling to improve inter-frame prediction coding efficiency. The following various exemplary implementations can be used alone or in combination in any order. In addition, these implementations can be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.
[0264] According to some aspects of the present disclosure, a simplified skip mode may be used for motion compensation padding (MCP) blocks. In the simplified skip mode, blocks in the MC padding region are predicted based on motion vectors pointing to reference blocks in reference pictures (as simplified references for the MC padding blocks).
[0265] According to one aspect of the present disclosure, the motion information of the MC filling block can be derived from one of a plurality of neighboring candidates (e.g., N1 plurality of neighboring candidates) at the picture boundary. Since a plurality of neighboring candidates are used, the chance of having no motion information for motion compensated boundary filling (and thus relying on repeated filling) is reduced.
[0266] Fig. 27 An example schematic diagram of deriving motion information of a motion compensated padding (MCP) block in some exemplary implementations is shown. Fig. 27 In the example of , N1=3 neighboring candidate positions A, B, and C may be used to derive motion information of an MCP block (eg, MCP block 2710, MCP block 2720, MCP block 2730, or MCP block 2740).
[0267] For example, in Fig. 27In the example, when the MCP block (2710) is located on the left side of the vertical picture boundary, position A may be the right adjacent position at the picture boundary, position B may be the upper right adjacent position at the picture boundary, and position C may be the lower right adjacent position at the picture boundary. When the MCP block (2720) is located on the right side of the vertical picture boundary, position A may be the left adjacent position at the picture boundary, position B may be the upper left adjacent position at the picture boundary, and position C may be the lower left adjacent position at the picture boundary.
[0268] Similarly, in Fig. 27 In the example, when the MCP block (2730) is located at the top of a horizontal picture boundary, position A may be a bottom adjacent position at the picture boundary, position B may be a bottom left adjacent position at the picture boundary, and position C may be a bottom right adjacent position at the picture boundary. When the MCP block (2740) is located at the bottom side of a horizontal picture boundary, position A may be an upper adjacent position at the picture boundary, position B may be an upper left adjacent position at the picture boundary, and position C may be an upper right adjacent position at the picture boundary.
[0269] In some exemplary implementations, each of the above candidate positions at the picture boundary corresponds to a block size of N×N. In some examples, N may be equal to 4. In another example, N may be equal to 8.
[0270] In some exemplary implementations, Fig. 27 The MCP region of the current picture in can be defined by four rectangles extending from the picture boundaries. Fig. 27 In the example of , the MCP area may include four rectangles (2701) to (2704). The range parameter R of rectangle (2701) to rectangle (2704) defines the luma samples extending outward from the picture boundary. In one example, R may be equal to 64 luma samples. In another example, R may be equal to include Fig.25 The filling range of the MC filling area and repeated filling area.
[0271] In some exemplary implementations, the MCP block size of the top / bottom boundary may be P×L, and the MCP block size of the left / right boundary may be L×P, such as Fig. 27 In one example, L is equal to the MCP range R or the distance from the reference block corresponding to the boundary block in the current picture to the reference picture boundary (e.g., similar to Fig.26 In some examples, P can be a predetermined size of luma samples. In one example, P is equal to 4. In another example, P is equal to 8.
[0272] In some exemplary implementations, three adjacent positions (eg, Fig. 27 A, B, and C in the MCP block). The first available candidate in the checking order can be used as the basis for the motion information prediction value of the MCP block. The MCP block can then be predicted using the motion information to generate the padding samples in the MCP block.
[0273] In some exemplary implementations, the neighboring positions selected for predicting the MCP block may be signaled. In some examples, a binary file may be used as a candidate index to signal one of the first 2 available neighboring positions (also referred to as neighboring candidates or neighboring candidate positions). In some exemplary implementations, variable length coding may be used to signal one of the 3 available neighboring candidates. In some examples, when there may be only one candidate block with available motion information, signaling may not be necessary and the available candidates may be used implicitly. Thus, if the decoder determines that motion information is only available for one of the candidates, the candidate is explicitly used without signaling. In some examples, the signaling of all MCP blocks may be completed for the entire frame at once in a predetermined order, as further described below.
[0274] In some example implementations, the predetermined order for signaling candidates for MCP may start from the left MCP block in the top MCP region and all MCP blocks follow in a clockwise direction until the top MCP block in the left MCP region.
[0275] Fig.28 A picture (2800) having a picture boundary in some exemplary implementations is shown. The MCP regions (2801) to (2804) extend from the picture boundary. Fig.28 Also shown is the predetermined order as described above, which starts from the left MCP block (2810) in the top MCP area (2801) and all MCP blocks follow in a clockwise direction until the top MCP block (2899) in the left MCP area (2804).
[0276] In some other exemplary implementations, the predetermined order may be starting from the top MCP area (2801) from left to right, to the bottom MCP area (2803) from left to right, to the left MCP area (2804) from top to bottom, and finally to the right MCP area (2802) from top to bottom.
[0277] In some exemplary implementations, when filling pixels in an MCP block, the pixels in the MCP block may be corrected with an offset. The offset may be determined as a difference between a DC value of a reconstructed candidate block (e.g., a candidate block at one of the A, B, C positions) and a DC value of a corresponding reference block of the reconstructed candidate block in a reference picture.
[0278] According to another aspect of the present disclosure, for MCP blocks adjacent to a picture boundary (e.g., left boundary, right boundary, upper boundary, lower boundary) of a picture, motion information may be derived from all inter-frame minimum blocks (e.g., each 4×4 block) at the picture boundary in the picture. The motion information may be derived in various ways. For example, the most common motion information from the inter-frame minimum blocks at the picture boundary may be determined, and the most common motion information may be used for motion compensation of all MCP blocks adjacent to the picture boundary.
[0279] In some examples, the usage of motion information may be collected from all inter-frame minimum blocks at the corresponding picture boundary. In some examples, the usage of motion information may be collected only from the inter-frame minimum blocks with reference index 0 in L0 and / or L1. In one example, the usage of motion information may be collected only from the inter-frame minimum blocks with reference index 0 in L0. In another example, the usage of motion information is collected only from the inter-frame minimum blocks with reference index 1 in L1.
[0280] According to another aspect of the present disclosure, for an MCP block adjacent to a picture boundary (e.g., left boundary, right boundary, upper boundary, lower boundary), motion information may be derived from all inter-frame minimum blocks at the corresponding picture boundary. For example, an average motion vector (instead of the most commonly used motion information) may be determined and used for motion compensation of all MCP blocks adjacent to the picture boundary.
[0281] In some exemplary implementations, the average value of the motion vector may be calculated based on all inter-frame minimum blocks at the corresponding picture boundary. In some exemplary implementations, the average value of the motion vector may be calculated based only on the inter-frame minimum blocks with reference index 0 in L0 and / or L1. In one example, the average value of the motion vector is calculated based only on the inter-frame minimum blocks with reference index 0 in L0. In another example, the average value of the motion vector is calculated based only on the inter-frame minimum blocks with reference index 1 in L1.
[0282] According to an aspect of the present disclosure, when a boundary block is intra-coded and a related MCP block has no available motion information, repetition padding may be used for the related MCP block.
[0283] According to one aspect of the present disclosure, the motion information of the MCP block can be derived from spatial candidates and temporal candidates, such as one candidate among multiple (e.g., N1) neighboring candidates at a picture boundary and N2 temporal co-located blocks (temporal candidates) of the current block (e.g., the current MCP block). Examples of N2 values include, but are not limited to, integers, such as 1, 2, 3, 4, etc.
[0284] In some exemplary implementations, the N2 temporally co-located blocks refer to N2 sub-blocks located at N2 predetermined relative positions on a block in a reference picture, where the coordinates of the block in the reference picture are the same as the coordinates of the current block.
[0285] Fig.29 Schematic diagram showing N2=2 time co-located blocks in some examples. Fig.29 , block (2901) is in the reference picture, and block (2901) is at the same coordinates as the current MCP block of the current picture. As an example, two temporally co-located blocks at predetermined relative positions are shown as block or sub-block C0 at the middle position and block or sub-block C1 at the lower right position.
[0286] In some exemplary implementations, multiple spatial neighboring blocks (e.g., at adjacent positions) of the MCP block and N temporal co-located blocks of the MCP block may be scanned, and multiple spatial neighboring blocks may be checked before the temporal co-located blocks, and the first available candidate may be used as the motion information prediction value of the MCP block.
[0287] According to some aspects of the present disclosure, for samples at a picture boundary in a padded area of a picture, further adjustment of sample values may be performed. In some examples, the adjustment of sample values may be based on a deblocking process of padded samples at a picture boundary. In some examples, the adjustment of padded sample values may be based on a smoothing filtering process of padded samples at a picture boundary.
[0288] Fig.30An example logic flow 3000 according to the above implementation is shown. The logic flow 3000 starts at step S3001. At step S3010, a motion compensation filling (MCP) block within an MCP area around a picture boundary of a second picture is determined. At step S3020, a candidate block is selected from a plurality of candidate blocks, the plurality of candidate blocks being temporal blocks or spatial neighboring blocks of the MCP block at the picture boundary of the second picture. At step S3030, a motion vector is derived from the selected candidate block of the MCP block, the motion vector pointing to a reference block in a reference frame of the second picture. At step S3040, the MCP block is filled with pixel values according to the reference block. At step S3050, the second picture is used as an inter-frame prediction reference frame for the first picture, and the MCP block in the second picture is used for motion compensation to encode the first picture in the code stream. The logic flow 3000 ends at step S3099.
[0289] Fig.31 An example logic flow 3100 according to the above implementation is shown. The logic flow 3100 starts at step S3101. At step S3110, a motion compensation padding (MCP) area around the picture boundary of the second picture is determined. At step S3120, an MCP block within the MCP area is identified. At step S3130, motion information of a set of inter-frame minimum blocks at the picture boundary of the second picture is determined. At step S3140, a motion vector of the MCP block is determined based on the motion information. At step S3150, the MCP block is filled with pixel values based on the motion vector. At step S3160, the second picture is used as an inter-frame prediction reference frame for the first picture, and the MCP block in the second picture is used for motion compensation to encode the first picture in the code stream. The logic flow 3100 ends at step S3199.
[0290] The above operations can be combined or arranged in any number or order as needed. Two or more steps and / or operations can be performed in parallel. The embodiments and implementations in the present disclosure can be used alone or in combination in any order. In addition, each method (or embodiment), encoder and decoder can be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable medium. The embodiments in the present disclosure can be applied to luminance blocks or chrominance blocks. The term "block" can be interpreted as a prediction block, a coding block, or a coding unit (i.e., CU). The term "block" herein can also be used to refer to a transform block. In the following items, when a block size is mentioned, it can refer to the width or height of the block, or the maximum value of the width and height of the block, or the minimum value of the width and height of the block, or the area size of the block (width*height), or the aspect ratio of the block (width:height, or height:width).
[0291] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.32 A computer system (3200) suitable for implementing certain embodiments of the disclosed subject matter is shown.
[0292] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode, etc.
[0293] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, etc.
[0294] Fig.32 The components for the computer system (3200) shown in the example are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing the disclosed embodiments. The configuration of the components should also not be interpreted as having any dependency or requirement related to any one or combination of the components shown in the exemplary embodiment of the computer system (3200).
[0295] The computer system (3200) may include certain human-machine interface input devices. The input human-machine interface devices may include one or more of the following (only one of each is shown): keyboard (3201), mouse (3202), touch pad (3203), touch screen (3210), data gloves (not shown), joystick (3205), microphone (3206), scanner (3207), camera (3208).
[0296] The computer system (3200) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (3210), a data glove (not shown), or a joystick (3205), but may also be a tactile feedback device that is not an input device), an audio output device (e.g., a speaker (3209), headphones (not depicted)), a visual output device (e.g., a screen (3210) including a CRT screen, an LCD screen, a plasma screen, an OLED screen, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual output or output in more than three dimensions through, for example, stereo output, virtual reality glasses (not depicted), a holographic display, and a smoke box (not depicted)) and a printer (not depicted).
[0297] The computer system (3200) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (3220) or similar media (3221) including CD / DVD, thumb drives (3222), removable hard drives or solid-state drives (3223), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security dongles (not depicted), and the like.
[0298] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.
[0299] The computer system (3200) may also include an interface (3254) to one or more communication networks (3255). The network may be, for example, a wireless network, a wired network, a fiber optic network. The network may also be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, and the like. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CAN buses, and the like.
[0300] The above-mentioned human-machine interface device, human-machine accessible storage device and network interface may be attached to the kernel (3240) of the computer system (3200).
[0301] The kernel (3240) may include one or more central processing units (CPUs) (3241), graphics processing units (GPUs) (3242), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (3243), hardware accelerators (3244) for certain tasks, graphics adapters (3250), and the like. These devices, as well as read-only memory (ROM) (3245), random-access memory (RAM) (3246), and internal mass storage (3247) such as internal non-user accessible hard drives, SSDs, and the like, may be connected via a system bus (3248). In some computer systems, the system bus (3248) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the kernel's system bus (3248), or connected to the kernel's system bus (3248) via a peripheral bus (3249). In one example, a display (3210) may be connected to a graphics adapter (3250). The architecture of the peripheral bus includes PCI, USB, etc.
[0302] The computer readable medium may have thereon computer codes for performing various computer-implemented operations. The medium and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer codes may be of a type well known and available to those skilled in the art of computer software.
[0303] Although the present disclosure has described a number of exemplary embodiments, there are changes, permutations, and various replacement equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and therefore fall within the spirit and scope of the present disclosure.
Claims
1. A method for encoding a current block in a first picture of a video based on a second picture of the video in a bitstream, comprising: Determining a motion compensation padding (MCP) block within a MCP region around a picture boundary of the second picture; Selecting a candidate block from a plurality of candidate blocks, the plurality of candidate blocks being temporal blocks or spatial neighboring blocks of the MCP block at the picture boundary of the second picture; deriving a motion vector from the selected candidate block of the MCP block, the motion vector pointing to a reference block in a reference frame of the second picture; Filling the MCP block with pixel values according to the reference block; as well as The second picture is used as an inter-frame prediction reference frame of the first picture, and the MCP block in the second picture is used for motion compensation to encode the first picture in the code stream.
2. The method according to claim 1, wherein: The multiple candidate blocks include N1 spatial neighboring blocks and N2 time blocks of the MCP block, where N1 and N2 are positive integers.
3. The method according to claim 2, wherein: The N2 time blocks include N2 time sub-blocks, and the N2 time sub-blocks are located at N2 predetermined positions in the reference frame relative to the MCP block.
4. The method according to claim 3, wherein: N2=2, the N2 temporal sub-blocks in the reference frame are co-located with the center sub-block and the bottom-right sub-block of the MCP block of the second picture.
5. The method according to claim 2, wherein: The N1 spatial neighboring blocks are scanned before scanning the N2 temporal blocks to derive the motion vector of the MCP block of the second picture.
6. The method according to claim 5, wherein: Deriving the motion vector of the MCP block includes identifying a first candidate block having an available motion vector from among the plurality of candidate blocks.
7. The method according to claim 1, wherein: The MCP area corresponds to a plurality of rectangles, each rectangle extending outward from one of the picture boundaries of the second picture in units of pixels by an MCP range consisting of a predetermined number of pixels.
8. The method according to claim 1, wherein: All of the multiple candidate blocks or a subset of the multiple candidate blocks include a group of neighboring blocks of the MCP block in the second picture at a corresponding boundary of the second picture, wherein each neighboring block in the group of neighboring blocks has a predetermined block size, and the group of neighboring blocks corresponds to a predetermined relative position relative to the MCP block in the second picture.
9. The method according to claim 8, wherein: Deriving the motion vector of the MCP block includes identifying a first candidate block having an available motion vector from among the plurality of candidate blocks according to a predetermined scanning order.
10. The method according to claim 8, wherein: All or a subset of the plurality of candidate blocks further include one or more blocks in the reference frame of the second picture that are temporally co-located with a boundary block in the second picture corresponding to the MCP block.
11. The method according to claim 10, wherein: The one or more blocks in the reference frame of the second picture that are temporally co-located with the boundary block in the second picture corresponding to the MCP block include one or more sub-blocks of the boundary block, and the one or more sub-blocks are located at a set of predetermined positions within the boundary block relative to the middle sub-block of the boundary block.
12. The method according to claim 11, wherein: Deriving the motion vector of the MCP block includes: identifying a first candidate block with an available motion vector from the multiple candidate blocks according to a predetermined scanning order, wherein the set of neighboring blocks is first scanned, and then the one or more blocks that are temporally co-located with the boundary block in the second picture corresponding to the MCP block are scanned until the first candidate block is identified.
13. The method according to any one of claims 1 to 12, further comprising signaling in the code stream that the candidate block is selected from the plurality of candidate blocks.
14. The method according to claim 13, wherein: A binary file of the candidate index is used to signal the selection of a candidate block from first two candidate blocks of the plurality of candidate blocks.
15. The method according to claim 13, wherein: Variable length coding is used to signal one candidate block among three available candidate blocks among the plurality of candidate blocks.
16. The method according to any one of claims 1 to 12, wherein: When only one candidate block among the plurality of candidate blocks has an available motion vector, the selection of the candidate block is implicit and not signaled in the codestream.
17. The method according to any one of claims 1 to 12, wherein: The signaling of candidate block selection for all MCP blocks of the second picture is aggregated in a predetermined MCP block scanning order.
18. The method according to any one of claims 1 to 12, further comprising: determining an offset based on a first DC value of the selected candidate block and a second DC value of a reference block in the reference frame of the candidate block; as well as The offset is applied when filling the MCP block.
19. A method for encoding a first picture of a video based on a second picture of the video in a bitstream, comprising: determining a motion compensation padding (MCP) area around a picture boundary of the second picture; Identifying an MCP block within the MCP region; determining motion information of a set of inter-frame minimum blocks at the picture boundary of the second picture; Determine a motion vector of the MCP block according to the motion information; Filling the MCP block with pixel values according to the motion vector; as well as The second picture is used as an inter-frame prediction reference frame of the first picture, and the MCP block is used to perform motion compensation, so as to perform inter-frame encoding on the first picture in the code stream.
20. An electronic device for encoding a current block in a first picture of a video based on a second picture of the video in a bitstream, the electronic device comprising a memory storing instructions and a processor, the processor executing the instructions to perform: Determining a motion compensation padding (MCP) block within a MCP region around a picture boundary of the second picture; Selecting a candidate block from a plurality of candidate blocks, the plurality of candidate blocks being temporal blocks or spatial neighboring blocks of the MCP block at the picture boundary of the second picture; deriving a motion vector from the selected candidate block of the MCP block, the motion vector pointing to a reference block in a reference frame of the second picture; Filling the MCP block with pixel values according to the reference block; as well as The second picture is used as an inter-frame prediction reference frame of the first picture, and the MCP block in the second picture is used for motion compensation to encode the first picture in the code stream.
