Video coding method, apparatus, and non-temporary computer-readable medium

Abstract

To provide a method of video coding implemented in a decoding device or an encoding device.SOLUTION: A method comprises the steps of: obtaining initial motion vectors for a current block; obtaining first predictions for a sample value in the current block based on the initial motion vectors; calculating a first matching cost according to the first predictions; determining whether an optical flow refinement process should be performed or not, according to at least one preset condition, the at least one preset condition comprising a condition as to whether the calculated first matching cost is equal to or larger than a threshold; and performing an optical flow refinement process for obtaining a final inter prediction for the sample value in the current block, when it is determined that the optical flow refinement process should be performed.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] Cross-reference of related applications This patent application claims priority to Indian Provisional Patent Application No. IN201931007114, filed on 22 February 2019. The disclosure of the aforementioned patent application is incorporated herein by reference in its entirety.

[0002] This invention relates to the field of picture processing, and more particularly to the improvement (refinement) of optical flow. [Background technology]

[0003] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the internet and mobile networks, real-time conversation applications like video chat, video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders in security applications.

[0004] Even relatively short videos can require a considerable amount of video data, which can pose challenges when the data needs to be streamed or otherwise transmitted over communication networks with limited bandwidth. Therefore, video data is generally compressed before being transmitted over modern communication networks. Video size can also be a concern when video is stored on a storage device, as memory resources may be limited. Often, video compression devices use software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Given limited network resources and the increasing demand for higher video quality, improved compression and decompression techniques that increase the compression ratio with little to no sacrifice of picture quality are desirable.

[0005] Recently, interpredictive coding has been improved by bipredictive optical flow improvements. This technique has the potential to increase the accuracy of interprediction of the current block of the picture being coded. However, bipredictive optical flow improvements are relatively expensive in terms of computational load. Therefore, a compromise between accurate interprediction and computational load must be found. This invention addresses this problem. [Overview of the Initiative] [Means for solving the problem]

[0006] Embodiments of this application provide apparatus and methods for encoding and decoding according to independent claims.

[0007] The other objectives described above are achieved by the subject matter of the independent claims. Further forms of implementation are evident from the dependent claims, specification, and drawings.

[0008] A method of video coding implemented in a decoding device or an encoding device is provided. The method includes: obtaining an initial motion vector for a current block (e.g., a coding block, a prediction block, or a sub-block); obtaining a first prediction (two prediction values for inter biprediction) for sample values within the current block based on the initial motion vector; calculating a first matching cost (e.g., consisting of or including a measure of similarity (or difference); see also the detailed description below) according to the first prediction; determining whether an optical flow improvement process should be executed according to at least one preset condition, wherein the at least one preset condition includes the condition of whether the calculated first matching cost is greater than or equal to a threshold; executing an optical flow improvement process to obtain a final inter prediction for sample values within the current block when it is determined that the optical flow improvement process should be executed; and

[0009] Therefore, according to the present invention, improvement of optical flow, particularly improvement of bidirectional optical flow, is conditionally performed. The relatively expensive improvement of optical flow is performed only under specific circumstances that allow for a suitable desired improvement in the accuracy of the entire inter-prediction process. If it is determined that the improvement of optical flow probably does not result in an improvement in the accuracy of inter-prediction that is worth the relatively high computational load required to perform the improvement of optical flow, the improvement of optical flow may be suppressed. Thus, the decoding time can be significantly reduced. The initial motion vector may be signaled within the bitstream. Alternatively, prediction of the motion vector and components of the difference of the motion vector may be provided with respect to the initial motion vector.

[0010] For example, at least one preset condition includes the condition that the current block can be predicted by improvement of the motion vector on the decoder side. This particular condition included in at least one preset condition may be examined first to avoid unnecessary computational amount.

[0011] According to a particular embodiment, when it is determined that all of at least one preset condition is satisfied, it is determined that the optical flow improvement process should be executed. At least one preset condition may, in principle, include one or more additional conditions. For example, at least one preset condition may include the condition that a specific flag (e.g., 1) is set to have an optical improvement process being executed. If all the conditions are not satisfied, according to a particular embodiment, the improvement of optical flow may not be performed at all to reduce the computational demand.

[0012] The first prediction regarding the sample values ​​within the current block may be obtained based on a first interpolation filter to achieve some subpixel accuracy. In particular, the first interpolation filter may be a relatively simple bilinear interpolation filter that allows for fast filtering.

[0013] A method of video coding invention performed in a decoding or encoding device may include some motion vector improvement distinct from optical flow improvement. Therefore, the method may include the steps of: obtaining an improved motion vector based on an initial motion vector and a first matching cost; obtaining a second prediction for sample values ​​in the current block according to the improved motion vector; and performing an optical flow improvement based on the second prediction (representing the already improved prediction) when it is determined that an optical flow improvement process should be performed. The overall accuracy of the interpretation process may be enhanced by the use of the improved motion vector.

[0014] It should be noted that the first prediction and first matching costs have already been calculated for motion vector improvement. Therefore, no additional calculations are needed to determine early termination / suppression of the optical flow improvement process, and the results of previous calculations included in the motion vector improvement process can be reused.

[0015] In each of the above embodiments of the method of the invention, the step of obtaining a first prediction about sample values ​​in the current block based on an initial motion vector may include the step of obtaining several pairs of candidates based on an initial motion vector, and the step of obtaining a first prediction about sample values ​​in the current block based on at least one of the pairs of candidates, and the step of calculating a first matching cost according to the first prediction may include the step of calculating a matching cost for each of the pairs of candidates based on the first prediction, and the step of determining the smallest matching cost among the determined matching costs as the first matching cost.

[0016] In the process of improving motion vectors, several pairs of candidates for the improved motion vectors may be obtained, each pair including a pair of initial motion vectors. For example, a pair of candidates for the improved motion vectors includes the initial motion vector pair (MV0, MV1) and pairs (MV0 + (0,1), MV1 + (0,-1)), (MV0 + (1,0), MV1 + (-1,0)), (MV0 + (0,-1), MV1 + (0,1)), (MV0 + (-1,0), MV1 + (1,0)), where (1,-1) represents a vector with a horizontal (or x) displacement of 1 and a vertical (or y) displacement of -1. For each pair, it is possible to determine a matching cost corresponding to that pair, and the first matching cost described above can be determined to be the smallest matching cost among the matching costs determined for the candidate pairs of improved motion vectors. In a particular example, the first matching cost can be the matching cost corresponding to the pair of initial motion vectors (MV0, MV1) or the improved motion vectors MV0' and MV1' (MV0' = MV0 + (0,1), MV1' = MV1 + (0,-1)).

[0017] The use of that type of primary matching cost can be advantageous from an overall coding perspective.

[0018] The second prediction mentioned above regarding the sample values ​​within the current block may be obtained according to a second interpolation filter. This second interpolation filter may be a 6-tap or 8-tap interpolation filter, which is relatively expensive but advantageous in terms of subpixel accuracy.

[0019] The improved motion vector described above may be obtained according to a second matching cost to control the suitability of the improved motion vector for interpretation. If the value of the second matching cost is greater than or equal to another threshold, it may be determined that an optical flow improvement process should be performed. Otherwise, it may be determined that no optical flow improvement process is worth performing.

[0020] According to another embodiment, the final interpretation is obtained by a weighted sum of second predictions only when it is determined that the optical flow improvement process should not be performed. When it is not considered appropriate to perform the relatively costly optical flow improvement process, the weighted sum of second predictions provides some degree of accuracy that may be considered sufficient.

[0021] Generally, a threshold, or another threshold, can be a value calculated based on the bit depth of the first prediction. Furthermore, a threshold can be obtained according to the number of predicted samples used to calculate the first matching cost according to the first prediction. Additionally, a threshold can be obtained according to the size of the current block (width and height expressed in pixels). For example, the threshold could be thr = nCbW × nCbH × K, where K is a value greater than zero, and nCbW and nCbH are the width and height of the current block. For example, K = 2.

[0022] Furthermore, the second matching cost mentioned above may be the cost obtained using a predefined model of the matching costs evaluated during the improvement of the motion vector and the shape of the matching costs near the location of the minimum matching cost. The predefined model in this context may be a linear combination model. Using a predefined model of the shape of the matching costs near the location of the minimum matching cost may improve the accuracy of the interpretation process.

[0023] The methods according to all of the embodiments described above may further include the step of generating an interprediction block that includes a final interprediction for the sample values ​​in the current block.

[0024] Furthermore, an encoder or decoder is provided, which includes some processing circuit for performing the method according to any one of the embodiments described above. Furthermore, a computer program product is provided, which includes program code for performing the method according to any one of the embodiments described above.

[0025] All of the above-described variations of the video coding method can be implemented in a decoder or encoder. Thus, a decoder or encoder is provided, comprising one or more processors and a non-temporary computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program configures the decoder to perform the method according to any one of the embodiments described above when executed by the processors.

[0026] All of the above variations of the video coding method can be implemented in a device for use in an image encoder and / or image decoder to address the above-mentioned needs. Accordingly, a device for use in an image encoder and / or image decoder is provided, the device including: an initial motion vector unit configured to acquire an initial motion vector with respect to the current block (e.g., a coding block or a prediction block or a subblock); a first prediction unit configured to acquire a first prediction about sample values ​​in the current block based on the initial motion vector; a first matching cost calculation unit configured to calculate a first matching cost (e.g., a measure of similarity or difference) according to the first prediction; an optical flow improvement process determination unit configured to determine whether an optical flow improvement process should be performed according to at least one pre-defined condition, wherein at least one pre-defined condition includes whether the calculated first matching cost is greater than or equal to a threshold; and an optical flow improvement process execution unit configured to execute the optical flow improvement process to acquire a final inter-prediction about sample values ​​in the current block when it is determined that an optical flow improvement process should be performed.

[0027] This device, described above and its variations described below, offers the same advantages as the method described above.

[0028] At least one pre-set condition may include the condition that the current block can be predicted by an improvement in the decoder's motion vector.

[0029] The optical flow improvement process determination unit may be configured to determine that the optical flow improvement process should be executed when it is determined that all of at least one pre-defined condition is met.

[0030] The device may include a first interpolation filter (for example, a bilinear interpolation filter), and the first prediction unit may be configured to obtain a first prediction about the sample values ​​in the current block by the first interpolation filter.

[0031] The device includes an improved motion vector unit configured to obtain an improved motion vector based on the initial motion vector and a first matching cost, A second prediction unit configured to obtain a second prediction about the sample values ​​in the current block according to an improved motion vector, It may also include, The optical flow improvement process execution unit may be configured to perform optical flow improvements based on a second prediction when the optical flow improvement process determination unit determines that the optical flow improvement process should be executed.

[0032] In the above-described embodiment of the device, the first prediction unit may be configured to obtain a first prediction about sample values ​​in the current block based on an initial motion vector by obtaining several pairs of candidates based on an initial motion vector and obtaining a first prediction about sample values ​​in the current block based on at least one of the candidate pairs. Furthermore, the first matching cost calculation unit may be configured to calculate a first matching cost according to the first prediction by determining the matching cost for each of the candidate pairs based on the first prediction and determining the smallest matching cost among the determined matching costs as the first matching cost.

[0033] According to the embodiment, the device may further include a second interpolation filter (for example, a relatively expensive 6-tap or 8-tap interpolation filter with relatively high subpixel accuracy), and a second prediction unit may be configured to obtain a second prediction regarding the sample values ​​in the current block by the second interpolation filter.

[0034] According to another embodiment, the device further includes a second matching cost calculation unit configured to calculate a second matching cost, and an improved motion vector unit configured to obtain an improved motion vector according to the second matching cost. In this case, an optical flow improvement process determination unit may be configured to determine that an optical flow improvement process should be performed when the value of the second matching cost is greater than or equal to another threshold.

[0035] The device may further include a weighted sum prediction unit configured to obtain the final inter-prediction by a weighted sum of second predictions only when the optical flow improvement process determination unit determines that the optical flow improvement process should not be performed.

[0036] Furthermore, the device may include a threshold calculation unit configured to calculate one threshold or another based on the bit depth of a first prediction. The device may also further include a threshold calculation unit configured to calculate a threshold according to the number of predicted samples used by the first matching cost calculation unit to calculate the first matching cost according to the first prediction. The device may also further include a threshold calculation unit configured to calculate a threshold according to the size of the current block. For example, the threshold could be thr = nCbW × nCbH × K, where K is a value greater than zero, and nCbW and nCbH are the width and height of the current block. For example, K = 2.

[0037] According to a particular embodiment, the second matching cost calculation unit is configured to calculate the second matching cost as the derived cost obtained using a predefined model (e.g., a linear combination model) relating to the matching cost evaluated during the motion vector improvement performed by the improved motion vector unit and the shape of the matching cost near the location of the minimum matching cost.

[0038] A device according to any of the embodiments described above may further include an interprediction block generation unit configured to generate an interprediction block containing a final interprediction for sample values ​​in the current block.

[0039] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the specification, drawings, and claims.

[0040] Embodiments of the present invention will be described in more detail below with reference to the accompanying figures and drawings. [Brief explanation of the drawing]

[0041] [Figure 1A] This is a block diagram showing an example of a video coding system configured to implement an embodiment of the present invention. [Figure 1B] This is a block diagram showing another example of a video coding system configured to implement embodiments of the present invention. [Figure 2] This is a block diagram showing an example of a video encoder configured to implement an embodiment of the present invention. [Figure 3] This is a block diagram illustrating an exemplary structure of a video decoder configured to implement an embodiment of the present invention. [Figure 4] This is a block diagram showing examples of encoding or decoding devices. [Figure 5] This is a block diagram showing another example of an encoding or decoding device. [Figure 6] This is a flowchart illustrating an embodiment of the optical improvement process. [Figure 7] This is a flowchart illustrating another embodiment of the optical improvement process. [Figure 8] This is a flowchart illustrating another embodiment of the optical improvement process. [Figure 9] This is a flowchart illustrating another embodiment of the optical improvement process. [Figure 10] This is a flowchart illustrating a video coding method performed in a decoding device or encoding device according to an embodiment of the present invention. [Figure 11] This figure shows a device for use in an image encoder and / or image decoder according to an embodiment of the present invention. [Modes for carrying out the invention]

[0042] In the following, unless otherwise specified, the same reference numeral refers to the same or at least functionally equivalent feature.

[0043] In the following description, references are made to the accompanying drawings, which form part of this disclosure and illustrate specific embodiments of the invention or specific embodiments of the invention in which they may be used. It is understood that embodiments of the invention may be used in other embodiments and may include structural or logical modifications not shown in the drawings. Accordingly, the following detailed description should not be understood to be restrictive, and the scope of the invention is defined by the appended claims.

[0044] For example, disclosures relating to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more steps of a particular method are described, a corresponding device may include one or more units for performing the steps of the described method, e.g., functional units (e.g., one unit performing one or more steps, or multiple units each performing one or more of the steps), even if such one or more units are not explicitly described or shown in the figures. Conversely, if a particular apparatus is described based on one or more units, e.g., functional units, a corresponding method may include one step for performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps each performing one or more of the functions of the units), even if such one or more steps are not explicitly described or shown in the figures. Furthermore, it is understood that the various exemplary embodiments and / or features of the aspects described herein may be combined with each other unless otherwise specified.

[0045] Video coding generally refers to the processing of a sequence of pictures that make up a video or video sequence. Instead of the term "picture," the terms "frame" or "image" may be used as synonyms in the field of video coding. Video coding (or coding in general) consists of two parts: video encoding and video decoding. Video encoding is performed on the source side and generally involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed on the destination side and generally involves the reverse processing compared to the encoder to reconstruct the video picture. Embodiments referring to "coding" of a video picture (or picture in general) are understood to be relating to the "encoding" or "decoding" of the video picture or each video sequence. The combination of the encoding and decoding parts is also called a codec (coding and decoding).

[0046] In lossless video coding, the original video picture can be reconstructed (assuming there is no transmission loss or other data loss during storage or transmission), meaning the reconstructed video picture will have the same quality as the original. In lossy video coding, further compression is performed, for example, by quantization, to reduce the amount of data representing the video picture, which cannot be fully reconstructed in the decoder, meaning the quality of the reconstructed video picture will be lower or worse than the quality of the original video picture.

[0047] Some video coding standards belong to the group of "lossy hybrid video codecs" (i.e., they combine spatial and temporal prediction in the sample domain with 2D transform coding to apply quantization in the transform domain). Each picture in a video sequence is generally divided into a set of non-overlapping blocks, and coding is generally performed at the block level. In other words, in an encoder, video is generally processed at the block (video block) level, i.e., encoded, by generating prediction blocks using, for example, spatial (intra-picture) and / or temporal (inter-picture) predictions, obtaining residual blocks by subtracting the prediction blocks from the current blocks (the blocks currently being processed), transforming the residual blocks, and quantizing the residual blocks in the transform domain to reduce (compress) the amount of data being transmitted. In a decoder, the reverse process compared to the encoder is applied to the encoded or compressed blocks in order to reconstruct the current blocks for representation. Furthermore, the encoder duplicates the decoder's processing loop to process subsequent blocks, that is, to generate identical predictions (e.g., intra and inter predictions) and / or reconstructions for coding.

[0048] Embodiments of the video coding system 10, video encoder 20, and video decoder 30 are described below with reference to Figures 1 to 3.

[0049] Figure 1A is a schematic block diagram showing an exemplary coding system 10 that may utilize the technology of the present application, for example, a video coding system 10 (or coding system 10 in short). The video encoder 20 (or encoder 20 in short) and video decoder 30 (or decoder 30 in short) of the video coding system 10 show examples of devices that may be configured to perform the technology described in the various examples in this application.

[0050] As shown in Figure 1A, the coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14, for example, in order to decode the encoded picture data 13.

[0051] The source device 12 includes an encoder 20 and may additionally, or optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18, and a communication interface or communication unit 22.

[0052] The picture source 16 may include or be any type of picture-taking device, e.g., a camera for taking pictures of the real world, and / or any type of picture-generating device, e.g., a computer graphics processor for generating computer-animated pictures, or any other type of device for acquiring and / or providing pictures of the real world, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may be any type of memory or storage for storing any of the pictures described above.

[0053] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 may also be called the raw picture or raw picture data 17.

[0054] The preprocessor 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may be an arbitrary component.

[0055] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21 (further details are described below, for example, based on Figure 2).

[0056] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via the communication channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0057] The destination device 14 includes a decoder 30 (for example, a video decoder 30) and may additionally, or optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0058] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof) directly from the source device 12 or from any other source, such as a storage device, such as a storage device for encoded picture data, and to provide the encoded picture data 21 to the decoder 30.

[0059] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0060] The communication interface 22 may be configured to process the encoded picture data using any kind of encoding or processing for transmission, such as packaging the encoded picture data 21 into an appropriate format, for example, packets, and / or transmitting it over a communication link or communication network.

[0061] The communication interface 28 that forms the counterpart to the communication interface 22 may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding decryption or processing and / or depackaging of the transmission to obtain encoded picture data 21.

[0062] Both communication interface 22 and communication interface 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated by the arrows relating to communication channel 13 in Figure 1A pointing from source device 12 to destination device 14, and may be configured, for example, to set up a connection and to send and receive messages to confirm and exchange any other information related to the communication link and / or data transmission, such as the transmission of encoded picture data.

[0063] The decoder 30 is configured to receive the encoded picture data 21 and provide the decoded picture data 31 or the decoded picture 31 (further details are described below, for example, based on Figure 3 or Figure 5).

[0064] The post-processor 32 of the destination device 14 is configured to obtain post-processed picture data 33, for example, a post-processed picture 33, by post-processing the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31. Post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display by, for example, the display device 34.

[0065] The display device 34 of the destination device 14 is configured to receive, for example, picture data 33 that has been post-processed for displaying the picture to a user or viewer. The display device 34 may be any type of display for showing the reconstructed picture, for example, an integrated or external display or monitor, or may include such a display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0066] Figure 1A shows the source device 12 and destination device 14 as separate devices, but the device embodiment may include both or both functions, such as the source device 12 or its corresponding function and the destination device 14 or its corresponding function. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0067] As will become apparent to those skilled in the art based on the description, the functions of different units or the presence and (strict) division of functions within the source device 12 and / or destination device 14 shown in Figure 1A may vary depending on the actual device and application.

[0068] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented by the processing circuitry shown in Figure 1B, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof dedicated to video coding. The encoder 20 may be implemented by the processing circuitry 46 to embody various modules considered in relation to the encoder 20 of Figure 2 and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented by the processing circuitry 46 to embody various modules considered in relation to the decoder 30 of Figure 3 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations, which will be considered later. As shown in Figure 5, if the technology is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable storage medium and execute the instructions in hardware that uses one or more processors to perform the technology of this disclosure. Either the video encoder 20 or the video decoder 30 may be incorporated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in Figure 1B.

[0069] The source device 12 and destination device 14 may include any type of handheld or stationary device, including a wide range of devices such as notebook or laptop computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, and broadcast transmitter devices, and may or may not use an operating system. In some cases, the source device 12 and destination device 14 may be wireless communication devices. Therefore, the source device 12 and destination device 14 may be wireless communication devices.

[0070] In some cases, the video coding system 10 shown in Figure 1A is merely an example, and the techniques of this disclosure may apply to video coding situations (e.g., encoding or decoding video) that do not necessarily involve any data communication between the encoding device and the decoding device. In other examples, the data may be retrieved from local memory or streamed over a network. The video encoding device may encode the data and store it in memory, and / or the video decoding device may retrieve the data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode the data into memory and / or retrieve the data from memory and decode it.

[0071] For convenience of explanation, embodiments of the present invention are described herein by reference to, for example, reference software for next-generation video coding standards developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG) Joint Collaboration Team on Video Coding (JCT-VC). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0072] Encoder and encoding method Figure 2 shows a schematic block diagram of an exemplary video encoder 20 configured to implement the technology of the present application. In the example of Figure 2, the video encoder 20 includes an input 201 (or input interface 201), a residual calculation unit 204, a transformation unit 206, a quantization unit 208, a dequantization unit 210, an inverse transformation unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output 272 (or output interface 272). The mode selection unit 260 may include an inter-prediction unit 244, an intra-prediction unit 254, and a partitioning unit 262. The inter-prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in Figure 2 may also be called a hybrid video encoder or a video encoder with a hybrid video codec.

[0073] The residual calculation unit 204, the conversion processing unit 206, the quantization unit 208, and the mode selection unit 260 can be considered to form the forward signal path of the encoder 20, while the dequantization unit 210, the inverse conversion processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoding picture buffer (DPB) 230, the inter-prediction unit 244, and the intra-prediction unit 254 can be considered to form the reverse signal path of the video encoder 20, and the reverse signal path of the video encoder 20 corresponds to the signal path of the decoder (see video decoder 30 in Figure 3). The dequantization unit 210, the inverse conversion processing unit 212, the reconstruction unit 214, the loop filter 220, the decoding picture buffer (DPB) 230, the inter-prediction unit 244, and the intra-prediction unit 254 can also be considered to form the “built-in decoder” of the video encoder 20.

[0074] Picture & Picture Separation (Picture & Block) The encoder 20 may be configured to receive, for example, a picture 17 (or picture data 17) via input 201, for example, a picture of a sequence of pictures that form a video or video sequence. The received picture or picture data may also be a pre-processed picture 19 (or pre-processed picture data 19). For simplicity, the following description will refer to picture 17. Picture 17 may also be called the current picture or the picture being coded (particularly in video coding to distinguish the current picture from other pictures, for example, already coded and / or coded pictures of the same video sequence, i.e., the video sequence that also contains the current picture).

[0075] A (digital) picture can be considered, or may be considered, a two-dimensional array or matrix of samples having intensity values. A sample in an array may also be called a pixel (a shortened form of picture element) or pel. The number of samples in the horizontal and vertical (or axis) directions of an array or picture defines the size and / or resolution of the picture. For color representation, generally three color components are used, meaning a picture may be represented by or contain three sample arrays. In the RGB format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is generally represented by a luminance and chrominance format or color space, for example, YCbCr, which includes a luminance component represented by Y (sometimes L is used instead) and two chrominance components represented by Cb and Cr. The luminance (or luma) component Y represents the brightness or intensity of the gray level (for example, as in a grayscale picture), while the two chrominance (or chroma) components Cb and Cr represent the chromaticity or color information components. Therefore, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format can be converted to or transformed into YCbCr format, and vice versa; the process is also known as transformation or conversion. If a picture is monochrome, it may contain only a luminance sample array. Thus, a picture may be, for example, a luma sample array in a monochrome format, or a luma sample array and two corresponding chroma sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0076] Embodiments of the video encoder 20 may include a picture partitioning unit (not shown in Figure 2) configured to partition a picture 17 into multiple (typically non-overlapping) picture blocks 203. These blocks may also be called root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTB) or coding tree units (CTU) (H.265 / HEVC and VVC). The picture partitioning unit may use the same block size for all pictures and corresponding grids that define the block size for all pictures in the video sequence, or it may be configured to change the block size between pictures or subsets or groups of pictures, partitioning each picture into its corresponding block.

[0077] In a further embodiment, the video encoder may be configured to directly receive a block 203 of picture 17, for example, one, some, or all of the blocks that make up picture 17. The picture block 203 may also be called the current picture block or the coded picture block.

[0078] Similar to picture 17, picture block 203 is smaller in dimensions than picture 17, but can also be considered, or may be considered, a two-dimensional array or matrix of samples having intensity values ​​(sample values). In other words, block 203 may contain, depending on the color format applied, for example, one sample array (e.g., a luma array for monochrome picture 17, or a luma or chroma array for color picture), three sample arrays (e.g., a luma and two chroma arrays for color picture 17), or any other number and / or type of array. The number of samples in the horizontal and vertical (or axis) directions of block 203 defines the size of block 203. Thus, a block may be, for example, an MxN (M columns × N rows) array of samples or an MxN array of conversion coefficients.

[0079] The embodiment of the video encoder 20 shown in Figure 2 may be configured to encode the picture 17 block by block, for example, encoding and prediction being performed for each block 203.

[0080] The embodiment of the video encoder 20 shown in Figure 2 may be further configured to divide and / or encode a picture by using slices (also called video slices), the picture may be divided into one or more (generally non-overlapping) slices or encoded using one or more (generally non-overlapping) slices, each slice may contain one or more blocks (e.g., CTUs).

[0081] Embodiments of the video encoder 20 shown in Figure 2 may be further configured to partition and / or encode a picture by using tile groups (also called video tile groups) and / or tiles (also called video tiles), the picture may be partitioned into one or more (generally non-overlapping) tile groups or encoded using one or more (generally non-overlapping) tile groups, each tile group may, for example, contain one or more blocks (e.g., CTUs) or one or more tiles, each tile may, for example, be rectangular in shape and may contain one or more blocks (e.g., CTUs), for example, complete or fragmental blocks.

[0082] Calculation of residuals The residual calculation unit 204 may be configured to calculate the residual block 205 (also called residual 205) based on picture block 203 and prediction block 265 (further details about prediction block 265 will be given later), for example, by subtracting the sample value of prediction block 265 from the sample value of picture block 203 for each sample (for each pixel) to obtain the residual block 205 in the sample region.

[0083] conversion The transformation processing unit 206 may be configured to apply a transformation, such as a discrete cosine transform (DCT) or discrete sine transform (DST), to the sample values ​​of the residual block 205 to obtain transformation coefficients 207 in the transformation domain. The transformation coefficients 207, also called transformation residual coefficients, may represent the residual block 205 in the transformation domain.

[0084] The conversion processing unit 206 may be configured to apply an integer approximation of DCT / DST, such as the conversion specified for H.265 / HEVC. Compared to the orthogonal DCT conversion, such an integer approximation is generally scaled by a certain rate. An additional scaling factor is applied as part of the conversion process to maintain the norm of the residual blocks processed by the forward and inverse conversions. The scaling factor is generally selected based on certain constraints, such as the scaling factor being a power of 2 for the shift operation, the bit depth of the conversion coefficients, and the trade-off between accuracy and implementation cost. For example, a particular scaling factor may be specified for the inverse conversion by the inverse conversion processing unit 212 (and the corresponding inverse conversion by the inverse conversion processing unit 312 in the video decoder 30, for example), and a corresponding scaling factor for the forward conversion by the conversion processing unit 206 of the encoder 20 may be specified accordingly.

[0085] Embodiments of the video encoder 20 (each a conversion processing unit 206) may be configured to output, for example, one or more conversions of a certain type, which may be left as is or encoded or compressed by the entropy coding unit 270, so that the video decoder 30 may receive the conversion parameters and use them for decoding.

[0086] quantization The quantization unit 208 may be configured to quantize the transformation coefficient 207 to obtain the quantized coefficient 209, for example, by applying scalar quantization or vector quantization. The quantized coefficient 209 may also be called the quantized transformation coefficient 209 or the quantized residual coefficient 209.

[0087] The quantization process may reduce the bit depth associated with some or all of the 207 conversion coefficients. For example, an n-bit conversion coefficient may be truncated to an m-bit conversion coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting the quantization parameter (QP). For example, with respect to scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. Applicable quantization step sizes may be indicated by the quantization parameter (QP). The quantization parameter may be an index to a predefined set of applicable quantization step sizes. For example, a small quantization parameter may correspond to finer quantization (smaller quantization step size), a large quantization parameter may correspond to coarser quantization (larger quantization step size), or vice versa. Quantization may involve division by the quantization step size, and the corresponding and / or inverse dequantization by the dequantization unit 210 may involve multiplication by the quantization step size. Some standards, e.g., embodiments by HEVC, may be configured to determine the quantization step size using quantization parameters. Generally, the quantization step size may be calculated based on the quantization parameters using a fixed-point approximation of the equations, which involves division. Additional multipliers may be introduced with respect to quantization and dequantization to restore the norm of the residual block, which may be modified due to the scaling used in the fixed-point approximation of the equations with respect to the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Alternatively, a customized quantization table may be used and signaled, for example, from encoder to decoder within the bitstream.Quantization is an irreversible operation, and the loss increases as the quantization step size increases.

[0088] Embodiments of the video encoder 20 (each a quantization unit 208) may be configured to output quantization parameters (QP) that are either raw or encoded by the entropy coding unit 270, for example, so that the video decoder 30 may receive the quantization parameters and apply them for decoding.

[0089] Dequantization The dequantization unit 210 is configured to obtain dequantized coefficients 211 by applying the dequantization of quantization unit 208 to the quantized coefficients, for example, by applying the inverse of the quantization scheme applied by quantization unit 208 to the quantized coefficients, based on or using the same quantization step size as quantization unit 208. The dequantized coefficients 211 are also called dequantized residual coefficients 211 and may correspond to the conversion coefficients 207—although they are generally not identical to the conversion coefficients due to losses due to quantization.

[0090] Inverse Transform The inverse transform processing unit 212 is configured to obtain a reconstructed residual block 213 (or the corresponding dequantized coefficient 213) in the sample region by applying the inverse transform of the transform applied by the transform processing unit 206, for example, an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) or other inverse transform. The reconstructed residual block 213 may also be called a transform block 213.

[0091] Rebuild The reconstruction unit 214 (for example, an adder or summer 214) is configured to obtain the reconstructed block 215 in the sample region by adding the transformed block 213 (i.e., the reconstructed residual block 213) to the predicted block 265 by adding the sample values ​​of the reconstructed residual block 213 and the sample values ​​of the predicted block 265 --sample by sample.

[0092] filtering The loop filter unit 220 (or simply "loop filter" 220) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or more generally, to filter the reconstructed sample to obtain a filtered sample. The loop filter unit is configured, for example, to smooth pixel transitions or otherwise improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening, smoothing filter, or a collaborative filter, or any combination thereof. The loop filter unit 220 is shown in Figure 2 as an in-loop filter, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be called a filtered reconstructed block 221.

[0093] Embodiments of the video encoder 20 (each a loop filter unit 220) may be configured to output loop filter parameters (such as sample-adaptive offset information) that are either raw or encoded by the entropy coding unit 270, for example, so that a decoder 30 may receive the same loop filter parameters or each respective loop filter and apply them for decoding.

[0094] Decode picture buffer The decoded picture buffer (DPB) 230 may be a memory that stores a reference picture or generally reference picture data for encoding video data by the video encoder 20. The DPB 230 may be formed by one of various memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may be further configured to store the same current picture or a different picture, for example, other already filtered blocks of an already reconstructed picture, for example, an already reconstructed and filtered block 221, for example, to provide a complete already reconstructed, i.e., decoded picture (and corresponding reference blocks and samples) and / or a partially reconstructed current picture (and corresponding reference blocks and samples) for interpretation. The decoded picture buffer (DPB) 230 may also be configured to store, for example, one or more unfiltered reconstructed blocks 215 or generally unfiltered reconstructed samples if the reconstructed blocks 215 are not filtered by the loop filter unit 220, or to store any other further processed versions of the reconstructed blocks or samples.

[0095] Mode Selection (Classification & Prediction) The mode selection unit 260 includes a partitioning unit 262, an inter-prediction unit 244, and an intra-prediction unit 254, and is configured to receive or acquire original picture data, for example, the original block 203 (the current block 203 of the current picture 17), and reconstructed picture data, for example, filtered and / or unfiltered reconstructed samples or blocks from one or more already decoded pictures of the same (current) picture and / or from, for example, the decoded picture buffer 230 or other buffers (for example, line buffers not shown). The reconstructed picture data is used as reference picture data for predictions, for example, inter-prediction or intra-prediction, to acquire prediction blocks 265 or predictor 265.

[0096] The mode selection unit 260 may be configured to determine or select a partitioning and prediction mode (e.g., intra or inter-prediction mode) for the prediction mode of the current block (without partitioning), and to generate a corresponding prediction block 265 used for calculating the residual block 205 and reconstructing the reconstructed block 215.

[0097] Embodiments of the mode selection unit 260 may be configured to select a partitioning and prediction mode (for example, from partitioning and prediction modes supported by or available to the mode selection unit 260) that provides the best match or, in other words, the smallest residual (smallest residual means better compression for transmission or storage) or the smallest signaling overhead (smallest signaling overhead means better compression for transmission or storage), or that considers both or balances them. The mode selection unit 260 may be configured to determine the partitioning and prediction mode based on rate distortion optimization (RDO), i.e., to select a prediction mode that provides the smallest rate distortion. In this context, terms such as “best,” “smallest,” and “optimal” do not necessarily refer to the overall “best,” “smallest,” and “optimal,” and may also refer to termination or selection criteria such as a value being above or below a threshold, or potentially leading to a “suboptimal choice,” but satisfying other constraints that reduce complexity and processing time.

[0098] In other words, the partitioning unit 262 may be configured to partition block 203 into smaller block partitions or subblocks (which also form a block) using, for example, quadtree partitioning (QT), binary partitioning (BT), or ternary partitioning (TT), or any combination thereof, iteratively, and to perform predictions with respect to each of the block partitions or subblocks, mode selection including selection of the tree structure of the partitioned block 203, and prediction mode applied to each of the block partitions or subblocks.

[0099] The following describes in more detail the sorting (by the sorting unit 260, for example) and prediction (by the inter-prediction unit 244 and intra-prediction unit 254) processes performed by the exemplary video encoder 20.

[0100] classification The partitioning unit 262 may partition (or divide) the current block 203 into smaller partitions, for example, smaller blocks the size of a square or rectangle. These smaller blocks (which may also be called subblocks) may be further partitioned into even smaller partitions. This is also called tree partitioning or hierarchical tree partitioning, for example, the root block at root tree level 0 (hierarchy level 0, depth 0) may be recursively partitioned, for example, into two or more blocks at the next lowest tree level, for example, into nodes at tree level 1 (hierarchy level 1, depth 1), and these blocks may again be partitioned into two or more blocks at the next lowest level, for example, tree level 2 (hierarchy level 2, depth 2), and so on until a termination criterion is met, for example, the maximum tree depth or the minimum block size is reached and partitioning ends, and so on. Blocks that are not further partitioned are also called leaf blocks or leaf nodes of the tree. A tree that uses partitioning into two sections is called a binary tree (BT), a tree that uses partitioning into three sections is called a ternary tree (TT), and a tree that uses partitioning into four sections is called a quaternary tree (QT).

[0101] As stated herein, the term “block” may refer to a portion of a picture, particularly a portion of a square or rectangle. For example, in relation to HEVC and VVC, a block may be a coding tree unit (CTU), coding unit (CU), prediction unit (PU), and transform unit (TU), as well as / or corresponding blocks, such as a coding tree block (CTB), coding block (CB), transform block (TB), or prediction block (PB).

[0102] For example, a coding tree unit (CTU) could be or include a CTB of a luminous sample, two corresponding CTBs of a chroma sample of a picture having three sample sequences, or a CTB of a sample of a picture coded using three separate color planes and syntax structures used to code a monochrome picture or sample. Correspondingly, a coding tree block (CTB) could be an NxN block of samples for some value of N such that the division of its components into CTBs is a partition. A coding unit (CU) could be or include a coding block of a luminous sample, two corresponding coding blocks of a chroma sample of a picture having three sample sequences, or a coding block of a sample of a picture coded using three separate color planes and syntax structures used to code a monochrome picture or sample. Correspondingly, a coding block (CB) could be an MxN block of samples for some values ​​of M and N such that the division of the CTB into coding blocks is a partition.

[0103] For example, in an embodiment using HEVC, a coding tree unit (CTU) may be divided into CUs by using a quadtree structure that is represented as a coding tree. The decision of whether to code a picture area using interpicture (time) prediction or intrapicture (spatial) prediction is made at the CU level. Each CU may be further divided into one, two, or four PUs according to the PU division type. Within a single PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU division type, the CU may be divided into transformation units (TUs) by another quadtree structure similar to a coding tree for the CU.

[0104] For example, in embodiments of the latest video coding standards currently under development, called Multipurpose Video Coding (VVC), combined quadtree and binary tree (QTBT) partitions are used, for example, to partition coding blocks. In the QTBT block structure, CUs can have either a square or rectangular shape. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The leaf nodes of the quadtree are further partitioned by a binary or ternary (or triple) tree structure. The leaf nodes of the partitioning tree are called coding units (CUs), and their segmentation is used for prediction and transformation processing without any further partitioning. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In parallel, multipartitions, such as ternary partitions, can be used in conjunction with the QTBT block structure.

[0105] In one example, the mode selection unit 260 of the video encoder 20 may be configured to perform any combination of the classification techniques described herein.

[0106] As described above, the video encoder 20 is configured to determine or select the best or optimal prediction mode from a set of (for example, predetermined) prediction modes. The set of prediction modes may include, for example, an intra-prediction mode and / or an inter-prediction mode.

[0107] Intra Prediction A set of intra-predictive modes may include, for example, 35 different intra-predictive modes defined in HEVC, such as DC (or mean) mode and non-directional modes such as planar mode, or directional modes, or for example, 67 different intra-predictive modes defined for VVC, such as DC (or mean) mode and non-directional modes such as planar mode, or directional modes.

[0108] The intra-prediction unit 254 is configured to generate an intra-prediction block 265 using reconstructed samples of neighboring blocks of the same current picture, based on one of the intra-prediction modes in a set of intra-prediction modes.

[0109] The intra-prediction unit 254 (or generally the mode selection unit 260) is further configured to output intra-prediction parameters (or generally information indicating a selected intra-prediction mode for a block) to the entropy coding unit 270 in the form of syntax elements 266 for inclusion in the encoded picture data 21, for example, so that the video decoder 30 may receive the prediction parameters and use them for decoding.

[0110] Interpretation A set of (or possible) interpretation modes depends on the available reference picture (i.e., a previously at least partially decoded picture stored in DBP230) and other interpretation parameters, such as whether the entire reference picture is used to search for the best-matching reference block, or only a portion of the reference picture, such as only the search window area around the current block, and / or whether pixel interpolation, such as half / semi-pel and / or quarter-pel interpolation, is applied.

[0111] In addition to the prediction mode described above, skip mode and / or direct mode may also be applicable.

[0112] The interpretation unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (neither of which are shown in Figure 2). The motion estimation unit may be configured to receive or acquire, for motion estimation, picture block 203 (the current picture block 203 of the current picture 17) and decoded picture 231, or at least one or more already reconstructed blocks, for example, one or more other / different reconstructed blocks of already decoded picture 231. For example, a video sequence may include the current picture and already decoded picture 231, or in other words, the current picture and already decoded picture 231 may be part of a sequence of pictures that make up the video sequence or may make up a sequence of such pictures.

[0113] The encoder 20 may be configured, for example, to select a reference block from multiple reference blocks of the same or different pictures among several other pictures, and to provide the motion estimation unit with an offset (spatial offset) between the reference picture (or reference picture index) and / or the position (x, y coordinates) of the reference block and the position of the current block as an interpretation parameter. This offset is also called the motion vector (MV).

[0114] The motion compensation unit is configured to obtain interprediction parameters, for example, receive them, and perform interprediction based on or using the interprediction parameters to obtain interprediction block 265. Motion compensation performed by the motion compensation unit may include fetching or generating prediction blocks based on motion / block vectors determined by motion estimation, possibly performing subpixel-precision interpolation. Interpolation filtering may generate additional pixel samples from known pixel samples, and thus potentially increase the number of candidate prediction blocks that can be used to code picture blocks. Upon receiving a motion vector about the current picture block's PU, the motion compensation unit may find the prediction block pointed to by the motion vector in one of the reference picture lists.

[0115] The motion compensation unit may also generate block and video slice-related syntax elements for use by the video decoder 30 when decoding the picture blocks of the video slice. In addition to slices and their respective syntax elements, or as an alternative to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be generated or used.

[0116] Entropy coding The entropy coding unit 270 is configured to apply, for example, an entropy coding algorithm or scheme (e.g., variable length coding (VLC), context adaptive VLC (CAVLC), arithmetic coding, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique) or bypass (uncompressed) to the quantized coefficients 209, inter-prediction parameters, intra-prediction parameters, loop filter parameters, and / or other syntax elements, for example, to obtain encoded picture data 21 that can be output via output 272 in the form of an encoded bitstream 21, for example, so that the video decoder 30 may receive the parameters and use them for decoding. The encoded bitstream 21 may be sent to the video decoder 30 or stored in memory for later transmission or retrieval by the video decoder 30.

[0117] Other variations of the video encoder 20 and other structures may be used to encode a video stream. For example, a non-transformation encoder 20 may directly quantize the residual signal with respect to a particular block or frame without a transformation processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and a dequantization unit 210 combined into a single unit.

[0118] Decoder and decoding method Figure 3 shows an example of a video decoder 30 configured to implement the technology of the present application. The video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21) encoded by, for example, the encoder 20, in order to obtain a decoded picture 331. The encoded picture data or bitstream contains information for decoding the encoded picture data, for example, the picture blocks of the encoded video slice (and / or tile group or tile) and the associated syntax elements.

[0119] In the example in Figure 3, the decoder 30 includes an entropy decoding unit 304, a dequantization unit 310, an inverse transformation processing unit 312, a reconstruction unit 314 (e.g., an aggregater 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an interpretation unit 344, and an intraprediction unit 354. The interpretation unit 344 may be a motion compensation unit or may include a motion compensation unit. In some examples, the video decoder 30 may perform a decoding path that is generally the reverse of the encoding path described in relation to the video encoder 100 in Figure 2.

[0120] As described in relation to encoder 20, the dequantization unit 210, inverse processing unit 212, reconstruction unit 214, loop filter 220, decoding picture buffer (DPB) 230, inter-prediction unit 344, and intra-prediction unit 354 can also be considered to form the “built-in decoder” of video encoder 20. Thus, dequantization unit 310 may be functionally identical to dequantization unit 110, inverse processing unit 312 may be functionally identical to inverse processing unit 212, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoding picture buffer 330 may be functionally identical to decoding picture buffer 230. Therefore, the descriptions given for each unit and function of video encoder 20 apply mutatis mutandis to each unit and function of video decoder 30.

[0121] Entropy decoding The entropy decoding unit 304 is configured to analyze the bitstream 21 (or generally the encoded picture data 21) and, for example, perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and / or decoded coding parameters (not shown in Figure 3), such as inter-prediction parameters (e.g., reference picture index and motion vector), intra-prediction parameters (e.g., intra-prediction mode or index), transformation parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the encoding scheme described in relation to the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may be further configured to provide the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit 360 and other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or video block level. In addition to slices and their respective syntax elements, or as an alternative to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received and / or used.

[0122] Dequantization The dequantization unit 310 may be configured to receive quantization parameters (QP) (or generally information related to dequantization) and quantized coefficients from the encoded picture data 21 (for example, by the entropy decoding unit 304, for example, by parsing and / or decoding), and to apply dequantization to the decoded quantized coefficients 309 based on the quantization parameters to obtain dequantized coefficients 311, which may also be called transformed coefficients 311. The dequantization process may include using the quantization parameters determined by the video encoder 20 for each video block in the video slice (or tile or tile group) to determine the degree of quantization and, similarly, the degree of dequantization to be applied.

[0123] Inverse Transform The inverse transformation processing unit 312 may be configured to receive the dequantized coefficients 311, also called the transformation coefficients 311, and to apply a transformation to the dequantized coefficients 311 in order to obtain the reconstructed residual block 213 in the sample region. The reconstructed residual block 213 may also be called the transformation block 213. The transformation may be an inverse transformation, such as an inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transformation processing unit 312 may be further configured to receive transformation parameters or corresponding information from the encoded picture data 21 (for example, by parsing and / or decoding by the entropy decoding unit 304) in order to determine the transformation to be applied to the dequantized coefficients 311.

[0124] Rebuild The reconstruction unit 314 (for example, an adder or summer 314) may be configured to obtain the reconstructed block 315 in the sample region by adding the reconstructed residual block 313 to the predicted block 365, for example, by adding the sample values ​​of the reconstructed residual block 313 to the sample values ​​of the predicted block 365.

[0125] filtering The loop filter unit 320 (either within or after the coding loop) is configured to filter the reconstructed block 315 to smooth pixel transitions or otherwise improve video quality, for example, to obtain the filtered block 321. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening, smoothing filter, or a co-filter, or any combination thereof. The loop filter unit 320 is shown in Figure 3 as an in-loop filter, but in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0126] Decode picture buffer The decoded video block 321 of the picture is then stored in a decoded picture buffer 330, which stores the decoded picture 331 for use as a reference picture for subsequent motion compensation for other pictures and / or for output on the display, respectively.

[0127] The decoder 30 is configured to output the decoded picture 311, for example, via output 312, for presentation or viewing to the user.

[0128] prediction The inter-prediction unit 344 may be identical to the inter-prediction unit 244 (particularly the motion compensation unit), and the intra-prediction unit 354 may be functionally identical to the inter-prediction unit 254, and perform partitioning or partitioning decisions and predictions based on partitioning and / or prediction parameters or their respective information received from the decoded picture data 21 (for example, by analysis and / or decoding by the entropy decoding unit 304). The mode application unit 360 may be configured to perform block-by-block predictions (intra or inter-predictions) based on the reconstructed picture, block, or their respective samples (filtered or unfiltered) in order to obtain prediction blocks 365.

[0129] When a video slice is coded as an intra-coded (I) slice, the intra-prediction unit 354 of the mode application unit 360 is configured to generate a prediction block 365 for the picture block of the current video slice based on the signaled intra-prediction mode and data from already decoded blocks of the current picture. When a video picture is coded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., motion compensation unit) of the mode application unit 360 is configured to generate a prediction block 365 for the video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 304. With respect to inter-prediction, the prediction block may be generated from one of the reference pictures in one of the reference picture lists. The video decoder 30 may construct reference frame lists, List 0 and List 1, using default construction techniques based on the reference pictures stored in the DPB 330. The same or similar may apply to or by embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or as a substitute for slices (e.g., video slices), for example, video may be coded using I, P, or B tile groups and / or tiles.

[0130] The mode-applying unit 360 is configured to determine predictive information about the video block of the current video slice by analyzing motion vectors or related information and other syntax elements, and uses the predictive information to generate a predictive block about the current video block being decoded. For example, the mode-applying unit 360 uses some of the received syntax elements to determine the predictive mode used to code the video block of the video slice (e.g., intra or inter predictive), the slice type of inter predictive (e.g., B slice, P slice, or GPB slice), construction information about one or more of the reference picture lists for the slice, motion vectors for each intercoded video block of the slice, the status of the inter predictive for each intercoded video block of the slice, and other information for decoding the video block in the current video slice. The same or similar may apply for or by embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or as an alternative to slices (e.g., video slices), for example, video may be coded using I, P, or B tile groups and / or tiles.

[0131] Embodiments of the video decoder 30 shown in Figure 3 may be configured to partition and / or decode a picture by using slices (also called video slices), the picture may be partitioned into one or more (generally non-overlapping) slices or decoded using one or more (generally non-overlapping) slices, each slice may contain one or more blocks (e.g., CTUs).

[0132] Embodiments of the video decoder 30 shown in Figure 3 may be configured to partition and / or decode a picture by using tile groups (also called video tile groups) and / or tiles (also called video tiles), the picture may be partitioned into one or more (generally non-overlapping) tile groups or decoded using one or more (generally non-overlapping) tile groups, each tile group may, for example, contain one or more blocks (e.g., CTUs) or one or more tiles, each tile may, for example, be rectangular in shape and may contain one or more blocks (e.g., CTUs), for example, complete or fragmented blocks.

[0133] Other variations of the video decoder 30 may be used to decode the encoded picture data 21. For example, the decoder 30 may generate an output video stream without a loop filtering unit 320. For example, a non-transformation-based decoder 30 may directly dequantize the residual signal with respect to a particular block or frame without an inverse transformation unit 312. In another implementation, the video decoder 30 may have a dequantization unit 310 and an inverse transformation unit 312 combined into a single unit.

[0134] It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations such as clipping or shifting may be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering.

[0135] Note that further calculations may be applied to the derived motion vector of the current block (including, but not limited to, affine mode control point motion vectors, affine, planar, and ATMVP mode lower block motion vectors, and temporal motion vectors). For example, the value of a motion vector is constrained to a given range according to its representation bits. If the representation bits of a motion vector are bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" means exponentiation. For example, if bitDepth is set to be equal to 16, the range is -32768 to 32767, and if bitDepth is set to be equal to 18, the range is -131072 to 131071. For example, the values ​​of the derived motion vectors (e.g., the MVs of four 4x4 subblocks within one 8x8 block) are constrained such that the maximum difference between the integer parts of the MVs of the four 4x4 subblocks is less than or equal to N pixels, such as less than or equal to 1 pixel. Here, we provide two methods for constraining motion vectors according to bitDepth.

[0136] Method 1: Remove the MSB (most significant bit) by overflowing the operation. ux = ( mvx + 2 bitDepth ) % 2 bitDepth (1) mvx = ( ux >= 2 bitDepth-1 ) ? ( ux - 2 bitDepth ) : ux (2) uy = ( mvy + 2 bitDepth ) % 2 bitDepth (3) mvy = ( uy >= 2 bitDepth-1 ) ? ( uy - 2 bitDepth ) : uy (4) In the formula, mvx is the horizontal component of the motion vector of the image block or subblock, mvy is the vertical component of the motion vector of the image block or subblock, and ux and uy represent the intermediate values.

[0137] For example, when the value of mvx is -32769, after applying equations (1) and (2), the resulting value is 32767. In a computer system, a decimal number is stored as a two's complement. The two's complement of -32769 is 1,0111,1111,1111,1111 (17 bits), and at that time, the MSB is discarded, so the resulting two's complement is 0111,1111,1111,1111 (decimal 32767), which is the same as the output by applying equations (1) and (2). ux = (mvpx + mvdx + 2 bitDepth ) % 2 bitDepth (5) mvx = (ux >= 2 bitDepth-1 )? (ux - 2 bitDepth ) : ux (6) uy = (mvpy + mvdy + 2 bitDepth ) % 2 bitDepth (7) mvy = (uy >= 2 bitDepth-1 )? (uy - 2 bitDepth ) : uy (8)

[0138] The operation may be applied in the sum of mvp and mvd as shown in equations (5) to (8).

[0139] Method 2: Remove the overflow MSB by clipping the value vx = Clip3(-2 bitDepth-1 , 2 bitDepth-1 -1, vx) vy = Clip3(-2 bitDepth-1 , 2 bitDepth-1 -1, vy) Where vx is the horizontal component of the motion vector of the image block or sub-block, vy is the vertical component of the motion vector of the image block or sub-block, x, y, and z respectively correspond to the three input values of the MV clipping process, and the definition of the function Clip3 is as follows.

[0140]

number

[0141] Figure 4 is a schematic diagram of a video coding device 400 according to an embodiment of the present disclosure. The video coding device 400 is suitable for implementing embodiments disclosed as described herein. In embodiments, the video coding device 400 may be a decoder, such as the video decoder 30 in Figure 1A, or an encoder, such as the video encoder 20 in Figure 1A.

[0142] The video coding device 400 includes an incoming port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing data, a transmitter unit (Tx) 440 and an outgoing port 450 (or output port 450) for transmitting data, and memory 460 for storing data. The video coding device 400 may also include optical-electrical (OE) and electrical-optical (EO) components coupled to the incoming port 410, receiver unit 420, transmitter unit 440, and outgoing port 450 for transmitting or receiving optical or electrical signals.

[0143] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 communicates with the incoming port 410, the receiver unit 420, the transmitter unit 440, the outgoing port 450, and the memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the embodiments disclosed above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, including the coding module 470 greatly improves the functionality of the video coding device 400 and results in the transition of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0144] Memory 460 may include one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device to store such programs when selected to run, as well as instructions and data read during program execution. Memory 460 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0145] Figure 5 is a simplified block diagram of a device 500 that may be used as either or both of the source device 12 and destination device 14 in Figure 1, according to an exemplary embodiment.

[0146] The processor 502 of the device 500 can be a central processing unit. Alternatively, the processor 502 can be one or more devices of any other type, existing or to be developed, capable of manipulating or processing information. The disclosed implementation can be carried out by a single processor, for example, processor 502, as shown, but speed and efficiency advantages can be realized by using two or more processors.

[0147] The memory 504 of the device 500 can be, in implementation, a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by the processor 502 using the bus 512. Memory 504 may further include an operating system 508 and an application program 510, the application program 510 including at least one program that enables the processor 502 to perform the methods described herein. For example, the application program 510 may include applications 1 to N, further including a video coding application that performs the methods described herein.

[0148] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch display that combines the display with a touch-sensing element that is operable to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0149] Although shown here as a single bus, the bus 212 of device 500 may consist of multiple buses. Furthermore, the secondary storage 514 can be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 can be implemented in a wide variety of configurations.

[0150] Motion vector improvement (MVR) Typically, motion vectors are determined at least partially on the encoder side and signaled to the decoder within the encoded bitstream. However, motion vectors can also be improved in the decoder (and in the encoder) starting from an initial motion vector indicated in the bitstream. In such cases, for example, the similarity between already decoded pixel patches pointed to by the initial motion vector can be used to improve the accuracy of the initial motion vector. Such motion improvement has the advantage of reducing signaling overhead, meaning that the accuracy of the initial motion is improved in the same way at both the encoder and the decoder, and therefore additional signaling for improvement is not required.

[0151] It should be noted that the initial motion vector before improvement may not be the best motion vector that yields the best prediction. Since the initial motion vector is signaled within the bitstream, it may be impossible to represent the initial motion vector with very high accuracy (high bitrate), and therefore, the motion vector improvement process is used to make the initial motion vector better. The initial motion vector may be, for example, the motion vector used in predicting the neighboring blocks of the current block. In this case, it is sufficient to signal within the bitstream an indication that shows which neighboring block's motion vector will be used by the current block. Such a prediction mechanism is very effective in reducing the number of bits required to represent the initial motion vector. However, generally speaking, the accuracy of the initial motion vector may be low because the motion vectors of two neighboring blocks are not predicted to be identical.

[0152] To further improve the accuracy of motion vectors without further increasing signaling overhead, it may be beneficial to further refine the motion vectors derived at the encoder side and provided (signaled) within the bitstream. Motion vector improvements may be performed at the decoder without assistance from the encoder. The encoder may use the same improvements within its decoder loop to obtain the corresponding improved motion vectors available at the decoder. Improvements to the reconstructed current block in the current picture are performed by determining a template for the reconstructed sample, determining a search space around the initial motion information for the current block, and finding the portion of the reference picture that best matches the template within the search space. The best-matching portion determines the improved motion vector for the current block, and that improved motion vector is used to obtain the inter-predicted sample for the current block, i.e., the current block being reconstructed.

[0153] The improvement in motion vectors is seen in parts of the interpretation unit (244) in Figure 2 and 344 in Figure 3.

[0154] The motion vector can be improved by following these steps:

[0155] In general, the initial motion vector may be determined based on indications in the bitstream. For example, an index indicating the position in a list of candidate motion vectors may be signaled in the bitstream. In another example, the motion vector predictor index and the difference value of the motion vectors may be signaled in the bitstream. A motion vector determined based on indications in the bitstream is defined as the initial motion vector. In the case of bi-prediction, where the inter-prediction for the current block is obtained as a weighted combination of predicted blocks of samples determined by two motion vectors, the initial motion vector in the first reference picture of list L0 is denoted as MV0, and the initial motion vector in the second reference picture of list L1 is denoted as MV1.

[0156] The initial motion vectors are used to determine pairs of candidate motion vectors (MVs) for improvement. At least two pairs of candidate improvement vectors must be determined. Generally, pairs of candidate improvement motion vectors are determined based on the initial motion vector pair (MV0, MV1). Furthermore, pairs of candidate MVs are determined by adding small motion vector differences to MV0 and MV1. For example, pairs of candidate MVs may include: (MV0, MV1) (MV0 + (0,1), MV1 + (0,-1)) (MV0 + (1,0), MV1 + (-1,0)) (MV0 + (0,-1), MV1 + (0,1)) (MV0 + (-1,0), MV1 + (1,0)) ·... Here, (1,-1) represents a vector with a displacement of 1 in the horizontal (or x) direction and a displacement of -1 in the vertical (or y) direction.

[0157] It should be noted that the list above of candidate pairs is merely an example for illustrative purposes, and the present invention is not limited to any particular list of candidates.

[0158] The pairs of candidate motion vectors (MVs) for improvement form the search space for the motion vector improvement process.

[0159] In the current bi-prediction of blocks, the two prediction blocks obtained using the first motion vectors from list L0 and the second motion vectors from list L1 are combined into a single prediction signal, which can result in a better adaptation of the original signal than uni-prediction, resulting in less residual information and possibly more efficient compression.

[0160] In motion vector improvement, two prediction blocks obtained using the first and second motion vectors of each candidate MV pair are compared based on a similarity metric for each improved candidate MV pair. Typically, the candidate MV pair that yields the highest similarity is selected as the improved motion vector. The improved motion vector in the first reference picture of list L0 and the improved motion vector in the second reference picture of list L1 are denoted as MV0' and MV1', respectively. In other words, predictions corresponding to the motion vectors of list L0 and list L1 of the candidate motion vector pair are obtained, and then these predictions are compared based on a similarity metric. The candidate motion vector pair with the highest relevant similarity is selected as the improved MV pair.

[0161] Generally, the output of the improvement process is the improved MV. The improved MV may be the same as or different from the initial MV, depending on which pair of candidate MVs achieves the highest similarity, and the pair of candidate MVs formed by the initial MV is also among the candidate pairs of MVs. In other words, if the pair of candidate MVs that achieves the highest similarity is formed by the initial MV, then the improved MV and the initial MV are equal to each other.

[0162] Instead of choosing a position that maximizes the similarity metric, another approach is to choose a position that minimizes the difference metric. The measure of difference comparison could be SAD (Sum of Absolute Differences), MRSAD (Mean Removed Sum of Absolute Differences), SSE (Sum of Squared Residuals), etc. The SAD between two prediction blocks could be obtained using a pair of candidate MVs (CMV0, CMV1), and the SAD could be calculated as follows:

[0163]

number

[0164] In the formula, nCbH and nCbW are the height and width of the predicted block, the function abs(a) specifies the absolute value of argument a, and predSAmplesL0 and predSAmplesL1 are predicted block samples obtained by pairs of candidate MVs represented by (CMV0, CMV1).

[0165] Alternatively, a measure of difference comparison can be obtained by evaluating only a subset of samples within the prediction block to reduce the number of calculations. An example is shown below, where the sample rows are alternatively included in the SAD calculation (every other row is evaluated).

[0166]

number

[0167] An example of motion vector refinement is described in the JVET document JVET-M1001-v3, "Versatile Video Coding (Draft 4)," published at http: / / phenix.it-sudparis.eu / jvet / (ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11). Section 8.4.3 of the document, "Decoder side motion vector refinement process," illustrates motion vector refinement.

[0168] To reduce the internal memory requirements for improvement, in some embodiments, the motion vector improvement process may be performed independently on blocks of a luma sample obtained by partitioning coded blocks of samples exceeding a specific predetermined width or height within the luma sample into subblocks of samples that are less than or equal to the predetermined width and height of the luma. The improved MV pairs for each subblock within the partitioned coded blocks can be different. Interpretation for both luma and chroma is then performed for each subblock using the improved MV pairs for that subblock.

[0169] Each MV in an initial MV pair can have fractional pixel precision. In other words, an MV represents the displacement between the current block of samples and the resampled reference region, and this displacement can point to a fractional position horizontally and vertically from the integer grid of the reconstructed reference samples. Generally, two-dimensional interpolation of the reconstructed reference integer sample grid values ​​is performed to obtain sample values ​​at fractional sample offset positions. The process of obtaining predicted samples from the reconstructed reference picture using candidate MV pairs can be carried out by one of the following methods: The fractional part of the initial MV pair is rounded to the nearest integer position to obtain the integer grid value of the reconstructed reference picture. • Perform a 2-tap (e.g., bilinear) separable bilinear interpolation to obtain the predicted sample value with fractional pixel accuracy indicated by the initial MV pair. • Perform separable interpolation of more taps (e.g., 8 or 6 taps) to obtain the predicted sample value with fractional pixel accuracy indicated by the initial MV pair.

[0170] Candidate MV pairs can have any subpixel offset relative to the initial MV pair, but in some embodiments, for the sake of simplicity of search, candidate MV pairs that are integer pixel distances from the initial MV pair are selected. In such cases, the predicted samples of all candidate MV pairs can be obtained by making predictions about blocks of samples around the initial MV pair such that the predicted samples of all candidate MV pairs encompass all improvement positions around the initial MV pair.

[0171] In some embodiments, once the cost values ​​of the differences in all candidate pairs of MVs at integer distances from the initial pair of MVs are evaluated, additional pairs of candidate MVs at subpixel distance offsets from the best-cost position are added. Predicted samples are obtained for each of these positions using one of the methods described above, and the cost of the differences is evaluated and compared to obtain the position with the lowest difference. In certain other embodiments, to avoid this computationally expensive prediction process for each subpixel distance position around the best-cost integer-distance position, the evaluated integer-distance cost values ​​are memorized, and a parametric error surface is fitted to the neighborhood of the integer-distance position. The minimum value of this error surface is then analytically calculated and used as the position with the smallest difference. In such cases, the cost value of the difference is said to be derived from the calculated integer-distance cost values.

[0172] The application of motion vector improvements for a given coded block of a sample may be conditioned by specific coding properties of the coded block of the sample. Some examples of such coding properties may be: The distance, expressed in terms of the number of pictures (when sampled at a uniform frame rate), from the current picture to the two reference pictures used for the biprediction of the coded block of the sample, decreases equally on both sides of the current picture. • The initial difference between two predicted blocks obtained using the initial MV pair is below a predetermined per-sample threshold.

[0173] Bi-predictive optical flow improvement (BPOF) The improvement of the optical flow for biprediction is a process that increases the accuracy of biprediction for a block without any apparent additional signaling within the bitstream other than that signaled for biprediction. The improvement of the optical flow for biprediction is part of the interprediction unit (244) in Figure 2 and part of 344 in Figure 3.

[0174] In biprediction, two interpredictions are obtained according to two motion vectors, and then the predictions are combined by applying a weighted average. The combined prediction can yield reduced residual energy because the quantization noise in the two reference patches cancels out, thereby providing higher coding efficiency than single prediction. The weighted combination of bipredictions is given by the formula Bi-prediction = Prediction1 * W1 + Prediction2 * W2 + K This can be performed by, where W1 and W2 are weight coefficients that may be signaled in the bitstream or predefined on the encoder or decoder side. K is an additive factor that may also be signaled in the bitstream or predefined on the encoder or decoder side. For example, biprediction is, Bi-prediction = (Prediction1 + Prediction2) / 2 It can be obtained using the formula, where W1 and W2 are set to 1 / 2 and K is set to 0.

[0175] The goal of optical flow improvement is to increase the accuracy of biprediction. Optical flow is the visible pattern of motion of an image object between two consecutive frames. Optical flow is caused by the movement of the object or camera. The optical flow improvement process increases the accuracy of biprediction by applying the optical flow equation (solving the optical flow equation).

[0176] In the example, pixel I(x,y,t) is in the first frame (x and y correspond to spatial coordinates, and t corresponds to the time dimension). The object represented by the pixel moves a distance (dx,dy) in the next frame, which is acquired after time dt. Since those pixels are the same and their intensity does not change, the optical flow equation is: I(x,y,t) = I(x+dx, y+dy, t+dt) It is given by.

[0177] I(x,y,t) specifies the intensity (sample value) of the pixel at coordinates (x,y,t).

[0178] In another example, small displacements and higher-order terms in the Taylor series expansion are ignored, and the optical flow equation becomes

[0179]

number

[0180] It can also be written as follows, in the formula,

[0181]

number

[0182] This is the spatial gradient of the sample in the horizontal and vertical directions at position (x,y),

[0183]

number

[0184] This is the time-partial derivative at (x,y).

[0185] Improvements to optical flow utilize the above principles to enhance the quality of biprediction.

[0186] Implementing improvements to optical flow generally involves the following steps: 1. Calculate the gradient of the sample. 2. Calculate the difference between the first prediction and the second prediction. 3. Error Δ between two reference patches obtained using the optical flow formula.

[0187]

number

[0188] Calculate the displacement of the pixel or group of pixels that minimizes I (0) This corresponds to the sample values ​​of the first prediction, I (1) This is the sample value for the second prediction, ∂I (0) / ∂x and ∂I (0) / ∂y is the gradient in the -x and -y directions, τ1 and τ0 represent the distance to the reference picture, and the first and second predictions are obtained. Motion vector (v x , v y The result is obtained through a minimization process. Some methods minimize the sum of squared residuals, while others minimize the sum of absolute errors. 4. Use the following implementation of the optical flow formula. Nod BIO = 1 / 2·(I (0) + I (1) + v x / 2·(τ1∂I (1) / ∂x - τ0∂I (0) / ∂x) + v y / 2·(τ1∂I (1) / ∂y - τ0∂I (0) / ∂y)) In the formula, pred BIO This specifies the corrected prediction, which is the output of the optical flow improvement process.

[0189] The gradient of the sample can be obtained by the following formula. ·∂I(x, y, t) / ∂x = I(x + 1, y, t) - I(x - 1, y, t) ·∂I(x, y, t) / ∂y = I(x, y + 1, t) - I(x, y - 1, t)

[0190] In some embodiments, to reduce the complexity of estimating displacement for each pixel, the displacement is estimated with respect to groups of pixels. In some examples, to compute an improved biprediction for a block of 4x4 luma samples, the displacement is estimated using the sample values ​​of an 8x8 luma sample block with the 4x4 block of the sample as its center.

[0191] The input to the optical flow improvement process is a prediction sample from two reference pictures, and the output of the optical flow improvement is a combined prediction (predBIO) calculated by the optical flow formula.

[0192] An example of optical flow improvement is described in section 8.4.7.4, "Bidirectional optical flow prediction process," of document JVET-M1001, Versatile Video Coding (Draft 4).

[0193] The terms optical flow improvement, bipredictive optical flow improvement, and bidirectional optical flow improvement are used interchangeably in this disclosure as the terms are essentially equivalent.

[0194] In the example, the motion vector improvement and the optical flow improvement are applied sequentially as follows: Step 0: Obtain the initial motion vector as shown in Figure 8, 1010. Step 1: Motion vector improvement is applied, and the improved motion vector 1020 is obtained. Step 2, the prediction is improved by the motion vector obtained from 1040. The obtained prediction is I (0) and I (1) These are the inputs to the optical flow improvement process. Step 3, the optical flow improvement process is applied to the prediction to obtain the corrected prediction. The corrected prediction is obtained by the optical flow formula, and pred BIO It is written as follows.

[0195] However, the optical flow improvement process is computationally intensive. Decoding time increases with the application of optical flow improvements.

[0196] In one embodiment of the present invention, a method for determining whether or not to apply optical flow improvements is not disclosed, and this determination may be made by calculations performed during the motion vector improvement process.

[0197] More specifically, the results of calculations performed during the motion vector improvement process are used to determine whether or not optical flow improvements should be applied.

[0198] The object of the present invention is to skip the application of optical flow improvements according to specified conditions so that the average decoding time is reduced (by skipping the necessary calculations).

[0199] According to the first exemplary embodiment, the following steps are applied to obtain a prediction about the current coding block. Step 0: Obtain the initial motion vector based on the indication information in the bitstream. Step 1: Obtain a first prediction based on the initial motion vector and the M-tap interpolation filter. Step 2: Obtain the matching cost based on the first prediction. Step 3: Obtain the improved motion vector according to the initial motion vector and matching cost. Step 4: Obtain a second prediction using the improved motion vector and K-tap interpolation filter. Step 5: Determine whether to run the optical flow improvement process based on the matching cost. In the example, the matching cost is compared to a threshold, and the optical flow improvement process is run when the matching cost value is greater than or equal to the threshold. Step 5 may also be run before Step 3 or Step 4. Step 6: When it is determined that the optical flow improvement process needs to be performed, the optical flow improvement is applied with the second prediction as input and the modified second prediction as output. If it is determined negatively, the optical flow improvement is not applied to the second prediction. In other words, when it is determined that the optical flow improvement process needs to be performed, the final prediction for the current coding block is obtained by the second prediction and the optical flow improvement process. Otherwise, the final prediction for the current coding block is obtained by the second prediction without the application of the optical flow improvement process.

[0200] A detailed explanation of the steps is as follows:

[0201] In step 0, two initial motion vectors are obtained as input. The initial motion vectors may be determined based on indication information in the bitstream. For example, an index may be signaled in the bitstream, and the index indicates the position in the list of candidate motion vectors. In another example, the difference between the motion vector predictor index and the motion vector may be signaled in the bitstream. The motion vector determined based on the indication information in the bitstream is defined as the initial motion vector.

[0202] In another example, the indication of a reference picture can be obtained from the bitstream, and the initial motion vector is obtained based on the indication of the reference picture. The indication of the reference picture is used to determine the reference picture pointed to by the initial motion vector.

[0203] Steps 1, 2, and 3 correspond to the motion vector improvement process, as explained in the example above. The initial motion vector is improved through motion vector improvement. In one example, the matching cost is a measure of similarity used in motion vector improvement.

[0204] Step 1 yields a first prediction corresponding to the initial motion vector. In the example, in the motion vector improvement process, there are at least two pairs of candidate motion vectors, one of which is typically the pair (MV0, MV1) formed by the initial motion vector. In other words, the set of candidate motion vectors typically contains two or more pairs, one of which is typically (MV0, MV1). The other pair of candidate motion vectors is determined based on (MV0, MV1) by adding small disturbances to the motion vectors (as described in the example above).

[0205] In step 1, a first prediction corresponding to each pair of candidate motion vectors is obtained based on an M-tap interpolation filter. For example, one prediction corresponding to MV0 can be obtained by identifying the location of a rectangular block in a reference picture (a picture already encoded in the encoder or decoded in the decoder), the block being pointed to by MV0. The interpolation filter is then advantageously applied to the samples within the block pointed to by MV0. To provide a more accurate motion estimate, the resolution of the reference picture may be increased by interpolating samples between pixels. Fractional pixel interpolation can be performed by a weighted average of the nearest pixels. Here, the M-tap filter can generally be (but not limited to) a 2, 4, 6, or 8-tap filter, meaning the filter has M multiplication coefficients. A prediction corresponding to MV1 can similarly be obtained by identifying the location of a rectangular block in the same or a different reference picture. The size of the rectangle is proportional to the size of the currently coded block.

[0206] In step 2, the matching cost associated with each pair of candidate motion vectors is determined according to the first prediction.

[0207] Step 2 yields at least one matching cost (e.g., a measure of similarity) corresponding to one of the pairs of candidate motion vectors (MVs) for improvement. The higher the similarity between the two prediction blocks, the smaller the matching cost.

[0208] The matching cost is used in the improvement of the initial motion vector in step 3. The improved motion vector is selected according to the matching cost.

[0209] In step 4, the improved motion vector and K-tap interpolation filter yield a second prediction. In the case of two predictions, where there are two improved motion vectors (MV0' and MV1'), two second predictions are obtained.

[0210] The second prediction is obtained by applying a second interpolation filter (K-tap filter) which may or may not be identical to the first interpolation filter (M-tap filter). The second prediction is obtained similarly to the first prediction by applying the second interpolation filter and according to the blocks pointed to by MV0' and MV1' in the reference picture.

[0211] In step 5, the matching cost is used to determine whether or not to perform the optical flow improvement process, based on the following:

[0212] When the matching cost value is below a predefined threshold, optical flow improvement is not applied. When the matching cost value is equal to or greater than the threshold, the optical flow improvement process is executed. If the optical flow improvement process is executed, the final prediction sample is corrected.

[0213] In step 6, if the matching cost is greater than or equal to the threshold according to the output of step 5, the optical flow improvement process is applied to the second prediction, which is obtained by MV0' and MV1' (improved motion vectors). The final prediction for the current coding block is obtained by performing the optical flow improvement process on the second prediction, which is indicated by MV0' and MV1'. If the matching cost is less than the threshold, the final prediction is obtained by the second prediction indicated by MV0' and MV1' without applying optical flow improvement, i.e., step 6 is not performed.

[0214] In one implementation, the matching cost in step 2 is the matching cost corresponding to the pair of initial motion vectors (which are one of the pairs of candidate motion vectors (MVs) for improvement). The matching cost could correspond to the pair MV0, MV1.

[0215] In another implementation, the matching cost in step 2 is equal to the smallest matching cost among the pairs of candidate motion vectors (MVs). In other words, the matching cost corresponding to each pair of candidate motion vectors is obtained, and the matching cost is equal to the smallest matching cost among them. In one example, the pair of improved motion vectors (MV0', MV1') has the smallest matching cost, and therefore that pair of improved motion vectors (MV0', MV1') is selected, so the matching cost is the matching cost corresponding to the improved motion vector pair MV0' and MV1'.

[0216] For example, a pair of MVs can be constructed in the following way:

[0217] Candidate MV pairs are determined by adding the small motion vector difference between MV0 and MV1. For example, candidate MV pairs may include: (MV0, MV1) (MV0 + (0,1), MV1 + (0,-1)) (MV0 + (1,0), MV1 + (-1,0))

[0218] Throughout this application, MV0 and MV1 are initial motion vectors, and MV0' and MV1' are improved motion vectors.

[0219] According to another implementation, when the optical flow improvement process is not performed, the final prediction is obtained by the following formula: Bi-prediction = Prediction1 * W1 + Prediction2 * W2 + K In the formula, W1 and W2 are weight coefficients, which may be signaled within the bitstream, or which may be predefined on the encoder or decoder side. K is an additive factor, which may also be signaled within the bitstream, or which may be predefined on the encoder or decoder side. In the example, the biprediction is, Bi-prediction = (Prediction1 + Prediction2) / 2 This can be obtained using the formula, where W1 and W2 are set to 1 / 2 and K is set to 0. Prediction1 and Prediction2 are second predictions obtained by K-tap interpolation filtering, where Prediction1 corresponds to the first improved MV(MV0') and Prediction2 corresponds to the second improved MV(MV1').

[0220] The above equation realizes a weighted combination of two predictions, and the result is the final prediction for the block.

[0221] The threshold can be a predefined value, and the value of the threshold may depend on the size of the prediction block. For example, the threshold can be thr = nCbW × nCbH × K, where K is a value greater than zero, and nCbW and nCbH are the width and height of the prediction block.

[0222] The first embodiment is further illustrated by the flowchart in Figure 6.

[0223] In one implementation, an M-tap filter is a 2-tap filter (e.g., a bilinear filter) where one of the taps is equal to zero. In this implementation, the M-tap filter uses two multiplier coefficients, and the value of one of the coefficients is always equal to zero. Which coefficient has a value equal to zero is determined based on a fractional sample point, which is pointed to by a motion vector. In this case, depending on the fractional component of the motion vector, the value of either the first multiplier coefficient or the second multiplier coefficient may be zero.

[0224] Such a filter having two taps, one of which is zero, can be illustrated by the table below.

[0225] [Table 1]

[0226] The fractional sample position (p) can be obtained according to the components of the initial or improved motion vector. For example, if the -x component of the motion vector is given by MV0x, the fractional sample position can be obtained as p = MV0x%16, where "%" is the modulo operation. Generally, p = MV0x%K, where K represents the number of fractional sample positions between two sample positions. The interpolation filter illustrated above can also be called a one-tap filter because only one of the filter taps is non-zero at a time.

[0227] In one implementation, the value of K is equal to 8. In other examples, the value of M is less than 8.

[0228] In one implementation, both the value of M and the value of K are equal to 8.

[0229] The initial motion vector, which is the input to the motion vector improvement unit, is obtained at 710. A search space is constructed (740) around the initial motion vector by the motion vector improvement unit. In one example, the search space consists of pairs of candidate motion vectors, where the first motion vector of the pair corresponds to a first reference picture and the second motion vector of the pair corresponds to a second reference picture. A first prediction corresponding to each pair of candidate motion vectors is obtained at step 710 by applying an M-tap interpolation filter. As part of the improvement of the motion vector, a matching cost corresponding to one of the pairs of motion vectors within the search space is calculated (720). The matching cost is used as part of two processes. The first process is the improvement of the motion vector (740) in which the matching cost is used to determine which pair of motion vectors is selected as the improved pair of motion vectors (750). The second process is the determination of whether the improvement of the optical flow (770) is applied. After the improved motion vector is obtained, a second prediction regarding the current block is obtained at (760). If the matching cost is above a threshold, the improvement of the optical flow is applied and the prediction at 760 is modified by 770 to obtain a modified prediction (780). The modified prediction generally has sample values different from the second prediction at step 760.

[0230] In one example, the motion vector improvement process is executed two or more times to further improve the motion vector. In this example, first, the initial motion vector is improved by the motion vector improvement process to obtain a first improved motion vector. Thereafter, the improvement of the motion vector is executed once more, in which case the first improved motion vector is considered as the initial motion vector for the improvement of the second motion vector.

[0231] According to a second exemplary embodiment, the following steps are applied to obtain a prediction regarding the current coding block. Step 0: Obtain an initial motion vector based on the indication information in the bitstream. Step 1: Obtain a first prediction based on the initial motion vector and the M-tap interpolation filter. Step 2: Obtain N matching costs based on the first prediction. Step 3: Obtain an improved motion vector according to the initial motion vector and the N matching costs based on the first function. Step 4: Obtain a second prediction by the improved motion vector and the K-tap interpolation filter. Step 5: Determine whether to execute the optical flow improvement process according to the N matching costs. The derived cost is obtained by the N matching costs and the second function. In an example, the derived cost is compared with a threshold, and the optical flow improvement process is executed when the value of the derived cost is greater than or equal to the threshold. Step 5 may also be executed before Step 3 or Step 4. Step 6: When it is determined that the optical flow improvement process needs to be executed, correct at least one sample of the prediction of the current coding block by applying the improvement of the optical flow.

[0232] When it is determined that the optical flow improvement process needs to be executed, the improvement of the optical flow is applied with the second prediction as the input and the corrected second prediction as the output. If it is determined negatively, the improvement of the optical flow is not applied to the second prediction. In other words, when it is determined that the optical flow improvement process needs to be executed, the final prediction of the current coding block is obtained by the second prediction and the optical flow improvement process. Otherwise, the final prediction of the current coding block is obtained by the second prediction without applying the optical flow improvement process.

[0233] The detailed description of the steps is as follows.

[0234] In step 0, two initial motion vectors are obtained as input. The initial motion vectors may be determined based on indication information in the bitstream. For example, an index may be signaled in the bitstream, and the index indicates the position in the list of candidate motion vectors. In another example, the difference between the motion vector predictor index and the motion vector may be signaled in the bitstream. The motion vector determined based on the indication information in the bitstream is defined as the initial motion vector.

[0235] In another example, the indication of a reference picture can be obtained from the bitstream, and the initial motion vector is obtained based on the indication of the reference picture. The indication of the reference picture is used to determine the reference picture pointed to by the initial motion vector.

[0236] Steps 1, 2, and 3 correspond to the motion vector improvement process, as explained in the example above. The initial motion vector is improved through motion vector improvement. In one example, the matching cost is a measure of similarity used in motion vector improvement.

[0237] Step 1 yields a first prediction corresponding to the initial motion vector. In the example, in the motion vector improvement process, there are at least two pairs of candidate motion vectors, one of which is typically the pair (MV0, MV1) formed by the initial motion vectors. The other pair of candidate motion vectors is determined based on (MV0, MV1) by adding small disturbances to the motion vectors (as described in the example above).

[0238] In step 1, a first prediction corresponding to each pair of candidate motion vectors is obtained based on an M-tap interpolation filter.

[0239] In step 2, N matching costs associated with N pairs of candidate motion vectors are determined according to the first prediction.

[0240] Step 2 yields N matching costs (a measure of similarity) corresponding to N pairs of candidate motion vectors (MVs). The higher the similarity between two prediction blocks, the smaller the matching cost.

[0241] The aforementioned N matching costs are used in improving the initial motion vector in step 3.

[0242] The improved motion vector is determined by the first function and N matching costs.

[0243] In one example, the improved motion vector can be obtained using the following function. - If (sad

[0003] + sad[5]) is equal to (sad

[0004] << 1), then dmvOffset

[0000] is set to be equal to 0. - Otherwise, the following applies: dmvOffset

[0000] = ( ( sad

[0003] - sad

[0005] ) << 3 ) / ( sad

[0003] + sad

[0005] - ( sad

[0004] << 1 ) ) - If (sad

[0001] + sad[7]) is equal to (sad

[0004] << 1), then dmvOffset

[0001] is set to be equal to 0. - Otherwise, the following applies: dmvOffset

[0001] = ( ( sad

[0001] - sad

[0007] ) << 3 ) / ( sad

[0001] + sad

[0007] - ( sad

[0004] << 1 ) ) In the formula, dmvOffset[0] and dmvOffset[1] specify the difference between the initial motion vector and the improved motion vector. In the example, dmvOffset[0] and dmvOffset[1] specify the -x and -y components of the difference between the improved motion vector and the initial motion vector. sad[0] through sad[7] are the N matching costs corresponding to the N pairs of candidate motion vectors. The improved motion vector is obtained by adding dmvOffset to the initial motion vector.

[0244] Other functions may exist that can be used to determine an improved motion vector according to N matching costs. The first function of the present invention is not limited to the above formula.

[0245] In step 4, the improved motion vector and K-tap interpolation filter yield a second prediction. In the case of two predictions, where there are two improved motion vectors (MV0' and MV1'), two second predictions are obtained.

[0246] The second prediction is obtained by applying a second interpolation filter (K-tap filter) which may or may not be identical to the first interpolation filter (M-tap filter). The second prediction is obtained similarly to the first prediction by applying the second interpolation filter and according to the blocks pointed to by MV0' and MV1' in the reference picture.

[0247] In step 5, the derived cost is obtained by the second function and the N matching costs. The derived cost is used to determine whether or not to perform the optical flow improvement process. If the value of the derived cost is less than a predefined threshold, the optical flow improvement process is not applied. If the value of the derived cost is greater than or equal to the threshold, the optical flow improvement process is performed. If the optical flow improvement process is performed, the final prediction sample is corrected.

[0248] In step 6, according to the output of step 5, if the derived cost is greater than the threshold, the optical flow improvement process is applied to the second prediction, and the second prediction is obtained by MV0' and MV1' (improved motion vectors). The final prediction for the current coding block is obtained by performing an optical flow improvement process on the second prediction, and the second prediction is indicated by MV0' and MV1'. If the matching cost is less than the threshold, the final prediction is obtained by the second prediction indicated by MV0' and MV1' without applying the improvement of the optical flow, that is, step 6 is not executed.

[0249] According to another implementation, when the optical flow improvement process is not executed, the final prediction is obtained by the following formula. Bi-prediction = Prediction1 * W1 + Prediction2 * W2 + K Where W1 and W2 are weight coefficients, W1 and W2 may be signaled in the bitstream or may be predefined on the encoder side or the decoder side. K is also an additive factor that may be signaled in the bitstream or may be predefined on the encoder side or the decoder side. In the example, the bi-prediction is Bi-prediction = (Prediction1 + Prediction2) / 2 is used, where W1 and W2 are set to 1 / 2 and K is set to 0. Prediction1 and Prediction2 are the second predictions obtained by K-tap interpolation filtering, Prediction1 corresponds to the first improved MV (MV0'), and Prediction2 corresponds to the second improved MV (MV1').

[0250] The above formula realizes a weighted combination of two predictions, and the result is the final prediction for the block.

[0251] The threshold can be a predefined value, and the value of the threshold depends on the size of the prediction block. For example, the threshold can be thr = nCbW × nCbH × K, where K is a value greater than zero, and nCbW and nCbH are the width and height of the prediction block.

[0252] A second embodiment is further illustrated by the flowchart in Figure 7.

[0253] In one implementation, an M-tap filter is a two-tap filter (e.g., a bilinear filter) where one of the taps is equal to zero. In this implementation, the M-tap filter uses two multiplier coefficients, one of which is always equal to zero. The coefficient that is equal to zero is determined based on a fractional sample point indicated by the motion vector. In this case, depending on the fractional component of the motion vector, the value of either the first or second multiplier coefficient can be zero.

[0254] Such a filter having two taps, one of which is zero, can be illustrated by the table below.

[0255] [Table 2]

[0256] The fractional sample position (p) can be obtained according to the components of the initial or improved motion vector. For example, if the -x component of the motion vector is given by MV0x, the fractional sample position can be obtained as p = MV0x%16, where "%" is the modulo operation. Generally, p = MV0x%K, where K represents the number of fractional sample positions between two sample positions. The interpolation filter illustrated above can also be called a one-tap filter because only one of the filter taps is non-zero at a time.

[0257] Another example of a bilinear interpolation filter is shown below, in which case both filter coefficients are non-zero.

[0258] [Table 3]

[0259] In one implementation, the value of K is equal to 8. In other examples, the value of M is less than 8.

[0260] In one implementation, both the value of M and the value of K are equal to 8.

[0261] In one implementation, the second function can be a function for linearly combining N matching costs according to dmvOffset, which is obtained in step 3. The linear combination of x and y is any expression of the form ax + by, where a and b are constants. In the example, the constants a and b can be determined based on dmvOffset. An example of the second function is given below.

[0262] In one implementation, the second function can be as follows: ·Sad[1]*A + Sad[2]*B + Sad[3]*C + Sad[4]*D, where A, B, C, and D are greater than or equal to zero. In one example, A, B, C, and D may be numbers between 0 and 1 that add up to 1 (i.e., A+B+C+D=1). In another example, A, B, C, and D may be numbers greater than or equal to zero that add up to a predetermined number P, which could be 1, 2, 4, 8, 16, etc. A, B, C, and D may be predetermined numbers. A, B, C, and D can be derived according to dmvOffset

[0000] and dmvOffset

[0001] . In the example, A = dmvOffset

[0000] , B = P1 - dmvOffset

[0000] , C = dmvOffset

[0001] , and D = P2 - dmvOffset

[0001] . In the formulas, P1 and P2 can be equal to 1, 4, 8, 16, etc. The above equation is given as an example. The equation represents a linear combination of four matching costs to obtain the derived cost. In the equation, the dmvOffset which may be obtained in step 3 is used. The dmvOffset represents the difference between the improved motion vector and the initial motion vector. In one particular implementation, the dmvOffset is defined as the difference between MV0 and MV0'. More specifically, dmvOffset[0] may be the difference between the -x components of MV0 and MV0', while dmvOffset[1] may be the difference between the -y components of MV0 and MV0'.

[0263] In another implementation, the second function can be as follows: ·Sad[1]*A + Sad[2]*B + Sad[3]*C, where A, B, and C are greater than or equal to zero. In one example, A, B, and C may be numbers between 0 and 1 that add up to 1 (i.e., A+B+C=1). In another example, A, B, and C may be numbers greater than or equal to zero that add up to a predetermined number P, which could be 1, 2, 4, 8, 16, etc. A, B, and C can be predetermined numbers. A, B, and C can be derived according to dmvOffset

[0000] and dmvOffset

[0001] . In the example, A = P - dmvOffset

[0000] - dmvOffset

[0001] , B = dmvOffset

[0000] , and C = dmvOffset

[0001] . In the formula, P can be equal to 1, 4, 8, 16, etc. The above equation is given as an example. The equation represents a linear combination of three matching costs to obtain the derived cost. In the equation, the dmvOffset which may be obtained in step 3 is used. The dmvOffset represents the difference between the improved motion vector and the initial motion vector. In one example, the dmvOffset is defined as the difference between MV0 and MV0'. More specifically, dmvOffset[0] may be the difference between the -x components of MV0 and MV0', while dmvOffset[1] may be the difference between the -y components of MV0 and MV0'.

[0264] In another implementation, the second function for obtaining the derived cost could be: The parametric error surface function is calculated using the cost values ​​(e.g., SAD values) of the five evaluated differences between the improved MV pair and the candidate MV pair that is an integer distance from the improved MV pair. E(x,y) = A*(x - x0) 2 + B*(y - y0) 2 + C The equation is fitted, where (x0, y0) corresponds to the position where the difference between the two reference patches is minimized, C is the cost value at (x0, y0), and A and B are model coefficients. These five unknowns can be solved exactly if five cost values ​​are available. In other words, the equation for E(x,y) assumes that the form of the matching cost as a function of spatial positions near the position of minimum matching cost is parabolic.

[0265] In one embodiment, pairs of candidate MVs to the left, top, right, and bottom of an improved MV pair, with a distance of one integer pixel, are used. In this case, given the evaluated values ​​of E(x,y) for (x,y) positions (0,0), (-1,0), (0,-1), (1,0), and (0,1), and the parametric equation for E(x,y), the five unknowns A, B, C, x0, y0 can be solved as follows:

[0266]

number

[0267] On the other hand, if cost values ​​for six or more positions are available, the five unknowns can be solved using the least squares method or a similar technique. In that case, the obtained value of C becomes the derived cost.

[0268] In one implementation, the second function can be as follows:

[0269]

number

[0270] In the formula, K is a scalar greater than 0, and sad[0] through sad[4] are the N matching costs.

[0271] In one example, the motion vector improvement process is performed two or more times to further improve the motion vector. In this example, the initial motion vector is first improved by the motion vector improvement process to obtain a first improved motion vector. Then, the motion vector improvement is performed again, in which case the first improved motion vector is considered the initial motion vector for the improvement of the second motion vector.

[0272] An initial motion vector, which is the input to the motion vector improvement unit, is obtained at 925. A search space is constructed around the initial motion vector by the motion vector improvement unit (930). In one example, the search space consists of pairs of candidate motion vectors, where the first motion vector of a pair corresponds to a first reference picture, and the second motion vector of a pair corresponds to a second reference picture. A first prediction corresponding to each pair of candidate motion vectors is obtained at step 910 by applying an M-tap interpolation filter. As part of the motion vector improvement, matching costs corresponding to N pairs of motion vectors in the search space are calculated (915). The N matching costs are used as part of two processes: the first process is motion vector improvement (930), in which the matching costs are used to calculate improved pairs of motion vectors (935) by a function that takes the N matching costs as input. The second process is a decision on whether or not an optical flow improvement (950) is applied, which is made at 945. After the improved motion vector is obtained, a second prediction for the current block is obtained by (940). If the matching cost is greater than the threshold, optical flow improvements are applied, and the prediction in 940 is modified by 950 to obtain the modified prediction (955-960). The modified prediction generally has different sample values ​​from the second prediction in step 940. If the matching cost is less than the threshold, optical flow improvements are not applied, and the second prediction is set as the output (the final prediction for the current block).

[0273] According to a third exemplary embodiment of the present invention, the following steps are applied to obtain a prediction regarding the current coding block. Step 0: Obtain a pair of initial motion vectors based on the indication information in the bitstream. Step 1: Obtain the first set of predicted samples based on the initial MV pair and the M-tap interpolation filter. Step 2: Use the first set of predicted samples to obtain the first matching cost corresponding to the initial MV pairs. Step 3: Determine if the current coding block is eligible to perform a motion vector improvement. Step 4: If in Step 3 the current coding block is determined to be eligible to run MVR, Step 4a: Use the motion vector improvement process to obtain the improved pairs of MVs and the matching costs corresponding to the improved pairs of MVs, according to the initial pairs of MVs and their matching costs. Step 4b: Obtain a second set of samples predicted by the improved MV pairs and the K-tap interpolation filter. Step 4c: Determine whether to run the optical flow improvement process according to the second matching cost. In the example, the matching cost is compared to a threshold, and the optical flow improvement process is run when the value of the matching cost is greater than or equal to the threshold. Step 5: Otherwise (if in Step 3 it is determined that the current coding block is not suitable for running MVR), Step 5a: Obtain the initial MV pair and a second set of samples predicted by the K-tap interpolation filter. Step 5b: Determine whether to run the optical flow improvement process according to the first matching cost. In the example, the matching cost is compared to a threshold, and the optical flow improvement process is run when the value of the matching cost is greater than or equal to the threshold. Step 6: If it is determined (in either Step 4c or Step 5b) that the optical flow improvement process needs to be performed, the optical flow improvement is applied with the second prediction as input and the modified second prediction as output. If it is determined negatively, the optical flow improvement is not applied to the second prediction. In other words, if it is determined that the optical flow improvement process needs to be performed, the final prediction for the current coding block is obtained by the second prediction and the optical flow improvement process. Otherwise, the final prediction for the current coding block is obtained by the second prediction without the application of the optical flow improvement process.

[0274] This embodiment is further illustrated in the flowchart of Figure 9. Block 1110 receives an initial pair of MVs for the current coding block for prediction at references L0 and L1. Block 1110 corresponds to step 1, and a first set of predicted samples is obtained using the initial pair of MVs and the reconstructed reference samples of pictures L0 and L1. Block 1120 corresponds to step 2, and a first matching cost (or difference metric such as SAD) is evaluated among the first set of predicted blocks of samples corresponding to the initial pair of MVs (as described in the background MVR section). Block 1130 corresponds to step 3, and the suitability conditions for the current coding block on which to perform the MVR are examined. Block 1140 corresponds to step 4a, where, if the current coding block is found to be eligible to perform MVR, the improved MV pair is obtained by performing MVR (as described in the background MVR section), and the second matching cost (or difference metric) corresponding to the improved MV pair is obtained. Block 1150 corresponds to step 4b, where the second set of predicted samples is obtained using the improved MV pair (horizontally and vertically) with a K-tap interpolation filter. Block 1160 corresponds to step 4c, where it is checked whether the second matching cost is below a predetermined threshold, and if it is below the threshold, the improvement and bi-prediction based on the optical flow of bi-prediction are skipped. Block 1180 corresponds to step 5a, where the current coding block skips MVR and obtains the second set of predicted samples using the initial MV pair with a K-tap interpolation filter. Block 1185 corresponds to step 5b, where it is checked whether the first matching cost is below a predetermined threshold. If it is below the threshold, the BPOF is skipped.Blocks 1170 and 1195 correspond to part of step 6, and if the check in step 4c or step 5b indicates that the second or first matching cost is below a predetermined threshold, and BPOF is skipped because it is below the threshold, a weighted average of the bipredictions without BPOF is performed using the second set of predicted samples. Block 1175 corresponds to part of step 6, and if the check in step 4c or step 5b indicates that the second or first matching cost is not below a predetermined threshold, and BPOF is skipped because it is below the threshold, the estimated optical flow is obtained, and the final biprediction is obtained using the second set of predicted samples, the gradient of the second set of predicted samples, and the estimated optical flow.

[0275] It should be noted that the decision to skip or perform BPOF may vary for each subblock of the MVR within the coding unit, by using a first or second matching cost calculated with respect to the subblock of samples within the current coding unit required by the motion vector improvement process, in order to determine the early termination of the improvement process based on the optical flow of biprediction. BPOF is applied or skipped for all BPOF application units (e.g., 4x4 blocks at the pixel level or sample level) within the subblock, based on the decision made in step 4c or step 5.

[0276] In certain embodiments, it is possible to perform further early termination with respect to each BPOF application unit within a subblock by obtaining a partial matching cost corresponding to each BPOF application unit within the subblock of the MVR.

[0277] A predetermined threshold is generally selected as a sample-by-sample threshold that depends on the bit depth of the first prediction or first set of predicted samples. For example, if the first predicted sample value obtained using bilinear (2-tap) interpolation is constrained to have a bit depth of b, then the sample-by-sample threshold is k*2. (b-10) If the number of samples for which the matching cost is calculated is N, then the predetermined threshold for which the matching cost for the current lower block is compared is k*N*2. (10-b) The sample value for k is 2 (with respect to a bit depth of 10), N is 8 × 16 = 128, and b is 8. The value of N should be used accordingly, since the matching cost of a given pair of candidate MVs can be calculated using a greatly reduced set of the first predicted samples. For example, if every other row of an 8 × 16 block of predicted samples is used, N is calculated to be 8 × 8 = 64.

[0278] According to embodiments of the present invention, an early termination method is provided to conditionally skip the application of the optical flow improvement process, which is considered computationally intensive. As a result, the average decoding time is reduced.

[0279] Furthermore, the conditions for conditionally skipping the application of optical flow are determined based on parameters calculated by another process (which calculates the matching cost during the motion vector improvement process). Since already calculated values ​​are used, no additional calculations need to be performed.

[0280] In particular, a method for video coding performed in a decoding or encoding device, as shown in Figure 10, is provided. The method includes the following steps, which may be performed in a given order: An initial motion vector is obtained with respect to the current block 1210. The current block may be the current coding block. A first prediction about the sample values ​​in the current block is obtained based on the initial motion vector 1220. A matching cost is calculated according to the first prediction 1230.

[0281] After the first matching cost is obtained, it is determined whether or not the optical flow improvement process should be performed according to at least one pre-defined condition 1240, the at least one pre-defined condition includes whether the calculated matching cost (e.g., by a measure of similarity; see description above) is greater than or equal to a predefined threshold. When it is determined that the optical flow improvement process should be performed, the optical flow improvement process is performed to obtain the final inter-prediction for the sample values ​​in the current block 1250. When it is determined that the optical flow improvement process should not be performed, computational costs may be saved by skipping the optical flow improvement process.

[0282] This method can be carried out in the apparatus described above with reference to Figures 1a to 5.

[0283] In particular, the method can be implemented in the context of a motion vector improvement process on the decoder side. The inputs to such a process are as follows: Luma position (xSb, ySb) specifies the top-left sample of the current coding subblock relative to the top-left Luma sample of the current picture. The variable sbWidth specifies the width of the current coding subblock using Luma samples. The variable sbHeight specifies the height of the current coding subblock using LumaSample. Luma motion vectors mvL0 and mvL1 with 1 / 16 fractional sample precision Selected Luma reference picture sample array refPicL0 L and refPicL1 L

[0284] The output of this process is the delta luma motion vectors dMvL0 and dMvL, as well as a variable dmvrSad that specifies the minimum absolute difference sum of the first predictions (see SAD calculation above).

[0285] The delta-motion vector dMvL0 can be derived by dMvL0

[0000] += 16 * intOffX and dMvL0

[0001] += 16 * intOffY, where intOffX and intOffY are integer sample offsets in the x and y directions, respectively. Furthermore, the delta-motion vector dMvL can be calculated as dMvL1

[0000] = -dMvL0

[0000] and dMvL1

[0001] = -dMvL0

[0001] .

[0286] The first predicted Luma sample value is derived by bilinear interpolation of a fraction of samples. In the decoding process of the interpreted block, a bidirectional optical flow sample prediction process may or may not be applied. If the bidirectional optical flow sample prediction process is not applied, a weighted sample prediction process is applied to an improved second prediction obtained based on the improved motion vector. If the bidirectional optical flow sample prediction process is applied, the bidirectional optical flow sample prediction process receives a second prediction obtained based on the improved motion vector as well as the input, and outputs the final prediction.

[0287] A flag may be used to signal whether a bidirectional optical flow sample prediction process is applicable. For example, a flag being TRUE might be considered a necessary condition for the bidirectional optical flow sample prediction process to be performed. However, this necessary condition may not be a sufficient condition for the bidirectional optical flow sample prediction process to be performed. A sufficient condition might be both that the flag is TRUE and that the matching cost described above is greater than or equal to a predefined threshold. For example, the matching cost might be determined based on a variable dmvrSad that specifies the minimum sum of absolute differences of the first prediction.

[0288] On the other hand, if the flag is FALSE, it may be considered a sufficient condition to perform the weighted sample prediction process without performing the optical flow sample prediction process.

[0289] Furthermore, a device 1300 for use in an image encoder and / or image decoder is provided, as shown in Figure 11. According to this exemplary embodiment, the device 1300 includes an initial motion vector unit 1310 configured to acquire an initial motion vector relating to the current block. Furthermore, the device 1300 includes a prediction unit 1320 configured to acquire a first prediction relating to the sample values ​​in the current block based on the initial motion vector. Furthermore, the device 1300 includes a matching cost calculation unit 1330 configured to calculate the matching cost according to the first prediction.

[0290] Device 1300 includes an optical flow improvement process determination unit 1340 configured to determine whether an optical flow improvement process should be performed according to at least one pre-set condition, the at least one pre-set condition being whether the calculated matching cost is greater than or equal to a threshold. Furthermore, device 1300 includes an optical flow improvement process execution unit 1350 configured to execute the optical flow improvement process to obtain a final inter-prediction for the sample values ​​in the current block when it is determined that the optical flow improvement process should be performed.

[0291] Mathematical operators The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are more strictly defined, and additional operations such as exponentiation and real-valued division are defined. The numbering and counting rules generally start from 0, for example, "1st" is equivalent to 0, "2nd" is equivalent to 1, and so on.

[0292] Arithmetic operators The following arithmetic operators are defined as follows:

[0293]

number

[0294] Logical operators The following logical operators are defined as follows: x && y: Boolean "product" of x and y x || y Boolean "union" of x and y ! Boolean logic "negation" x ? y : If x is true or not equal to 0, it evaluates to the value y; otherwise, it evaluates to the value z.

[0295] Relational operators The following relational operators are defined as follows: > larger >= Above < Less than <= Below == equal != Not equal

[0296] When a relational operator is applied to a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is treated as a different value for that syntax element or variable. The value "na" is considered not to be equal to any other value.

[0297] Bitwise operators The following bitwise operators are defined as follows: The AND operator performs a bitwise "logical AND". When used with integer arguments, it operates on the two's complement representation of the integer value. When used with binary arguments containing fewer bits than the other argument, the shorter argument is extended by adding higher-order bits equal to zero. | Bitwise "logical OR". When performed on integer arguments, it operates on the two's complement representation of the integer value. When performed on binary arguments containing fewer bits than another argument, the shorter argument is extended by adding higher-order bits equal to 0. ^ Bitwise "exclusive OR". When performed on integer arguments, it operates on the two's complement representation of the integer value. When performed on binary arguments containing fewer bits than another argument, the shorter argument is extended by adding higher-order bits equal to 0. An arithmetic right shift of y digits in the two's complement representation of an integer x >> yx. This function is defined only for non-negative integer values ​​of y. The bit shifted to the most significant bit (MSB) as a result of the right shift has a value equal to the MSB of x before the shift. x << y: Arithmetic left shift of the two's complement integer representation of x by y bits in binary. This function is only defined for non-negative integer values of y. The bit shifted into the least significant bit (LSB) as a result of the left shift has a value equal to 0.

[0298] Assignment operator The following arithmetic operators are defined as follows. = Assignment operator ++ Increment. That is, x++ is equivalent to x = x + 1, and when used as an array index, it is evaluated with the value of the variable before the increment operation. -- Decrement. That is, x-- is equivalent to x = x - 1, and when used as an array index, it is evaluated with the value of the variable before the decrement operation. += Increment by the specified amount. That is, x += 3 is equivalent to x = x + 3, and x += (-3) is equivalent to x = x + (-3). -= Decrement by the specified amount. That is, x -= 3 is equivalent to x = x - 3, and x -= (-3) is equivalent to x = x - (-3).

[0299] Range notation The following notations are used to specify a range of values. x = y..z: x takes integer values from y to z, including y and z, assuming that x, y, and z are integer values and z is greater than y.

[0300] Mathematical functions The following mathematical functions are defined.

[0301]

Number

[0302] Asin(x) is a trigonometric inverse sine function that acts on an argument x in the range of -1.0 to 1.0, including -1.0 and 1.0, and has output values ​​in radians in the range of -π÷2 to π÷2, including -π÷2 and π÷2. Atan(x) is the trigonometric inverse tangent function that acts on the argument x and has an output value in radians ranging from -π÷2 to π÷2, including -π÷2 and π÷2.

[0303]

number

[0304] Ceil(x) The smallest integer greater than or equal to x. Clip1 Y ( x ) = Clip3( 0, ( 1 << BitDepth Y ) - 1, x ) Clip1 C ( x ) = Clip3( 0, ( 1 << BitDepth C ) - 1, x )

[0305]

number

[0306] Cos(x) is the trigonometric cosine function acting on an argument x in radians. Floor(x): The largest integer less than or equal to x.

[0307]

number

[0308] Ln(x) is the natural logarithm of x (a logarithm with base e, where e is the base constant of the natural logarithm, 2.718281828...). Log2(x) is the logarithm of x with base 2. Log10(x) is the base-10 logarithm of x.

[0309]

number

[0310] Round( x ) = Sign( x ) * Floor( Abs( x ) + 0.5 )

[0311]

number

[0312] Sin(x) is the trigonometric sine function acting on an argument x in radians.

[0313]

number

[0314] Tan(x) is the trigonometric tangent function that acts on an argument x in radians.

[0315] Order of operations When precedence in an expression is not explicitly indicated using parentheses, the following rules apply: - Higher-priority operations are evaluated before any lower-priority operations. - Operations with the same priority are evaluated from left to right.

[0316] The table below clearly shows the order of operations from highest to lowest, with higher positions in the table indicating higher priority.

[0317] With respect to operators also used in the C programming language, the precedence used herein is the same as that used in the C programming language.

[0318] Table: Priority of operations from the highest (top of the table) to the lowest (bottom of the table)

[0319] [Table 4]

[0320] Text description of logical operations In the text, in the following form, namely, if (condition 0) Statement 0 else if (condition 1) Statement 1 ... else / * Comment providing information about the remaining conditions * / statement n Statements of logical operations that are mathematically described in this form may be written as follows: The following applies: - If condition 0, statement 0 - Instead, if condition 1 is true, then statement 1 - ... - Otherwise (comments providing information about the remaining conditions), statement n

[0321] Each "If..., ..., otherwise..., ..., ..." statement in the text is introduced by "If..., ..." immediately followed by "The following applies..." or "The following applies...". The final condition of "If..., ..., otherwise..., ..., ..., ..., ..." is always "The following applies...". Alternating "If..., ..., otherwise..., ..., ..., ..." statements can be identified by matching "The following applies..." or "The following applies..." with the final "The following applies...".

[0322] In the text, in the following form, namely, if( condition 0a && condition 0b ) Statement 0 else if( condition 1a || condition 1b ) Statement 1 ... else statement n Statements of logical operations that are mathematically described in this form may be written as follows: The following applies: - If all of the following conditions are true, then statement 0 - Condition 0a - Condition 0b - Otherwise, if one or more of the following conditions are true, then statement 1 - Condition 1a - Condition 1b - ... - Otherwise, statement n

[0323] In the text, in the following form, namely, if (condition 0) Statement 0 if (Condition 1) Statement 1 Statements of logical operations that are mathematically described in this form may be written as follows: When condition 0, statement 0 When condition 1 is met, statement 1

[0324] Although embodiments of the present invention have been described primarily in relation to video coding, it should be noted that embodiments of the coding system 10, encoder 20, and decoder 30 (and correspondingly system 10), as well as other embodiments described herein, may also be configured for processing or coding still pictures, i.e., for processing or coding individual pictures independently of any preceding or consecutive pictures, similar to video coding. Generally, when the processing and coding of a picture is limited to a single picture 17, only the interpretation units 244 (encoder) and 344 (decoder) may not be available. All other functions (also called tools or technologies) of the video encoder 20 and video decoder 30, such as residual calculation 204 / 304, transformation 206, quantization 208, dequantization 210 / 310, (inverse) transformation 212 / 312, partitioning 262 / 362, intra prediction 254 / 354, and / or loop filters 220, 320, and entropy coding 270, and entropy decoding 304, may be used equally for processing still pictures.

[0325] For example, the encoder 20 and decoder 30, and embodiments of the functions described herein in relation to the encoder 20 and decoder 30, for example, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted over a communication medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates the transfer of computer programs from one place to another, for example, by a communication protocol. Thus, generally speaking, the computer-readable medium may correspond to (1) a non-transient, tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementations of the technologies described herein. Computer program products may include computer-readable medium.

[0326] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is appropriately called computer-readable media. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio waves, and microwaves, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio waves, and microwaves are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary, tangible storage media. As used herein, "disk" and "disc" include compact discs (CDs), laser discs (registered trademark), optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. Combinations of the above should also be included in the scope of computer-readable media.

[0327] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, when used herein, the term “processor” may refer to either the above-described structures or any other structure suitable for implementing the technologies described herein. In addition, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the technologies may all be implemented in one or more circuits or logic elements.

[0328] The technology disclosed herein may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to highlight the aspects of functionality of a device configured to perform the disclosed technology, but implementation by different hardware units is not necessarily required. Rather, as described above, various units may be combined in a codec hardware unit or provided by a set of interoperable hardware units, including one or more of the aforementioned processors, in conjunction with suitable software and / or firmware. [Explanation of Symbols]

[0329] 10 Video coding system, coding system 12 Source device 13 Encoded picture data, communication channel 14 Destination device 16 Picture Sources 17. Picture, picture data, raw picture, raw picture data, monochrome picture, color picture, current picture 18 Preprocessors, pre-processing units, picture preprocessors 19 Pre-processed pictures, pre-processed picture data 20 video encoders, encoders 21 Encoded picture data, encoded bitstream 22 Communication interface, communication unit 28 Communication interface, communication unit 30 decoders, video decoders 31 Decrypted picture data, decrypted picture 32 Post-processors, Post-processing Units 33 Post-processed picture data, post-processed picture 34 Display Devices 46 Processing Circuit 100 video encoders 201 Input, Input Interface 203 Picture Block, Original Block, Current Block, Divided Block, Current Picture Block 204 Residual Calculation Unit, Residual Calculation 205 Residual block, residual 206 Conversion processing unit, conversion 207 Conversion coefficient 208 Quantization Unit, Quantization 209 Quantized coefficients, quantized transformation coefficients, quantized residual coefficients 210 Dequantization Unit, Dequantization 211 Dequantized coefficients, dequantized residual coefficients 212 Inverse transformation processing unit, (inverse) transformation 213 Reconstructed residual blocks, dequantized coefficients, transform blocks 214 Reconstruction Unit, Adder, Combiner 215 Reconstructed Blocks 216 buffers 220 Loop Filter Unit, Loop Filter 221 Filtered blocks, filtered and reconstructed blocks 230 Decoded Picture Buffer (DPB) 231 Decrypted picture 244 Interpretation Units 254 Intra Prediction Unit, Inter Prediction Unit, Intra Prediction 260 Mode Selection Unit 262 division units, division 265 prediction blocks, predictors 266 Syntax Elements 270 Entropy coding units, entropy coding 272 outputs, output interface 304 Entropy decoding unit, residual calculation, entropy decoding 309 Quantized coefficients 310 Dequantization Unit, Dequantization 311 Dequantized coefficients, transformation coefficients 312 Inverse conversion processing unit, (inverse) conversion, output 313 Reconstructed residual block 314 Reconstruction Unit, Combiner, Adder 315 Reconstructed Blocks 320 Loop Filters, Loop Filter Units 321 Filtered blocks, decoded video blocks 330 Decoded Picture Buffer (DPB) 331 Decrypted picture 344 Interpretation Units 354 Intra Prediction Unit, Intra Prediction 360 Mode Applicable Unit 362 divisions 365 Prediction Block 400 video coding devices 410 Incoming port, input port 420 Receiver Unit (Rx) 430 Processors, Logical Units, Central Processing Units (CPUs) 440 Transmitter Unit (Tx) 450 outgoing ports, output ports 460 memory 470 coding modules 500 devices 502 Processors 504 memory 506 data 508 Operating Systems 510 Application Programs 512 Bus 514 Secondary Storage 518 displays 1300 devices 1310 Initial motion vector unit 1320 prediction units 1330 Matching Cost Calculation Unit 1340 Optical Flow Improvement Process Determination Unit 1350 Optical Flow Improvement Process Execution Unit

Claims

1. A video coding method implemented in a decoding device, A step of obtaining a pair of initial motion vectors for the current block, wherein the current block is a subblock, The steps include obtaining two first predicted blocks of the current block based on the pair of initial motion vectors, A step of calculating a first matching cost according to the two first prediction blocks, wherein the first matching cost is the sum of absolute differences (SAD) between the two first prediction blocks. A step of obtaining a refined pair of motion vectors based on the aforementioned pair of initial motion vectors, The steps include obtaining two second predicted blocks of the current block according to the refined pair of motion vectors, The steps include obtaining a second matching cost corresponding to the two second prediction blocks, A step of determining whether an optical flow refinement process should be performed according to at least one pre-set condition, wherein the at least one pre-set condition includes a condition of whether the second matching cost is less than a threshold, If it is determined that the second matching cost is not below a threshold, the optical flow refinement process is performed on the two second prediction blocks in order to obtain the final prediction block of the current block. The steps include determining that the optical flow refinement process should not be performed if the second matching cost is less than the threshold, If it is determined that the optical flow refinement process should not be performed, the final prediction block is obtained by a weighted sum of the two second prediction blocks. The steps include obtaining the quantized residual coefficients of the current block, The steps include applying inverse quantization to the quantized residual coefficient to obtain the dequantized residual coefficient, A step of obtaining a reconstructed residual block of the current block based on the quantized residual coefficients, To obtain the reconstructed block of the current block, the steps include adding the reconstructed residual block to the final predicted block, A method characterized by comprising the following:

2. The steps include determining whether the current block is suitable for performing a moving vector refinement (MVR), If it is determined that the current block is suitable for performing MVR, the steps include obtaining the refined motion vector pair based on the initial motion vector pair, The method according to claim 1, further comprising:

3. The method according to claim 1 or 2, wherein the at least one preset condition includes whether the current block can be predicted by decoder-side motion vector refinement.

4. The method according to claim 3, wherein it is determined that the optical flow refinement process should not be performed if the second matching cost is less than the threshold and it is determined that the current block can be predicted by the decoder-side motion vector refinement.

5. The method according to claim 3, wherein the optical flow refinement process is performed on the two second prediction blocks if it is determined that the second matching cost is not less than the threshold, or if it is determined that the current block is not permitted to be predicted by the decoder-side motion vector refinement.

6. The method according to any one of claims 1 to 5, wherein the two first predicted blocks of the current block are obtained based on a bilinear interpolation filter.

7. The method according to any one of claims 1 to 6, wherein the two second predicted blocks of the current block are obtained according to an 8-tap interpolation filter.

8. The method according to any one of claims 1 to 7, wherein the threshold is obtained according to the size of the current block.

9. A decoder equipped with a processing circuit, wherein the processing circuit is A step of obtaining a pair of initial motion vectors for the current block, wherein the current block is a subblock, The steps include obtaining two first predicted blocks of the current block based on the pair of initial motion vectors, A step of calculating a first matching cost according to the two first prediction blocks, wherein the first matching cost is the sum of absolute differences (SAD) between the two first prediction blocks. A step of obtaining a refined pair of motion vectors based on the aforementioned pair of initial motion vectors, The steps include obtaining two second predicted blocks of the current block according to the refined pair of motion vectors, The steps include obtaining a second matching cost corresponding to the two second prediction blocks, A step of determining whether an optical flow refinement process should be performed according to at least one pre-set condition, wherein the at least one pre-set condition includes a condition of whether the second matching cost is less than a threshold, If it is determined that the second matching cost is not below a threshold, the optical flow refinement process is performed on the two second prediction blocks in order to obtain the final prediction block of the current block. The steps include determining that the optical flow refinement process should not be performed if the second matching cost is less than the threshold, If it is determined that the optical flow refinement process should not be performed, the final prediction block is obtained by a weighted sum of the two second prediction blocks. The steps include obtaining the quantized residual coefficients of the current block, The steps include applying inverse quantization to the quantized residual coefficient to obtain the dequantized residual coefficient, A step of obtaining a reconstructed residual block of the current block based on the quantized residual coefficients, To obtain the reconstructed block of the current block, the steps include adding the reconstructed residual block to the final predicted block, A decoder characterized by being configured to perform the following action.

10. The processing circuit is The steps include determining whether the current block is suitable for performing a moving vector refinement (MVR), If it is determined that the current block is suitable for performing MVR, the steps include obtaining the refined motion vector pair based on the initial motion vector pair, The decoder according to claim 9, configured to further perform the following.

11. The decoder according to claim 9 or 10, wherein the at least one preset condition includes whether the current block can be predicted by decoder-side motion vector refinement.

12. The decoder according to claim 11, wherein it is determined that the optical flow refinement process should not be performed if the second matching cost is less than the threshold and it is determined that the current block can be predicted by the decoder-side motion vector refinement.

13. The decoder according to claim 11, wherein the optical flow refinement process is performed on the two second prediction blocks if it is determined that the second matching cost is not less than the threshold, or if it is determined that the current block is not permitted to be predicted by the decoder-side motion vector refinement.

14. The decoder according to any one of claims 9 to 13, wherein the two first predicted blocks of the current block are obtained based on a bilinear interpolation filter.

15. The decoder according to any one of claims 9 to 14, wherein the two second predicted blocks of the current block are obtained according to an 8-tap interpolation filter.

16. The decoder according to any one of claims 9 to 15, wherein the threshold is obtained according to the size of the current block.

17. A video coding method implemented in an encoding device, A step of obtaining a pair of initial motion vectors for the current block, wherein the current block is a subblock, The steps include obtaining two first predicted blocks of the current block based on the pair of initial motion vectors, A step of calculating a first matching cost according to the two first prediction blocks, wherein the first matching cost is the sum of absolute differences (SAD) between the two first prediction blocks. A step of obtaining a refined pair of motion vectors based on the aforementioned pair of initial motion vectors, The steps include obtaining two second predicted blocks of the current block according to the refined pair of motion vectors, The steps include obtaining a second matching cost corresponding to the two second prediction blocks, A step of determining whether an optical flow refinement process should be performed according to at least one pre-set condition, wherein the at least one pre-set condition includes a condition of whether the second matching cost is less than a threshold, If it is determined that the second matching cost is not below a threshold, the optical flow refinement process is performed on the two second prediction blocks in order to obtain the final prediction block of the current block. The steps include determining that the optical flow refinement process should not be performed if the second matching cost is less than the threshold, If it is determined that the optical flow refinement process should not be performed, the final prediction block is obtained by a weighted sum of the two second prediction blocks. A step of obtaining the residual block of the current block based on the current block and the final predicted block, The steps include obtaining residual coefficients based on the residual block, The steps include: applying quantization to the residual coefficient to obtain the quantized residual coefficient; The steps include encoding the quantized residual coefficients into a bitstream, A method characterized by comprising the following:

18. The steps include determining whether the current block is suitable for performing a moving vector refinement (MVR), If it is determined that the current block is suitable for performing MVR, the steps include obtaining the refined motion vector pair based on the initial motion vector pair, The method according to claim 17, further comprising:

19. The method according to claim 17 or 18, wherein the at least one preset condition includes whether the current block can be predicted by decoder-side motion vector refinement.

20. The method according to claim 19, wherein it is determined that the optical flow refinement process should not be performed if the second matching cost is less than the threshold and it is determined that the current block can be predicted by the decoder-side motion vector refinement.

21. The method according to claim 19, wherein the optical flow refinement process is performed on the two second prediction blocks if it is determined that the second matching cost is not less than the threshold, or if it is determined that the current block is not permitted to be predicted by the decoder-side motion vector refinement.

22. The method according to any one of claims 17 to 21, wherein the two first predicted blocks of the current block are obtained based on a bilinear interpolation filter.

23. The method according to any one of claims 17 to 22, wherein the two second predicted blocks of the current block are obtained according to an 8-tap interpolation filter.

24. The method according to any one of claims 17 to 23, wherein the threshold is obtained according to the size of the current block.

25. An encoder characterized by comprising a processing circuit configured to perform the method described in any one of claims 17 to 24.

26. A non-temporary computer-readable medium comprising a set of instructions for performing the method according to any one of claims 1 to 8.

27. A non-temporary computer-readable medium comprising an instruction set for performing the method according to any one of claims 17 to 24.

28. A method for generating a data structure, wherein the data structure includes a bitstream, the bitstream includes encoded picture data, instruction information for a pair of initial motion vectors for the current block, and quantized residual coefficients for the current block, the method is A step of obtaining a pair of initial motion vectors for the current block, wherein the current block is a subblock, The steps include obtaining two first predicted blocks of the current block based on the pair of initial motion vectors, A step of calculating a first matching cost according to the two first prediction blocks, wherein the first matching cost is the sum of absolute differences (SAD) between the two first prediction blocks. A step of obtaining a refined pair of motion vectors based on the aforementioned pair of initial motion vectors, The steps include obtaining two second predicted blocks of the current block according to the refined pair of motion vectors, The steps include obtaining a second matching cost corresponding to the two second prediction blocks, A step of determining whether an optical flow refinement process should be performed according to at least one pre-set condition, wherein the at least one pre-set condition includes a condition of whether the second matching cost is less than a threshold, If it is determined that the second matching cost is not below a threshold, the optical flow refinement process is performed on the two second prediction blocks in order to obtain the final prediction block of the current block. The steps include determining that the optical flow refinement process should not be performed if the second matching cost is less than the threshold, If it is determined that the optical flow refinement process should not be performed, the final prediction block is obtained by a weighted sum of the two second prediction blocks. A step of obtaining the residual block of the current block based on the current block and the final predicted block, The steps include obtaining residual coefficients based on the residual block, The steps include: applying quantization to the residual coefficient to obtain the quantized residual coefficient; The steps include encoding the quantized residual coefficients into a bitstream, A method characterized by comprising the following:

Images