Computing method of integer grid reference sample position for block-level boundary sample gradient calculation in bidirectional prediction optical flow calculation and bidirectional prediction correction

Abstract

Provided is a bidirectional optical flow prediction method, which is performed according to a reference sample value at a sample position in an extended region of a prediction block used in bidirectional optical flow prediction of a current block, in which a sub-pixel position in the extended region of the prediction block in a reference image is rounded, and a sub-pixel position in the extended region of the prediction block in the reference image is rounded. And obtaining an integer pixel sample position in the reference image, wherein a reference sample value at the integer pixel sample position in the obtained reference image is used for calculating a boundary sample gradient corresponding to the boundary of the prediction block in the reference image.

Description

[0001] This application is a divisional application. The original application has the application number 202080045664.3 and the original application date is June 23, 2020. The entire contents of the original application are incorporated herein by reference.

[0002] Related applications cross-application

[0003] This application claims priority to Indian Patent Application No. IN201931025013, filed with the Indian Patent Office on June 24, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The embodiments of this application generally relate to the field of image processing, and more specifically, to optical flow correction. Background Technology

[0005] Video decoding (video encoding and decoding) is widely used in digital video applications such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversational applications (such as video chat), video conferencing, DVDs and Blu-ray discs, video content capture and editing systems, and cameras for security applications.

[0006] Even with shorter videos, a large amount of video data needs to be described, which can be challenging when the data needs to be transmitted over bandwidth-constrained communication networks or otherwise. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to limited memory resources. Video compression devices typically use software and / or hardware on the source side to encode video data before transmission or storage, thereby reducing the amount of data required to represent digital video images. The compressed data is then received at the destination by a video decompression device used to decode the video data. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques that can increase compression ratios with minimal impact on image quality. Summary of the Invention

[0007] This application provides embodiments of the apparatus and methods for encoding and decoding as described in the independent claims.

[0008] The above and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0009] A first embodiment of the present invention provides a bidirectional optical flow prediction method implemented by an encoding device or a decoding device. The bidirectional optical flow prediction method is based on reference sample values ​​at sample positions in the extended region of the prediction block used in the current block's bidirectional optical flow prediction. The method includes: obtaining the motion vector (MV) of the current block; obtaining sub-pixel sample positions in the extended region of the prediction block in a reference image based on the MV; processing the obtained sub-pixel sample positions to obtain integer pixel sample positions in the reference image; and performing bidirectional optical flow prediction by referring to the reference sample values ​​at the obtained integer pixel sample positions in the reference image.

[0010] Processing the obtained sub-pixel sample positions to obtain integer pixel sample positions in the reference image can reduce the memory space occupied for bi-directional optical flow (BDOF) prediction and allow the establishment of sub-block-level pipelines in hardware designs, since the extended sample values ​​(i.e., sample positions outside the M×N sample positions corresponding to the sub-block) are not obtained by motion-compensated interpolation of the corresponding reference samples.

[0011] The reference sample value at the integer pixel sample position in the obtained reference image can be referred to as the sample value at the sub-pixel sample position in bidirectional optical flow prediction.

[0012] The process of processing the obtained sub-pixel sample position includes: rounding the obtained sub-pixel sample position to obtain the integer pixel sample position in the reference image.

[0013] By rounding the sub-pixel positions in both the horizontal and vertical directions, the boundary gradients are more accurate. This improves the optical flow calculated based on the sample gradients, and also improves the correction performed using the calculated optical flow. In previous methods, fractional offsets in the horizontal and vertical directions were ignored when selecting integer sample values ​​for the extended sample positions. This method provides consistent decoding gain compared to previous methods. Simultaneously, it does not significantly increase complexity, as complexity is only increased when rounding in the horizontal and vertical directions to add offsets.

[0014] The process of processing the obtained sub-pixel sample position may include: rounding the obtained sub-pixel sample position to the nearest integer pixel sample position in the horizontal and vertical directions respectively, and obtaining the integer pixel sample position in the reference image based on the nearest integer pixel sample position in the horizontal and vertical directions.

[0015] The method may further include: before performing bidirectional optical flow prediction, obtaining reference sample values ​​at integer pixel sample positions in the obtained reference image as prediction sample values ​​for bidirectional optical flow prediction, wherein performing bidirectional optical flow prediction by referring to the reference sample values ​​includes: performing bidirectional optical flow prediction using the prediction sample values.

[0016] In one implementation, rounding the obtained sub-pixel sample positions in the horizontal and vertical directions respectively includes: comparing the fractional portion of the obtained sub-pixel sample positions in the corresponding directions with a threshold, wherein, if the fractional portion is equal to or greater than the threshold, the nearest high-order integer pixel sample position in the corresponding direction is obtained as the closest integer pixel sample position in the corresponding direction; if the fractional portion is less than the threshold, the nearest low-order integer pixel sample position in the corresponding direction is obtained as the closest integer pixel sample position in the corresponding direction.

[0017] The nearest high-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain a truncated integer pixel sample position in the corresponding direction, and adding an offset of 1 to the truncated integer pixel sample position in the corresponding direction; the nearest low-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain a truncated integer pixel sample position in the corresponding direction, and adding an offset of 0 to the truncated integer pixel sample position in the corresponding direction.

[0018] For a subpixel precision of 1 / P, the threshold is P / 2. For a subpixel precision of 1 / 16, the threshold is 8.

[0019] In one implementation, rounding the obtained sub-pixel sample position in the horizontal and vertical directions respectively includes applying a shift operation and a reverse shift operation to the components of the obtained sub-pixel sample position in the corresponding directions.

[0020] The integer pixel sample positions (x_pos, y_pos) in the reference image are obtained according to the following equation:

[0021] x_pos=(x_spel_pos+(1<<(shift_val–1)))>>shift_val

[0022] y_pos=(y_spel_pos+(1<<(shift_val–1)))>>shift_val

[0023] Where (x_spel_pos, y_spel_pos) represents the obtained sub-pixel sample position, and shift_val is selected according to the sub-pixel sample precision used. For sub-pixel sample precision of half pixel, quarter pixel, 1 / 8 pixel, or 1 / 16 pixel, shift_val is 1, 2, 3, or 4, respectively.

[0024] In one implementation, the MV of the current block is a modified MV obtained by performing motion vector refinement (MVR) on the initial MV.

[0025] The MVR includes an integer distance correction stage for the initial MV, wherein, when the integer distance correction stage of the MVR produces a non-zero displacement of the corrected MV relative to the initial MV, the integer pixel sample position in the reference image is obtained by rounding the obtained sub-pixel sample position.

[0026] In the case where the corrected MV has zero displacement relative to the initial MV during the integer distance correction stage of the MVR, the integer pixel sample position in the reference image is obtained by discarding a fraction of the obtained sub-pixel sample position.

[0027] In one implementation, when the corrected MV is generated with zero displacement relative to the initial MV during the integer distance correction phase of the MVR, the integer pixel sample positions (x_pos, y_pos) in the reference image are obtained according to the following equation:

[0028] x_pos = x_spel_pos >> shift_val

[0029] y_pos=y_spel_pos>>shift_val

[0030] Where (x_spel_pos, y_spel_pos) represents the obtained sub-pixel sample position, and shift_val is selected according to the sub-pixel sample precision used. For sub-pixel sample precision of half pixel, quarter pixel, 1 / 8 pixel, or 1 / 16 pixel, shift_val is 1, 2, 3, or 4, respectively.

[0031] The current block can be the current decoding block, the current decoding sub-block, or the current prediction block.

[0032] The reference sample values ​​at integer pixel sample locations in the obtained reference image are used to calculate the boundary sample gradient corresponding to the boundary of the prediction block in the reference image.

[0033] According to one aspect of the first embodiment, an encoder is provided, including processing circuitry for performing any of the methods provided in the first embodiment.

[0034] According to another aspect of the first embodiment, a decoder is provided, including processing circuitry for performing any of the methods provided in the first embodiment.

[0035] According to another aspect of the first embodiment, a computer program product is provided, the computer program product including instructions, which, when executed by a computer, perform any of the methods provided in the first embodiment.

[0036] According to another aspect of the first embodiment, an encoder is provided, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing instructions executed by the one or more processors, wherein, when the one or more processors execute the instructions, the encoder performs any of the methods provided in the first embodiment.

[0037] According to another aspect of the first embodiment, a decoder is provided, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing instructions executable by the one or more processors, wherein, when the one or more processors execute the instructions, the decoder performs any of the methods provided in the first embodiment.

[0038] According to another aspect of the first embodiment, an encoder is provided, including a prediction unit for performing bidirectional optical flow prediction of a current block; a first acquisition unit for acquiring the motion vector (MV) of the current block; a second acquisition unit for acquiring sub-pixel sample positions in an extended region of the prediction block in a reference image based on the MV; and a processing unit for processing the acquired sub-pixel sample positions to obtain integer pixel sample positions in the reference image; wherein the prediction unit is used to perform bidirectional optical flow prediction by referring to reference sample values ​​at the obtained integer pixel sample positions in the reference image.

[0039] In one implementation, processing the obtained sub-pixel sample position may include: rounding the obtained sub-pixel sample position to obtain the integer pixel sample position in the reference image.

[0040] In one implementation, processing the obtained sub-pixel sample position may include: rounding the obtained sub-pixel sample position to the nearest integer pixel sample position in the horizontal and vertical directions, respectively, and obtaining the integer pixel sample position in the reference image based on the nearest integer pixel sample positions in the horizontal and vertical directions.

[0041] The step of rounding the obtained sub-pixel sample positions in the horizontal and vertical directions may include: comparing the fractional portion of the obtained sub-pixel sample positions in the corresponding directions with a threshold, wherein...

[0042] If the fraction is equal to or greater than the threshold, the position of the nearest high-order integer pixel sample in the corresponding direction is obtained as the position of the nearest integer pixel sample in the corresponding direction; if the fraction is less than the threshold, the position of the nearest low-order integer pixel sample in the corresponding direction is obtained as the position of the nearest integer pixel sample in the corresponding direction.

[0043] The nearest high-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain a truncated integer pixel sample position in the corresponding direction, and adding an offset of 1 to the truncated integer pixel sample position in the corresponding direction; the nearest low-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain a truncated integer pixel sample position in the corresponding direction, and adding an offset of 0 to the truncated integer pixel sample position in the corresponding direction.

[0044] According to another aspect of the first embodiment, a decoder is provided, comprising: a prediction unit for performing bidirectional optical flow prediction of a current block; a first acquisition unit for acquiring the motion vector (MV) of the current block from the bitstream of an encoded video; a second acquisition unit for acquiring sub-pixel sample positions in an extended region of the predicted block in a reference image based on the MV; and a processing unit for processing the acquired sub-pixel sample positions to obtain integer pixel sample positions in the reference image; wherein the prediction unit is used to perform bidirectional optical flow prediction by referring to reference sample values ​​at the obtained integer pixel sample positions in the reference image.

[0045] In one implementation, processing the obtained sub-pixel sample position may include: rounding the obtained sub-pixel sample position to obtain the integer pixel sample position in the reference image.

[0046] In one implementation, processing the obtained sub-pixel sample position may include: rounding the obtained sub-pixel sample position to the nearest integer pixel sample position in the horizontal and vertical directions, respectively, and obtaining the integer pixel sample position in the reference image based on the nearest integer pixel sample positions in the horizontal and vertical directions.

[0047] The step of rounding the obtained sub-pixel sample positions in the horizontal and vertical directions may include: comparing the fractional portion of the obtained sub-pixel sample positions in the corresponding directions with a threshold, wherein...

[0048] If the fraction is equal to or greater than the threshold, the position of the nearest high-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction; if the fraction is less than the threshold, the position of the nearest low-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction.

[0049] The nearest high-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain a truncated integer pixel sample position in the corresponding direction, and adding an offset of 1 to the truncated integer pixel sample position in the corresponding direction; the nearest low-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain a truncated integer pixel sample position in the corresponding direction, and adding an offset of 0 to the truncated integer pixel sample position in the corresponding direction.

[0050] According to another aspect of the first embodiment, a storage medium is provided for storing a bitstream obtained by using any of the methods of the first embodiment.

[0051] A second embodiment of the present invention provides a method for obtaining predicted sample values ​​of sample positions in an extended region of a prediction block used in bidirectional optical flow prediction of a current decoding block. The method includes: obtaining a first MV corresponding to the predicted sample in the extended region of the decoding block; obtaining a second MV by discarding a fractional portion of the first MV; obtaining a first integer pixel position of the extended region in a reference image based on the second MV; determining a fractional offset vector based on the first MV; and obtaining a second integer pixel position in the reference image based on the first integer pixel position and the fractional offset vector, wherein the second integer pixel position is set as the first integer pixel position plus a positional offset, the positional offset being determined by comparing each component of the fractional offset vector with a threshold. The method further includes: filling the predicted sample with pixel values ​​from the second integer pixel position in the reference image.

[0052] The position offset component being 1 or 0 can depend on the comparison result between the corresponding component of the fractional offset vector and the threshold, wherein the component includes an X component and a Y component.

[0053] The threshold can be K, where the component of the position offset can be set to 1 when the corresponding component of the fractional offset is equal to or greater than K; otherwise, it is set to 0. K can be equal to 7.

[0054] According to one aspect of the second embodiment, an encoder or decoder is provided, including processing circuitry for performing any of the methods provided in the second embodiment.

[0055] According to another aspect of the second embodiment, a computer program product is provided, including program code for performing any of the methods provided in the second embodiment.

[0056] According to another aspect of the second embodiment, a decoder or encoder is provided, including one or more processors and a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium is coupled to the processor and stores a program executed by the processor, wherein, when the processor executes the program, it causes the decoder or encoder to perform any of the methods provided in the second embodiment.

[0057] One or more embodiments will be described in detail in the following drawings and description. Other features, objectives, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0058] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0059] Figure 1A This is a block diagram of an exemplary video decoding system for implementing embodiments of the present invention;

[0060] Figure 1B This is a block diagram of another exemplary video decoding system for implementing embodiments of the present invention;

[0061] Figure 2 This is a block diagram of an exemplary video encoder used to implement embodiments of the present invention;

[0062] Figure 3 This is a block diagram of an exemplary structure for implementing a video decoder according to an embodiment of the present invention;

[0063] Figure 4 This is a block diagram of an exemplary encoding or decoding device;

[0064] Figure 5This is a block diagram of another exemplary encoding or decoding device;

[0065] Figure 6 This is a diagram illustrating the process of rounding sub-pixel positions in one direction of the reference image during the acquisition of integer samples for BDOF gradient calculation.

[0066] Figure 7 This is a diagram showing the integer positions in the sub-pixel positions of the reference image during the integer sample acquisition process used for BDOF gradient calculation.

[0067] Figure 8 A flowchart of a video encoding / decoding method provided in an embodiment of the present invention;

[0068] Figure 9 This is a block diagram of an exemplary encoding / decoding device provided in an embodiment of the present invention.

[0069] In the following text, the same reference numerals denote the same features or at least functionally equivalent features, unless otherwise expressly specified. Detailed Implementation

[0070] In the following description, reference is made to the accompanying drawings, which form part of this invention, illustrating by way of description specific aspects of embodiments of the invention or specific aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description is not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0071] For example, it should be understood that the content disclosed with reference to the described method also applies to the corresponding device or system used to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (e.g., one unit performs one or more steps, or each of a plurality of units performs one or more of a plurality of steps), even if such one or more units are not explicitly described or illustrated in the accompanying drawings. On the other hand, for example, if a specific apparatus is described based on one or more units, such as functional units, the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or each of a plurality of steps performs the function of one or more of a plurality of units), even if such one or more steps are not explicitly described or illustrated in the accompanying drawings. Furthermore, it should be understood that, unless specifically indicated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0072] Video decoding generally refers to the processing of image sequences that form a video or video sequence. In the field of video decoding, the terms "frame" and "picture / image" can be used synonymously. Video decoding (or generally referred to as decoding) consists of two parts: video encoding and video decoding. Video encoding is performed at the source end and typically involves processing (e.g., compressing) the raw video image to reduce the amount of data required to represent the video image (for more efficient storage and / or transmission). Video decoding is performed at the destination end and typically involves inverse processing relative to the encoder to reconstruct the video image. The "decoding" of the video image (or generally referred to as an image) involved in the embodiments should be understood as the "encoding" or "decoding" of the video image or corresponding video sequence. The encoding and decoding parts are also collectively referred to as a codec (encoding and decoding).

[0073] In lossless video decoding, the original video image can be reconstructed, meaning the reconstructed video image has the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video decoding, further compression, such as quantization, is performed to reduce the amount of data required to represent the video image. In this case, the decoder cannot completely reconstruct the video image; the quality of the reconstructed video image is lower or worse than the original video image.

[0074] Several video decoding standards belong to the "lossy hybrid video codec" group (i.e., combining spatial and temporal prediction in the sample domain with 2D transform decoding for quantization in the transform domain). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, usually requiring block-level decoding. In other words, on the encoder side, the video is typically processed (i.e., encoded) at the block (video block) level, for example, by generating prediction blocks through spatial (intra-frame) prediction and / or temporal (inter-frame) prediction; the prediction blocks are subtracted from the current block (the currently processed block / the block to be processed) to obtain residual blocks; the residual blocks are transformed and quantized in the transform domain to reduce the amount of data to be sent (compressed). On the decoder side, the encoded or compressed blocks are processed inversely relative to the encoder to reconstruct the representation of the current block. Furthermore, the processing steps of the encoder and decoder are identical, such that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing, i.e., decoding subsequent blocks.

[0075] In the following embodiment of the video decoding system 10, the video encoder 20 and the video decoder 30 are based on Figures 1A to 3 Describe it.

[0076] Figure 1AThis is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can utilize the techniques of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent examples of devices that can be used to perform various techniques according to the various examples described in this application.

[0077] like Figure 1A As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, for example, to a destination device 14; the destination device decodes the encoded image data 13.

[0078] The source device 12 includes an encoder 20, and optionally includes an image source 16, a preprocessor (or preprocessing unit) 18 (such as an image preprocessor 18), a communication interface or communication unit 22.

[0079] Image source 16 may include or be any type of image capture device, and / or any type of image generation device, or any other type of device. The image capture device may be a camera, etc., for capturing real-world images; the image generation device may be a computer graphics processor, etc., for generating computer-generated animation images; or any other type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory / storage for storing any of the aforementioned images.

[0080] In order to distinguish it from the preprocessing unit 18 and the processing performed by the preprocessing unit 18, the image or image data 17 may also be referred to as the raw image or raw image data 17.

[0081] The preprocessor 18 receives (raw) image data 17 and preprocesses the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include, for example, trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 is an optional component.

[0082] Video encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for detailed description).

[0083] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send the encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0084] Destination device 14 includes decoder 30 (e.g., video decoder 30), and optionally may include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32) and display device 34.

[0085] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other arbitrary processed version) directly from the source device 12 or from any other source device such as a storage device, for example, a storage device for storing encoded image data, and to provide the encoded image data 21 to the decoder 30.

[0086] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0087] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing for transmission on a communication link or communication network.

[0088] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation process to obtain encoded image data 21.

[0089] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1A The communication channel 13, indicated by the arrow pointing from the source device 12 to the destination device 14, is a one-way communication interface, or configured as a two-way communication interface, and can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission).

[0090] Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (hereinafter referred to as...) Figure 3 or Figure 5 (Describe in detail)

[0091] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 (also known as reconstructed image data), for example, the decoded image 31, to obtain post-processed image data 33, for example, the post-processed image 33. The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.

[0092] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 to display the image to a user or viewer. The display device 34 can be or includes any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCOS), a digital light processor (DLP), or any other type of display.

[0093] although Figure 1A The illustration shows source device 12 and destination device 14 as independent devices, but device embodiments may also include both source device 12 and destination device 14, or both source device 12 and destination device 14's functionality, i.e., simultaneously including source device 12 or its corresponding functionality and destination device 14 or its corresponding functionality. In such embodiments, the functionality of source device 12 or its corresponding functionality and 14 or its corresponding functionality can be implemented using the same hardware and / or software, separate hardware and / or software, or any combination thereof.

[0094] According to the description, Figure 1A The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0095] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, can be transmitted via, for example, Figure 1BThe processing circuitry shown can be implemented using one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. Encoder 20 can be implemented using processing circuitry 46 to include reference... Figure 2 The encoder 20 describes various modules and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 The decoder 30 describes various modules and / or any other decoder system or subsystem described herein. Processing circuitry can be used to perform the various operations described below. Figure 5 As shown, if part of the technology is implemented in software, the device can store the instructions for that software in a suitable non-transitory computer-readable storage medium, and execute those instructions in hardware via one or more processors, thereby performing the technology of the present invention. The video encoder 20 or video decoder 30 can be integrated into a single device as part of a combined codec, such as... Figure 1B As shown.

[0096] Figure 1B The video decoding system 40 shown includes processing circuitry that implements the video encoder 20 and the video decoder 30. Furthermore, one or more imaging devices 41 such as a camera for capturing real-world images, an antenna 42, one or more memories 44, one or more processors 43, and / or a display device 45 such as the aforementioned display device 34 may be provided as part of the video decoding system 40.

[0097] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 can be used for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0098] In some cases, Figure 1AThe video decoding system 10 shown is merely an example, and the technology of this application can be applied to video decoding systems (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other examples, data may be retrieved from local memory, streamed over a network, etc. The video encoding device may encode data and store it in memory, and / or the video decoding device may retrieve and decode data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve and decode data from memory.

[0099] For ease of description, embodiments of the present invention are described, for example, with reference to the high-efficiency video coding (HEVC), versatile video coding (VVC) reference software, and next-generation video coding standards developed by the joint collaboration team on video coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0100] Encoders and Encoding Methods

[0101] Figure 2 This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. Figure 2 In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filtering unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown in the figure). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0102] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 can form the forward signal path of the encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 can form the backward signal path of the video encoder 20. The backward signal path of the video encoder 20 corresponds to the decoder (see [link to decoder]). Figure 3 The signal path of the video decoder 30 in the video encoder 20. The inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer 230, inter-frame prediction unit 244 and intra-frame prediction unit 254 also constitute the "built-in decoder" of the video encoder 20.

[0103] Image and image segmentation (images and patches)

[0104] Encoder 20 can be used to receive image 17 (or image data 17) via input terminal 201, etc. Image 17 can be an image in a series of images that make up a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be decoded (especially in video decoding, to distinguish the current image from other images, other images are, for example, previously encoded images and / or previously decoded images in the same video sequence (i.e., a video sequence that also includes the current image).

[0105] A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. Samples in the array can also be called pixels (short for image elements). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the image size and / or resolution. Color is typically represented using three color components; that is, an image can be represented as or comprise three arrays of samples. In RBG format or color space, an image includes corresponding red, green, and blue sample arrays. However, in video decoding, each pixel is typically represented in a luminance and chrominance format or color space, such as YCbCr format, which includes a luminance component represented by Y (sometimes also L) and two chrominance components represented by Cb and Cr. The luminance component Y represents brightness or grayscale intensity (e.g., both are the same in grayscale images), while the two chrominance components Cb and Cr represent chrominance or color information components. Therefore, an image in YCbCr format consists of a luminance sample array composed of luminance sample values ​​(Y) and two chrominance sample arrays composed of chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa. This process is also called color transformation or conversion. If the image is monochrome, it may only include the luminance sample array. Therefore, an image can be, for example, a monochrome luminance sample array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance sample array and two corresponding chrominance sample arrays.

[0106] In an embodiment of the video encoder 20, the video encoder 20 may include an image segmentation unit ( Figure 2 (Not shown in the image) is used to segment the image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be called root blocks or macroblocks (H.264 / AVC standard), coding tree blocks (CTB), or coding tree units (CTU) (H.265 / HEVC and VVC standards). The image segmentation unit can be used to apply the same block size to all images in a video sequence and to the corresponding grid with a defined block size, or to vary the block size between images, subsets, or groups of images, and to segment the blocks of each image.

[0107] In other embodiments, the video encoder may be used to directly receive image blocks 203 of the image 17, for example, one, several, or all of the image blocks that make up the image 17. Image block 203 may also be referred to as the current image block or the image block to be decoded.

[0108] Similar to image 17, image block 203 is also, or can be considered as, a two-dimensional array or matrix composed of samples with intensity values ​​(sample values), but the size of image block 203 is smaller than that of image 17. In other words, image block 203 may include a single sample array (e.g., if image 17 is a black and white image, image block 203 includes a luma array; if image 17 is a color image, image block 203 includes a luma array or a chroma array), or three sample arrays (e.g., if image 17 is a color image, image block 203 includes a luma array and two chroma arrays), or any other number and / or type of array determined by the color format used. The number of samples in the horizontal and vertical directions (or axes) of block 203 determines the size of block 203. Therefore, a block can be an M×N (M columns × N rows) sample array, or an M×N transform coefficient array, etc.

[0109] exist Figure 2 In the embodiment of the video encoder 20 shown, the video encoder 20 can be used to encode the image 17 block by block, for example, to encode and predict each image block 203.

[0110] Figure 2 The video encoder 20 shown can also be used to segment and / or encode images using slices (also known as video slices). An image can be segmented into one or more slices (typically non-overlapping) or encoded using one or more slices (typically non-overlapping), each slice may include one or more blocks (e.g., CTUs). A slice may include a set of rectangular tiles, or it may be a set of CTU rows in raster order within a single tile.

[0111] Figure 2 The embodiment of the video encoder 20 shown can also be used to segment and / or encode an image using chunk groups (also called video chunk groups) and / or chunks (also called video chunks), wherein one or more chunk groups (typically non-overlapping) can be used to segment or encode an image, each chunk group may include one or more blocks (e.g., CTUs) or one or more chunks, wherein each chunk may be a shape such as a rectangle and may include one or more blocks (e.g., CTUs), such as complete or partial blocks.

[0112] Residual calculation

[0113] The residual calculation unit 204 can be used to calculate the residual block 205 (also called residual 205) based on the image block 203 and the prediction block 265 (the prediction block 265 will be described in detail below) in the following manner, for example, by subtracting the sample value of the prediction block 265 from the sample value of the image block 203 one sample (one pixel) at a time to obtain the residual block 205 in the sample domain.

[0114] Transformation

[0115] The transform processing unit 206 can be used to transform the sample values ​​of the residual block 205, such as by discrete cosine transform (DCT) or discrete sine transform (DST), to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 can also be called transform residual coefficients and represent the residual block 205 in the transform domain.

[0116] Transform processing unit 206 can be used to apply integer approximations of DCT / DST, such as those specified for H.265 / HEVC. Compared to orthogonal DCT transforms, this integer approximation method is typically scaled by a certain factor. To maintain the norm of the residual block after forward and inverse transform processing, other scaling factors are used as part of the transform process. The scaling factor is usually selected based on certain constraints, such as whether the scaling factor is a power of 2 used for shift operations, the bit depth of the transform coefficients, and a balance between precision and implementation cost. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side via inverse transform processing unit 212, etc. (and for the corresponding inverse transform on the video decoder 30 side via inverse transform processing unit 312, etc.), and correspondingly, a corresponding scaling factor can be specified for the forward transform on the encoder 20 side via transform processing unit 206, etc.

[0117] In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the transform processing unit 206) can be used, for example, to directly output or encode or compress one or more transform parameters of one transform type, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0118] Quantification

[0119] Quantization unit 208 is used to quantize the transform coefficients 207 by performing scalar quantization or vector quantization to obtain quantization coefficients 209. Quantization coefficients 209 can also be called quantization transform coefficients 209 or quantization residual coefficients 209.

[0120] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling can be used to achieve finer or coarser quantization. A smaller quantization step size results in more precise quantization; a larger quantization step size results in coarser quantization. An appropriate quantization step size can be represented by the quantization parameter (QP). For example, the quantization parameter can be an index applicable to a predefined set of appropriate quantization step sizes. For example, a small quantization parameter can correspond to precise quantization (small quantization step size), while a large quantization parameter can correspond to coarse quantization (large quantization step size), and vice versa. Quantization operations can include division by the quantization step size, while corresponding dequantization or inverse dequantization operations performed by the dequantization unit 210, etc., can include multiplication by the quantization step size. In some embodiments, the quantization step size can be determined using the quantization parameter according to standards such as HEVC. Typically, the quantization step size can be calculated using a fixed-point approximation of the equations involving division and the quantization parameters. A scaling factor can be introduced into quantization and dequantization to recover the norm of the residual block, as this norm can be modified because scaling is used in the fixed-point approximation of the equations for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated to the decoder by the encoder via a bitstream or similar signal. Quantization is a lossy operation, and the loss increases with the quantization step size.

[0121] In an embodiment of the video encoder 20, the video encoder 20 (correspondingly, the quantization unit 208) can be used, for example, to directly output or to output a quantization parameter (QP) after being encoded by the entropy encoding unit 270, so that the video decoder 30 can receive and use the quantization parameter for decoding.

[0122] Inverse Quantization

[0123] The dequantization unit 210 performs the opposite of the quantization performed by the quantization unit 208 on the quantization coefficients to obtain the dequantization coefficients 211. For example, it performs the opposite of the quantization scheme performed by the quantization unit 208 using the same quantization step size as the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, which correspond to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly equal to the transform coefficients.

[0124] Inverse Transformation

[0125] The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the pixel domain. The reconstructed residual block 213 can also be referred to as the transform block 213.

[0126] reconstruction

[0127] The reconstruction unit 214 (e.g., adder or summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265. For example, the sample values ​​of the reconstruction residual block 213 and the sample values ​​of the prediction block 265 are added one by one to obtain the reconstruction block 215 in the sample domain.

[0128] Filtering

[0129] Loop filtering unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or is typically used to filter the reconstructed samples to obtain filtered sample values. The loop filtering unit can be used to smooth pixel transitions or improve video quality. Loop filtering unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening or smoothing filter, a co-filter, or any combination thereof. Although loop filtering unit 220 in... Figure 2 The loop filter 220 is shown as an in-loop filter, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filtered block 221 can also be called the filtered reconstruction block 221.

[0130] In an embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., sample adaptive offset information), for example, directly output or after being encoded by the entropy coding unit 270, so that (e.g.) the decoder 30 can receive and use the same loop filter parameters or the corresponding loop filter for decoding.

[0131] Decoding image buffer

[0132] The decoded picture buffer (DPB) 230 can be a memory that stores a reference image or typically stores reference image data, for use when the video encoder 20 encodes video data. The DPB 230 can be constructed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, of the same current image or different images, such as previously reconstructed images, and can provide the complete previously reconstructed image, i.e., the decoded image (and the corresponding reference blocks and pixels), and / or a partially reconstructed current image (and the corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 can also be used to store one or more unfiltered reconstruction blocks 215, or typically store unfiltered reconstruction samples, such as reconstruction blocks 215 that have not been filtered by the loop filter unit 220, or reconstruction blocks or reconstruction samples that have not undergone any other processing.

[0133] Pattern selection (segmentation and prediction)

[0134] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is used to receive or acquire raw image data such as raw block 203 (current block 203 of the current image 17) and reconstructed image data from the decoded image buffer 230 or other buffers (e.g., line buffers, not shown in the figure). The reconstructed image data is used as reference image data for predictions such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0135] The mode selection unit 260 can be used to determine or select a segmentation mode for the current block prediction mode (including non-segmentation mode) and prediction mode (e.g., intra-frame or inter-frame prediction mode), and generate a corresponding prediction block 265, which is used for the calculation of residual block 205 and the reconstruction of reconstruction block 215.

[0136] In embodiments of the mode selection unit 260, the mode selection unit 260 can be used to select a segmentation and prediction mode (e.g., from modes supported or available by the mode selection unit 260). The segmentation and prediction modes provide an optimal match, i.e., minimum residual (minimum residual implies better compression performance for transmission or storage), or minimum indication overhead (minimum indication overhead implies better compression performance for transmission or storage), or a balance between the two. The mode selection unit 260 can be used to determine the segmentation and prediction modes based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum rate distortion. In this document, the terms "optimal," "minimum," "best," etc., do not necessarily refer to "best," "minimum," "best," etc., overall, but can also refer to situations where termination or selection criteria are met. For example, a value exceeding or falling below a threshold or other limitation may lead to a "suboptimal choice," but will reduce complexity and processing time.

[0137] In other words, the segmentation unit 262 can be used, for example, to segment block 203 into smaller block portions or sub-blocks (which again form blocks) by iteratively using quad-tree-partitioning (QT), binary partitioning (BT), or triple-tree-partitioning (TT), or any combination thereof, and to perform a prediction, for example, on each of the block portions or sub-blocks, wherein the mode selection includes the selection of the tree structure of the segmented block 203, and each block portion or sub-block uses a prediction mode.

[0138] The segmentation (e.g., performed by segmentation unit 262) and prediction (e.g., performed by inter-frame prediction unit 244 and intra-frame prediction unit 254) processes performed by video encoder 20 will be described in detail below.

[0139] Segmentation Module

[0140] Segmentation unit 262 can divide (or partition) the current block 203 into smaller parts, such as square or rectangular blocks. These smaller blocks (also called sub-blocks) can be further divided into even smaller parts. This is also called tree segmentation or hierarchical tree segmentation, where the root block at root level 0 (level 0, depth 0), etc., can be recursively segmented into blocks at least two lower tree levels, such as nodes at tree level 1 (level 1, depth 1). These blocks can then be further segmented into blocks at least two lower tree levels, such as tree level 2 (level 2, depth 2), etc., until the segmentation ends due to the fulfillment of a termination criterion, such as reaching the maximum tree depth or the minimum block size. Blocks that are not further segmented are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary-tree (BT), a tree divided into three parts is called a ternary-tree (TT), and a tree divided into four parts is called a quad-tree (QT).

[0141] As stated above, the term "block" as used herein can be a portion of an image, particularly a square or rectangular portion. For example, referring to HEVC and VVC, a block can be or may correspond to a coding tree unit (CTU), coding unit (CU), prediction unit (PU), and transform unit (TU), and / or correspond to a corresponding block, such as a coding tree block (CTB), coding block (CB), transform block (TB), or prediction block (PB).

[0142] For example, a coding tree unit (CTU) can be or may include a CTB of luminance samples in an image with a 3-sample array, two corresponding CTBs of chrominance samples in the same image, or a CTB of samples in a black-and-white image or an image decoded using 3 independent color planes and syntax structures. These syntax structures are used to decode the samples. Correspondingly, a coding tree block (CTB) can be an N×N sample block, where N can be set to a value to divide the components into CTBs; this is a type of partitioning. A coding unit (CU) can be or may include a coding block of luminance samples in an image with a 3-sample array, two corresponding coding blocks of chrominance samples in the same image, or a coding block of samples in a black-and-white image or an image decoded using 3 separate color planes and syntax structures. These syntax structures are used to decode the samples. Correspondingly, a coding block (CB) can be an M×N sample block, where M and N can be set to certain values ​​to divide the CTB into coding blocks; this is a type of partitioning.

[0143] In some embodiments, such as HEVC, a coding tree unit (CTU) can be divided into multiple CUs using a quadtree structure represented as a coding tree. At the CU level, a decision is made as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to decode the image region. Each CU can also be divided into one, two, or four PUs depending on the PU partitioning type. Within a PU, the same prediction process is performed, and relevant information is sent to the decoder on a PU-by-PU basis. After obtaining the residual block through the prediction process, the CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU, depending on the PU partitioning type.

[0144] In some embodiments, such as according to the latest currently developed video coding standard (called Versatile Video Coding (VVC)), quad-tree and binary-tree (QTBT) partitioning is used to segment coding blocks. In the QTBT block structure, the CU can be square or rectangular. For example, coding tree units (CTUs) are first partitioned using a quad-tree structure. Quad-tree leaf nodes are further partitioned using binary or triple tree structures. The partitioned leaf nodes are called coding units (CUs), and such partitioning is used for prediction and transform processing without any further partitioning. This means that in the QTBT coding block structure, the CUs, PUs, and TUs have the same block size. Simultaneously, multiple partitioning methods, such as ternary tree partitioning, can be combined with the QTBT block structure.

[0145] In one example, the mode selection unit 260 in the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0146] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) prediction mode set. The prediction mode set may include intra-frame prediction modes and / or inter-frame prediction modes, etc.

[0147] Intra-frame prediction

[0148] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC.

[0149] Intra-prediction unit 254 is used to generate (intra) prediction block 265 using reconstructed samples of neighboring blocks in the same current image, based on intra-prediction modes in the intra-prediction mode set.

[0150] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information representing the selected intra-prediction mode of the block) to entropy coding unit 270 in the form of syntax element 266 to be included in the encoded image data 21, so that the video decoder 30 and the like can perform operations, such as receiving and using the prediction parameters for decoding.

[0151] Inter-frame prediction

[0152] The set of (possible) inter-frame prediction modes depends on the available reference image (i.e., at least partially decoded images stored in the DPB 230, etc.) and other inter-frame prediction parameters, such as whether the entire reference image or only a portion of the reference image (e.g., the search window region around the current block) is used to search for the best matching reference block, and / or, for example, whether pixel interpolation (e.g., half-pixel interpolation and / or quarter-pixel interpolation) is performed.

[0153] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be used.

[0154] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (neither of which is in the same frame). Figure 2 (As shown in the figure). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks (e.g., reconstructed blocks of one or more other / different previously decoded images 231) for motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or, in other words, the current image and the previously decoded image 231 may be part of or constitute the image sequence that makes up the video sequence.

[0155] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same image or different images in multiple previously decoded images, and provide the offset (spatial offset) between the position (x-coordinate, y-coordinate) of the reference image (or reference image index) and / or the reference block and the position of the current block as an inter-frame prediction parameter for the motion estimation unit. This offset is also called the motion vector (MV). In some implementations, the MV can be further corrected at the encoding and decoding ends to improve prediction accuracy and ultimately improve compression performance. Furthermore, the prediction obtained from the MV or the corrected MV can be further modified or adjusted by using bi-directional optical flow (BDOF) correction. In cases where fine motion exists at the pixel level, prediction can be compensated by bi-predictive optical flow (BPOF). Decoding-side motion vector refinement (DMVR) and BPOF techniques will be described in detail below.

[0156] The motion compensation unit is used to acquire (e.g., receive) inter-frame prediction parameters and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain inter-frame prediction block 265. The motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation (performing interpolation to improve accuracy). During interpolation filtering, other pixel samples can be generated based on known pixel samples, potentially increasing the number of candidate prediction blocks available for decoding the image block. After receiving the motion vector corresponding to the PU of the current image block, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0157] The motion compensation unit can also generate syntax elements associated with blocks and video stripes for use by the video decoder 30 when decoding image blocks of the video stripe. Alternatively, as an alternative to stripes and corresponding syntax elements, chunk groups and / or chunks and their corresponding syntax elements can be generated or used.

[0158] Entropy coding

[0159] Entropy coding unit 270 is used to apply or not apply entropy coding algorithms or schemes (such as variable length coding (VLC), context adaptive VLC (CAVLC) scheme, arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to (uncompressed) quantization coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output through output terminal 272 in the form of encoded bitstream 21, etc., so that video decoder 30, etc., can receive and use these parameters for decoding. The encoded bitstream 21 can be sent to video decoder 30 or stored in memory for subsequent transmission or retrieval by video decoder 30.

[0160] Other structural variations of the video encoder 20 can be used to encode the video stream. For example, for certain blocks or frames, the non-transform encoder 20 can directly quantize the residual signal without the transform processing unit 206. In another implementation, the encoder 20 can combine the quantization unit 208 and the inverse quantization unit 210 into a single unit.

[0161] Decoder and Decoding Method

[0162] Figure 3 An exemplary video decoder 30 for implementing the technology of this application is shown. The video decoder 30 is used to receive encoded image data 21 (e.g., encoded bitstream 21) encoded by encoder 20, for example, to obtain a decoded image 331. The encoded image data or encoded bitstream includes information for decoding the encoded image data, such as data representing image blocks of encoded video stripes (and / or chunks or blocks) and associated syntax elements.

[0163] exist Figure 3 In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (such as a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as... Figure 2 The video encoder 20 shown is performing the decoding process, which is the opposite of the encoding process.

[0164] As explained with reference to encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 also constitute the "built-in decoder" of video encoder 20. Therefore, inverse quantization unit 310 can have the same function as inverse quantization unit 210, inverse transform processing unit 312 can have the same function as inverse transform processing unit 212, reconstruction unit 314 can have the same function as reconstruction unit 214, loop filter 320 can have the same function as loop filter 220, and decoded picture buffer 330 can have the same function as decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0165] Entropy Decoding

[0166] The entropy decoding unit 304 is used to parse the bitstream 21 (or generally encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and / or decoded decoding parameters 366, such as inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or any one or all of other syntax elements. The entropy decoding unit 304 can be used to perform a decoding algorithm or scheme corresponding to the encoding scheme described by the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive video strip-level and / or video block-level syntax elements. Alternatively, as an alternative to stripes and corresponding syntax elements, chunk groups and / or chunks and their corresponding syntax elements can be received or used.

[0167] Inverse Quantization

[0168] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include determining the degree of quantization using the quantization parameters calculated by the video encoder 20 for each video block in the video strip, and also determining the degree of dequantization to be performed.

[0169] Inverse Transformation

[0170] The inverse transform processing unit 312 can be used to receive the dequantized coefficients 311 (also referred to as transform coefficients 311) and transform the dequantized coefficients 311 to obtain the reconstructed residual block 313 in the sample domain. The reconstructed residual block 313 can also be referred to as transform block 313. The transformation can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform to be performed on the dequantized coefficients 311.

[0171] reconstruction

[0172] The reconstruction unit 314 (e.g., adder or summer 314) can be used to add the reconstruction residual block 313 to the prediction block 365 in such a way as, for example, by adding the sample values ​​of the reconstruction residual block 313 and the sample values ​​of the prediction block 365 to obtain the reconstruction block 315 in the sample domain.

[0173] Filtering

[0174] Loop filtering unit 320 (in or after the decoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, thereby smoothing pixel abrupt changes or otherwise improving video quality. Loop filtering unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, a cooperative filter, or any combination thereof. Although loop filtering unit 320... Figure 3 The loop filter unit 320 is shown as an in-loop filter, but in other configurations, it can be implemented as a post-loop filter.

[0175] Decoding image buffer

[0176] Subsequently, the decoded video block 321 of an image is stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image for subsequent motion compensation and / or output or display of other images.

[0177] The decoder 30 is used to output the decoded image 311 through the output unit 312, etc., to present to the user or for the user to view.

[0178] predict

[0179] The function of the inter-frame prediction unit 344 can be the same as that of the inter-frame prediction unit 244 (especially the motion compensation unit); the function of the intra-frame prediction unit 354 can be the same as that of the inter-frame prediction unit 254, and it determines the segmentation or segmentation mode and performs prediction based on the segmentation and / or prediction parameters or corresponding information received from the encoded image data 21 (e.g., parsing and / or decoding by the entropy decoding unit 304, etc.). The mode application unit 360 can be used to perform prediction (intra-frame prediction or inter-frame prediction) on each block based on the reconstructed image, reconstructed block, or corresponding sample (filtered or unfiltered) to obtain the prediction block 365.

[0180] When a video stripe is decoded into an intra-coded (I) stripe, the intra-prediction unit 354 in the mode application unit 360 generates prediction blocks 365 for the image blocks of the current video stripe based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. When a video image is decoded into an inter-coded (B or P) stripe, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 generates prediction blocks 365 for the video blocks of the current video stripe based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, prediction blocks can be generated from a reference image in a list of reference images. The video decoder 30 can construct the reference frame list, i.e., list 0 and list 1, using the default construction technique based on the reference images stored in the DPB 330. In addition to stripes (such as video stripes) or as an alternative to stripes, the same or similar processes can be applied to or embodied in chunk groups (such as video chunk groups) and / or chunks (such as video chunks). For example, I, P or B chunk groups and / or chunks can be used to decode video.

[0181] The pattern application unit 360 is used to determine the prediction information of video blocks in the current video strip by parsing motion vectors or related information and other syntax elements, and to generate prediction blocks for the current video block being decoded using the prediction information. For example, the pattern application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction), inter-frame prediction stripe type (e.g., B stripe, P stripe, or GPB stripe), construction information of one or more reference image lists of the stripe, motion vectors of each inter-frame coded video block of the stripe, inter-frame prediction state of each inter-frame decoded video block of the stripe, and other information to decode the video blocks of the current video stripe. In addition to stripes (such as video stripes) or as an alternative to stripes, the same or similar process can be applied to or embodied in block groups (such as video block groups) and / or blocks (such as video blocks). For example, I, P, or B block groups and / or blocks can be used to decode video.

[0182] Figure 3 The embodiment of the video decoder 30 shown can be used to segment and / or decode images using strips (also known as video strips), wherein one or more strips (typically non-overlapping) can be used to segment or decode an image, and each strip may include one or more blocks (e.g., CTUs).

[0183] Figure 3The embodiment of the video decoder 30 shown can be used to segment and / or decode an image using chunk groups (also called video chunk groups) and / or chunks (also called video chunks), wherein one or more chunk groups (typically non-overlapping) can be used to segment or decode an image, each chunk group may include one or more blocks (e.g., CTUs) or one or more chunks, wherein each chunk may be a shape such as a rectangle and may include one or more blocks (e.g., CTUs), such as complete or partial blocks.

[0184] Other forms of video decoder 30 can be used to decode the encoded image data 21. For example, decoder 30 can generate an output video stream without the loop filter unit 320. For example, non-transform-based decoder 30 can directly dequantize the residual signals of certain blocks or frames without the inverse transform processing unit 312. In another implementation, video decoder 30 may have a dequantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0185] It should be understood that in encoder 20 and decoder 30, the processing results of the current stage can be further processed and then output to the next stage. For example, after interpolation filtering, motion vector derivation, or loop filtering, operations such as clipping or shifting can be further performed on the processing results of interpolation filtering, motion vector derivation, or loop filtering.

[0186] It should be noted that further operations can be performed on the derived motion vectors of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, and temporal motion vectors). For example, the value of the motion vector can be restricted to a predefined range based on its representation bit. If the representation bit of the motion vector is bitDepth, the value range of the motion vector is –2^(bitDepth–1) to 2^(bitDepth–1)–1, where the “^” symbol represents a power. For example, if bitDepth is set to 16, the range is –32768 to 32767; if bitDepth is set to 18, the range is –131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted so that the maximum difference between the integer parts of the MVs of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. This section provides two methods for limiting motion vectors based on bitDepth.

[0187] Method 1: Remove the most significant bit (MSB) of the overflow using the following operation.

[0188] ux = (mvx + 2) bitDepth )%2 bitDepth (1)

[0189] mvx = (ux >= 2) bitDepth-1 )? (ux-2 bitDepth ):ux (2)

[0190] uy=(mvy+2 bitDepth )%2 bitDepth (3)

[0191] mvy = (uy >= 2) bitDepth-1 )? (uy-2 bitDepth ):uy (4)

[0192] Where mvx is the horizontal component of the motion vector of the image block or sub-block, mvy is the vertical component of the motion vector of the image block or sub-block, and ux and uy represent intermediate values;

[0193] For example, if the value of mvx is -32769, then the value obtained after using formulas (1) and (2) is 32767. In computer systems, decimal numbers are stored in two's complement form. The two's complement of -32769 is 1,0111,1111,1111,1111 (17 bits). If we discard the MSB, then the resulting two's complement is 0111,1111,1111,1111 (decimal 32767), which is the same as the output obtained after using formulas (1) and (2).

[0194] ux = (mvpx + mvdx + 2) bitDepth )%2 bitDepth (5)

[0195] mvx = (ux >= 2) bitDepth-1 )? (ux-2 bitDepth :ux (6)

[0196] uy = (mvpy + mvdy + 2) bitDepth )%2 bitDepth (7)

[0197] mvy = (uy >= 2) bitDepth-1 )? (uy-2 bitDepth ):uy (8)

[0198] These operations can be applied during the summation of the motion vector prediction value mvp and the motion vector difference mvd, as shown in equations (5) to (8).

[0199] Method 2: Limit the value to remove overflowing MSB

[0200] vx = Clip3(-2) bitDepth-1 ,2 bitDepth-1 --1,vx)

[0201] vy = Clip3(-2) bitDepth-1 ,2 bitDepth-1 -1,vy)

[0202] Where vx is the horizontal component of the motion vector of the image block or sub-block; vy is the vertical component of the motion vector of the image block or sub-block; x, y, and z correspond to the three input values ​​of the MV correction process, and the function Clip3 is defined as follows:

[0203]

[0204] Figure 4 This is a schematic diagram of a video decoding device 400 provided for an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, for example... Figure 1A The video decoder 30 in the text can also be an encoder, for example... Figure 1A The video encoder 20 in the middle.

[0205] The video decoding device 400 includes: an input port 410 and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450, serving as input or output points for optical or electrical signals.

[0206] Processor 430 is implemented in both hardware and software. Processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470. Decoding module 470 implements the disclosed embodiments described above. For example, decoding module 470 performs, processes, prepares, or provides various decoding operations. Therefore, including decoding module 470 provides a substantial improvement to the functionality of video decoding device 400 and affects the transitions of video decoding device 400 to different states. Alternatively, decoding module 470 can be implemented with instructions stored in memory 460 and executed by processor 430.

[0207] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data device for programs selected for execution, as well as instructions and data read during program execution. For example, memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0208] Figure 5 A simplified block diagram of the apparatus 500 provided for an exemplary embodiment, wherein the apparatus 500 can be used as Figure 1A Either or both of the source device 12 and the destination device 14.

[0209] The processor 502 in device 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or multiple devices capable of manipulating or processing information that is currently present or will be developed later. While the disclosed implementation may be implemented using a single processor such as processor 502 as shown, using more than one processor can improve speed and efficiency.

[0210] In one implementation, the memory 504 in device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of device may be used as device 504. Memory 504 may include code and data 506 accessed by processor 502 via bus 512. Memory 504 may also include an operating system 508 and an application program 510, which includes at least one program that allows processor 502 to perform the methods described herein. For example, application program 510 may include applications 1 to N, which include a video decoding application that performs the methods described herein.

[0211] The device 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0212] Although described herein as a single bus, bus 512 of device 500 may consist of multiple buses. Furthermore, auxiliary memory 514 (not shown) may be directly coupled to other components of device 500 or accessible via a network, and may comprise a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 500 can have a wide variety of configurations.

[0213] Motion Vector Correction (MVR)

[0214] At least a portion of the motion vector is typically determined at the encoder and indicated to the decoder in the encoded bitstream. However, the motion vector can also be corrected in the decoder (and in the encoder) starting from the initial motion vector represented in the bitstream. In this case, for example, the accuracy of the initial motion vector can be improved using the similarity between pixel patches or blocks of already decoded pixels pointed to by the initial motion vector. The advantage of this motion correction is that it reduces indication overhead: the accuracy of the initial motion vector is improved in the same way in both the encoder and decoder, thus eliminating the need for additional indication for correction.

[0215] It's important to note that the initial motion vector before correction might not be the optimal motion vector for best prediction. The initial motion vector is indicated in the bitstream, and therefore may not represent it with very high accuracy (increasing the bit rate). Therefore, a motion vector correction process is used to improve the initial motion vector. For example, the initial motion vector could be the motion vector used when predicting neighboring blocks of the current block. In this case, it's simply indicated in the bitstream to represent the motion vector of the neighboring block used by the current block. This prediction mechanism can significantly reduce the number of bits required to represent the initial motion vector. However, because the motion vectors of two neighboring blocks are typically expected to be different, the accuracy of the initial motion vector will be lower.

[0216] Further refining the motion vectors derived at the encoder and provided (indicating) in the bitstream can further improve the accuracy of the motion vectors without increasing indication overhead. Motion vector refining can be performed in the decoder without the encoder's assistance. The encoder in the decoder loop can employ the same refining to obtain the corresponding refined motion vectors available in the decoder. Refining the current block being reconstructed in the current image is performed by determining a template for the reconstructed samples, determining a search space for initial motion information around the current block, and finding the best-matching reference image portion within the search space. The best-matching portion determines the refined motion vectors for the current block, which are then used to obtain inter-frame prediction samples for the current block (i.e., the current block being reconstructed).

[0217] Motion vector correction is Figure 2 Inter-frame prediction unit 244 and Figure 3 It is part of the inter-frame prediction unit 344.

[0218] Motion vector correction can be performed according to the following steps:

[0219] Typically, the initial motion vector can be determined from the representation in the bitstream. For example, an index can be indicated in the bitstream, representing the position in the candidate motion vector list. In another example, a motion vector prediction index and a motion vector difference can be indicated in the bitstream. The motion vector determined from the representation in the bitstream is defined as the initial motion vector. In the case of bidirectional prediction, the inter-frame prediction of the current block is obtained based on a weighted combination of prediction sample blocks, which are determined based on two motion vectors, with the initial motion vector of the first reference image in list L0 represented as MV0 and the initial motion vector of the second reference image in list L1 represented as MV1.

[0220] Using the initial motion vectors, determine corrected candidate motion vector (MV) pairs. At least two corrected candidate pairs need to be determined. Typically, corrected candidate motion vector pairs are determined based on the initial motion vector pair (MV0, MV1). Alternatively, candidate MV pairs are determined by adding a small motion vector difference to MV0 and MV1. For example, candidate MV pairs may include the following:

[0221] ·(MV0,MV1)

[0222] ·(MV0+(0,1),MV1+(0,–1))

[0223] ·(MV0+(1,0),MV1+(–1,0))

[0224] ·(MV0+(0,–1),MV1+(0,1))

[0225] ·(MV0+(–1,0),MV1+(1,0))

[0226] ·……

[0227] Where (1,–1) represents an integer pixel displacement of 1 in the horizontal (or x) direction and an integer pixel displacement of –1 in the vertical (or y) direction.

[0228] It should be noted that the above list of candidate MV pairs is merely illustrative and the present invention is not limited to a specific candidate list.

[0229] The search space for the motion vector correction process is formed by modifying the candidate motion vector (MV).

[0230] In the bidirectional prediction of the current block, the two predicted blocks obtained using the corresponding first motion vector of list L0 and the second motion vector of list L1 are combined into a single predicted signal or block. This fits the original signal better than unidirectional prediction, resulting in less residual information and more efficient compression.

[0231] In motion vector correction, two predicted blocks obtained using the corresponding first and second motion vectors of each candidate MV pair are compared based on a similarity metric. Typically, the candidate MV pair with the highest similarity is selected as the corrected motion vector. The corrected motion vectors of the first reference image in list L0 and the second reference image in list L1 are denoted as MV0' and MV1', respectively. In other words, predictions of the motion vectors in list L0 and list L1 corresponding to the candidate motion vector pairs are obtained and then compared based on a similarity metric. The candidate motion vector pair with the highest correlation similarity is selected as the corrected MV pair.

[0232] Typically, the output of the correction process is the corrected MV. The corrected MV can be the same as or different from the initial MV, depending on which candidate MV pair achieves the highest similarity. Candidate MV pairs formed from the initial MV are also considered MV pair candidates. In other words, if the highest candidate MV pair achieving the highest similarity is formed from the initial MV, then the corrected MV is the same as the initial MV.

[0233] Another approach is to choose the location that minimizes the dissimilarity metric rather than the location that maximizes the similarity metric. Dissimilarity comparison metrics can include SAD (sum of absolute differences), MRSAD (mean removed sum of absolute differences), SSE (sum of squared errors), etc. The SAD between two predicted blocks can be obtained from the candidate MV pair (CMV0, CMV1), and SAD can be calculated as follows:

[0234]

[0235] Where nCbH and nCbW are the height and width of the predicted block, the function abs(a) represents the absolute value of parameter a, and predSAmplesL0 and predSAmplesL1 are the predicted block samples obtained based on the candidate MV pair, which is represented as (CMV0, CMV1).

[0236] Alternatively, the dissimilarity comparison metric can be obtained by evaluating only a subset of samples in the prediction block, thus reducing the number of computations. Below is an example where sample rows are optionally included in the SAD calculation (evaluated every other row).

[0237]

[0238] The JVET (ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11) document JVET-M1001-v3, "Universal Video Coding (Draft 4)," illustrates an example of motion vector correction, which can be accessed publicly at http: / / phenix.it-sudparis.eu / jvet / . The section "8.4.3 Decoding the Motion Vector Correction Process" in the document provides an example of motion vector correction.

[0239] In some embodiments, to reduce the internal memory requirements for correction, motion vector correction can be performed independently on luma sample blocks obtained by segmenting decoded sample blocks exceeding a predetermined width or height into sample sub-blocks smaller than or equal to the predetermined width and height in the luma sample. The corrected motion vector (MV) pairs for each sub-block within the segmented decoded block can be different. Then, inter-frame prediction of luma and chroma is performed for each sub-block using the corrected MV pairs of the sub-blocks.

[0240] Each MV in the initial MV pair can have fractional pixel precision. In other words, MV represents the displacement between the current sample block and the resampled reference region, and this displacement can point to fractional positions in the horizontal and vertical directions of the integer grid of the reconstructed reference sample. Typically, two-dimensional interpolation is performed on the integer grid values ​​of the reconstructed reference sample to obtain the sample values ​​at the fractional sample offset positions. The process of obtaining predicted samples from the reconstructed reference image using candidate MV pairs can be performed by one of the following methods:

[0241] • Round the fractional part of the initial MV pair to the nearest integer position and obtain the integer grid value of the reconstructed reference image.

[0242] • Performing a 2-tap (e.g., bilinear) interpolation can separate bilinear interpolation to obtain predicted sample values ​​at fractional pixel precision represented by the initial MV pair.

[0243] • Performing more taps (e.g., 8 taps or 6 taps) can separate the interpolation to obtain the predicted sample values ​​at fractional pixel precision represented by the initial MV pairs.

[0244] While candidate MV pairs can have arbitrary offsets relative to the initial MV pair, in some embodiments, to simplify the search, candidate MV pairs are selected with integer pixel distances or integer displacements relative to the initial MV pair. Such candidate MV pairs with integer pixel distances relative to the initial MV pair can also be used in the first stage of the motion vector refinement (MVR) process, i.e., the integer distance refinement stage of MVR. In this case, predicted samples spanning all candidate MV pairs can be obtained by performing predictions on sample blocks surrounding the initial MV pair, thus covering all refinement locations around the initial MV pair.

[0245] In some embodiments, once the dissimilarity cost values ​​at all candidate MV pairs at integer distances from the initial MV pair have been evaluated, additional candidate MV pairs at subpixel distance offsets from the optimal cost value location are added. This second stage can serve as a fractional distance correction stage in the MVR process. Predicted samples for each of these locations are obtained using one of the methods described above, and the dissimilarity costs are evaluated and compared to obtain the location with the lowest dissimilarity cost. In some other embodiments, to avoid this computationally expensive prediction process for every subpixel distance location around the lowest cost integer distance location, the evaluated integer distance cost values ​​are saved, and the parameter error surface is fitted near the optimal integer distance location. The minimum value of this error surface is then analyzed and used as the location with the minimum dissimilarity. In this case, the dissimilarity cost value is derived from the calculated integer distance cost value.

[0246] Motion vector corrections can be applied to a given decoded sample block based on certain decoding properties. Some examples of such decoding properties might be:

[0247] • The difference between the number of images (when sampled at a uniform frame rate) or the difference in the image order count (POC) between the current image and the two reference images used for bidirectional prediction of the decoding sample block is equal and falls on the opposite side of the current image.

[0248] • The initial dissimilarity between the two predicted blocks obtained using the initial MV pair is less than a predetermined per-sample threshold.

[0249] Bidirectional Predictive Optical Flow Correction (BPOF)

[0250] Bi-directional optical flow (BPOF) correction is a process that improves the bi-directional prediction accuracy of a block. Besides indicating bi-directional prediction, it does not require explicit indication in the bitstream. Bi-directional optical flow correction is... Figure 2 Inter-frame prediction unit 244 and Figure 3 Part of 344.

[0251] In bidirectional prediction, two inter-frame predictions are obtained based on two motion vectors, and then these two inter-frame predictions are combined using a weighted average method. The combined prediction cancels out the quantization noise of the two reference patches, thus reducing residual energy and achieving higher decoding efficiency than unidirectional prediction. The weighted combination in bidirectional prediction can be performed using the following equation:

[0252] Two-way forecast = forecast 1 × W1 + forecast 2 × W2 + K.

[0253] Where W1 and W2 are weighting factors, which can be indicated in the bitstream or predefined at the encoding or decoding end. K is an additive factor, which can also be indicated in the bitstream or predefined at the encoding or decoding end. For example, bidirectional prediction can be obtained using the following formula:

[0254] Two-way prediction = (prediction 1 + prediction 2) / 2

[0255] In this case, W1 and W2 are set to 1 / 2, and K is set to 0.

[0256] The purpose of optical flow correction is to improve the accuracy of bidirectional prediction. Optical flow is the apparent motion pattern of an image object between two consecutive frames, caused by the motion of the object or the camera. The optical flow correction process improves the accuracy of bidirectional prediction by using the optical flow equation (solving the optical flow equation).

[0257] In one example, pixel I(x, y, t) is located in the first frame (x and y correspond to spatial coordinates, and t corresponds to the temporal dimension). The object represented by the pixel moves a distance (dx, dy) in the next frame after time dt. Assuming these pixels are identical and their intensity remains constant, the optical flow equation is given by the following equation:

[0258] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0259] Where I(x, y, t) represents the intensity (sample value) of the pixel at the coordinates (x, y, t).

[0260] In another example, neglecting small displacements of higher-order terms in the Taylor series expansion, the optical flow equation can also be written as:

[0261]

[0262] in and These are the horizontal and vertical spatial gradients of the location (x, y). It is the partial temporal derivative at position (x, y), v x = dx / dt and v y =dy / dt represents the movement speed.

[0263] Optical flow correction utilizes the above principles to improve the quality of bidirectional prediction.

[0264] The implementation of optical flow correction typically includes the following steps:

[0265] 1. Calculate the sample gradient;

[0266] 2. Calculate the difference between the first and second predictions;

[0267] 3. Calculate the displacement of a pixel or group of pixels, which minimizes the error Δ between two reference patches or blocks obtained using the optical flow equation.

[0268]

[0269] Among them, I (0) Corresponding to the first predicted sample value, I (1) This is the second predicted sample value, v x and v y It is the displacement velocity calculated in the –x and –y directions. as well as Let τi and τ0 be the gradients in the –x and –y directions, respectively, for the first and second predictions, respectively. τ1 and τ0 represent the temporal distances from the current image to the corresponding reference image in display order, thus obtaining the first and second predictions. Some methods minimize the sum of squared errors, while others minimize the sum of absolute errors.

[0270] 4. Use the following optical flow equation:

[0271]

[0272] Among them, pred BIO This represents the modified prediction, which is the output of the optical flow correction process.

[0273] The sample gradient can be obtained using the following formula:

[0274] ·

[0275] ·

[0276] In some embodiments, the displacement of a set of pixels is estimated to simplify the displacement estimation for each pixel. In some examples, to compute improved bidirectional prediction for a 4×4 luminance sample block, the displacement is estimated using sample values ​​from an 8×8 luminance sample block centered on the 4×4 sample block.

[0277] The input to the optical flow correction process is the predicted samples of two reference images, and the output of the optical flow correction is the combined prediction (predBIO) calculated by the optical flow equation.

[0278] An example of optical flow correction is explained in section 8.5.6.4, "Bidirectional Optical Flow Prediction Process," of the JVET-N1001 document, "General Video Coding (Draft 5)." This document is publicly available at http: / / phenix.it-sudparis.eu / jvet / and is cited below for reference.

[0279] The inputs to this process include:

[0280] – The two variables nCbW and nCbH represent the width and height of the current coded block;

[0281] – Two (nCbW+2)x(nCbH+2) brightness prediction sample arrays, predSamplesL0 and predSamplesL1;

[0282] – The prediction list uses the flags predFlagL0 and predFlagL1;

[0283] –Refer to indices refIdxL0 and refIdxL1;

[0284] – The bidirectional optical flow uses the flag bdofUtilizationFlag[xIdx][yIdx], where xIdx = 0..(nCbW>>2)-1 and yIdx = 0..(nCbH>>2)-1.

[0285] The output of this process is an array of (nCbW)x(nCbH) values, pbSamples.

[0286] The derivation of variables bitDepth, shift1, shift2, shift3, shift4, offset4, and mvRefineThres is as follows:

[0287] – Set the variable bitDepth to equal BitDepth. Y That is, the bit depth of the luminance component.

[0288] – Set the variable shift1 to equal Max(6, bitDepth-6).

[0289] – Set the variable shift2 to equal Max(4, bitDepth-8).

[0290] – Set the variable shift3 to equal Max(1, bitDepth-11).

[0291] – Set the variable shift4 to Max(3, 15-bitDepth) and the variable offset4 to 1<<(shift4-1).

[0292] – Set the variable mvRefineThres to equal 1. <Max(5,bitDepth-7)。

[0293] For xIdx = 0..(nCbW>>2)-1 and yIdx = 0..(nCbH>>2)-1, the following applies:

[0294] – Set the variables xSb to equal (xIdx<<2)+1, and ySb to equal (yIdx<<2)+1.

[0295] – If bdofUtilizationFlag[xIdx][yIdx] is false, then for x = xSb-1..xSb+2 and y = ySb-1..ySb+2, the derivation of the predicted sample values ​​for the current sub-block is as follows:

[0296] pbSamples[x][y] = Clip3(0,(2) bitDepth )-1,(predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1])>>shift4)

[0297] Otherwise (bdofUtilizationFlag[xIdx][yIdx] is true (TRUE)), the derivation of the predicted sample value of the current sub-block is as follows:

[0298] – For x = xSb-1..xSb+4 and y = ySb-1..ySb+4, perform the following steps in sequence:

[0299] 1. Predict the position (h) of each corresponding sample location (x, y) within the sample array. x ,v y The derivation is as follows:

[0300] h x =Clip3(1,nCbW,x)

[0301] v y =Clip3(1,nCbH,y)2. The derivation of variables gradientHL0[x][y], gradientVL0[x][y], gradientHL1[x][y], and gradientVL1[x][y] is as follows:

[0302] gradientHL0[x][y]=(predSamplesL0[h x +1][v y ]-predSampleL0[hx -1][v y )>>shift1

[0303] gradientVL0[x][y] = (predSampleL0[h x [v y +1] - predSampleL0[h x [v y -1])>>shift1

[0304] gradientHL1[x][y] = (predSamplesL1[h x +1][v y - predSampleL1[h x -1][v y )>>shift1

[0305] gradientVL1[x][y] = (predSampleL1[h x [v y +1] - predSampleL1[h x [v y -1])>>shift1

[0306] 3. The variables diff[x][y], tempH[x][y] and tempV[x][y] are derived as follows:

[0307] diff[x][y] = (predSamplesL0[h x [v y >> shift2) - (predSamplesL1[h x [v y >> shift2)

[0308] tempH[x][y] = (gradientHL0[x][y] + gradientHL1[x][y]) >> shift3

[0309] tempV[x][y] = (gradientVL0[x][y] + gradientVL1[x][y]) >> shift3

[0310] – The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are derived as follows:

[0311] sGx2 = Σ i Σ j(tempH[xSb+i][ySb+j]*tempH[xSb+i][ySb+j]) with i, j = -1..4

[0312] sGy2 = Σ i Σ j (tempV[xSb+i][ySb+j]*tempV[xSb+i][ySb+j]) with i, j = -1..4

[0313] sGxGy = Σ i Σ j (tempH[xSb+i][ySb+j]*tempV[xSb+i][ySb+j]) with i, j - 1..4

[0314] sGxGy m = sGxGy >> 12

[0315] sGxGy s = sGxGy & ((1 << 12) - 1)

[0316] sGxdI = Σ i Σ j (-tempH[xSb+i][ySb+j]*diff[xSb+i][ySb+j]) with i, j = -1..4

[0317] sGydI = Σ i Σ j (-tempV[xSb+i][ySb+j]*diff[xSb+i][ySb+j]) with i, j = -1..4

[0318] – The horizontal and vertical motion offsets of the current sub - block are derived as follows:

[0319] v x = sGx2 > 0? Clip3(-mvRefineThres, mvRefineThres, - (sGxdI << 3) >> Floor(Log2(sGx2))):0

[0320] v y = sGy2 > 0? Clip3(-mvRefineThres, mvRefineThres, ((sGydI << 3)-((v x * sGxGy m ) << 12 + v x * sGxGy s ) >> 1) >> Floor(Log2(sGx2))):0

[0321] – For x = xSb-1..xSb+2 and y = ySb-1..ySb+2, the predicted sample values ​​for the current sub-block are derived as follows:

[0322] bdofOffset = Round((v x *(gradientHL1[x+1][y+1]-gradientHL0[x+1][y+1]))>>1)+Round((v y *(gradientVL1[x+1][y+1]-gradientVL0[x+1][y+1]))>>1)

[0323] (The Round() operation is defined for floating-point input. The Round() operation can be omitted here.)

[0324] pbSamples[x][y] = Clip3(0,(2) bitDepth )-1,(predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset)>>shift4)

[0325] Because the terms optical flow correction, bidirectional predictive optical flow correction, and bidirectional optical flow correction are essentially equivalent, they can be used interchangeably in this invention.

[0326] Technical problem solved by the present invention

[0327] In the current version using BPOF, coding units are divided into multiple sub-blocks to reduce the memory footprint of BPOF and allow for sub-block-level pipelines in the hardware design. Coding units with a width greater than 16 are horizontally divided, with sub-block widths of 16. Coding units with a height greater than 16 are vertically divided, with sub-block heights of 16.

[0328] All sample locations within a 4×4 block require sample gradients (as described in the background section) to calculate optical flow and for correction based on the calculated optical flow. Obtaining a 4×4 sample gradient requires extended sample values ​​in two directions, i.e., multiple references beyond the 4×4 sample values. When calculating the gradient using a [–1 0 1] filter, each reference requires an extended sample value around the 4×4 block. In other words, a 4×4 sample gradient requires a 6×6 sample region (excluding the four corner samples of the 6×6 region).

[0329] At the sub-block level, a sub-block of size M×N requires (M+2)×(N+2) sample values ​​to compute the sample gradient at the M×N sample locations. To maintain low complexity, in the current version using BPOF, the extended sample values ​​(i.e., sample locations beyond the M×N sample locations corresponding to the sub-block) are not obtained through motion-compensated interpolation of the corresponding reference samples. Instead, extended sample locations at subpixel precision are obtained in each reference image, ignoring the fractional parts in the horizontal and vertical directions, to obtain the sample values ​​for the integer grid reconstruction reference sample locations in each reference.

[0330] These extended sample values ​​are used to calculate the boundary sample gradient for each reference. Optical flow calculation for the 4×4 block uses the sample gradient values ​​and the sample values ​​at the 6×6 sample locations centered at 4×4; therefore, the sample gradient values ​​at the extended locations are set to the sample gradient values ​​of the nearest sub-block boundary samples. Once the extended sample values ​​obtained from reconstructing the reference sample location values ​​from the integer grid are used for boundary gradient calculation, the sample values ​​of the extended samples are set to the sample values ​​of the nearest sub-block boundary samples.

[0331] The aspect of removing fractional portions in the horizontal and vertical directions of the sample location to obtain integer grid reconstruction reference sample values ​​for the extended sample introduces a small compression loss compared to performing two-dimensional separable K-tap interpolation by filling samples (taking the nearest boundary sample values) as shown in JVET-M0487.

[0332] The proposed method

[0333] This invention proposes a method that retains the advantage of low complexity in obtaining extended sample values, while almost eliminating the encoding loss caused by removing the fractional parts in the horizontal and vertical directions.

[0334] Specifically, the fractional offset is used to perform rounding operations in the horizontal and vertical directions to identify the closest integer sample position in the integer sample grid of the corresponding reconstructed reference frame, rather than ignoring the horizontal and vertical fractional offsets of the extended sample positions in each reconstructed reference sample grid. Then, the extended sample is assigned the sample value at the identified position.

[0335] A bidirectional optical flow prediction method is typically provided, which is based on sample values ​​at sample locations in the extended region of the prediction block used in the bidirectional optical flow prediction of the current block. Figure 8 The following steps are implemented as shown:

[0336] Step 1310: Obtain the motion vector (MV) of the current block.

[0337] Step 1320: Obtain the sub-pixel sample positions in the extended region of the prediction block in the reference image based on the MV.

[0338] In this context, the extended region of the predicted block in the reference image is based on the MV (Modular Value). Specifically, by using the MV, a predicted block can be found, the offset of which relative to the collocated block at the corresponding position of the current block is indicated by the MV. The extended region is a region comprising at least one row / column of pixels adjacent to, surrounding, or enclosing the predicted block. The extended region can be considered as a fill region around the predicted block in the reference image, which is used, for example, for performing K-tap interpolation filtering and / or calculating 3-tap sample gradients at the block boundaries.

[0339] Step 1330: Process the obtained sub-pixel sample positions to obtain integer pixel sample positions in the reference image.

[0340] Step 1340: Perform bidirectional optical flow prediction by referring to the reference sample values ​​at integer pixel sample positions in the obtained reference image.

[0341] The following examples illustrate how to implement the method.

[0342] Example 1

[0343] Let the top-left sample of the current sub-block of an M×N sample undergoing BPOF / BDOF bidirectional prediction be located at (sb_start_x, sb_start_y). Let (mvx0, mvy0) be the 1 / 16 pixel precision MV of the current sub-block relative to the first reference frame L0, and let (mvx1, mvxy1) be the 1 / 16 pixel precision MV of the current sub-block relative to the second reference frame L1.

[0344] The top-left position of the current sub-block in L0, corresponding to a 1 / 16 pixel precision position, is obtained using ((sb_start_x<<4)+mvx0,(sb_start_y<<4)+mvy0). The top-left position of the current sub-block in L0, corresponding to a 1 / 16 pixel precision position in L1, is obtained using ((sb_start_x<<4)+mvx1,(sb_start_y<<4)+mvy1). It can be understood that different sub-pixel precisions can be used, such as half a pixel, a quarter pixel, or 1 / 8 pixel. The top-left position of the current sub-block is determined by shifting it left by 1, 2, or 3 respectively.

[0345] For the M×N position in the L0 and L1 reference frames corresponding to the sub-block, based on the sub-pixel precision (e.g., 1 / 16) and the corresponding sample position, two-dimensional separable K-tap motion compensation interpolation is performed in the corresponding reference frame and used as the predicted sample value for calculating the sample gradient during the BDOF process.

[0346] To obtain the predicted sample values ​​for the corresponding remaining extended sample positions outside the center M×N sample position within the (M+2)×(N+2) sample location block, and to use these values ​​to calculate the gradient of the M×N sub-block boundary samples, the following process needs to be followed (for general sub-pixel sample accuracy):

[0347] Let x_spel_pos be the horizontal sub-pixel sample precision position of the extended sample location, and y_spel_pos be the vertical sub-pixel sample precision position of the extended sample location. Obtain the sample value at the extended sample location as the reconstruction reference sample value at position (x_pos, y_pos), calculated as follows:

[0348] x_pos=(x_spel_pos+(1<<(shift_val–1)))>>shift_val

[0349] y_pos=(y_spel_pos+(1<<(shift_val–1)))>>shift_val

[0350] The shift_val value is selected based on the subpixel sample precision used. For subpixel sample precision of half a pixel, quarter a pixel, 1 / 8 pixel, and 1 / 16 pixel, shift_val is 1, 2, 3, or 4, respectively.

[0351] By rounding the sub-pixel positions in both the horizontal and vertical directions, the boundary gradient becomes more accurate, such as... Figure 6 and Figure 7 As shown.

[0352] Figure 6 This shows which location in the reference image the pixel value was obtained from. Figure 6 (A) illustrates that in the prior art, the fractional part of the MV pointing to the fractional position in the reference image is discarded, and only the integer part is used to obtain the pixel value. For example, for MV(4,0) and MV(9,0) in Q4 format, the position (0,0) in the reference image is used to obtain the pixel value (reference sample). Conversely, in Figure 6In (B) of the present invention, the fractional part is rounded to the nearest integer position (which is ultimately used to obtain the pixel value). For example, for MV(4,0) and MV(9,0) in Q4 format, positions (0,0) and (16,0) in the reference image are used to obtain the corresponding pixel values, thereby greatly improving the accuracy of obtaining the reference sample.

[0353] Figure 7 (B) illustrates a method for determining the integer grid reference sample position based on the sub-pixel precision (i.e., 1 / 16 pixel) sample position. It can be seen that, in both the horizontal and vertical directions, the nearest integer sample position is selected based on (half) rounding of the horizontal and vertical components of the position. Therefore, the sub-pixel precision offsets from (i–8 / 16) to (i+7 / 16) in the horizontal direction and from (j–8 / 16) to (j+7 / 16) in the vertical direction use (i,j) as the nearest integer sample position. In contrast, Figure 7 (A) in the diagram illustrates the selection of integer sample positions by truncation. Subpixel precision offsets from i to (i+15 / 16) in the horizontal direction and from j to (j+15 / 16) in the vertical direction use (i,j) as integer sample positions.

[0354] At the left boundary sample position of the M×N block, the horizontal boundary sample gradient is calculated by taking the difference between the sample value at its right position (calculated using two-dimensional separable K-tap interpolation) and the sample value at its extended sample position to its left (as described above, the nearest reconstructed reference sample value after rounding). Similarly, at the right boundary sample position of the M×N block, the horizontal boundary sample gradient is calculated by taking the difference between the sample value at its right position (as described above, the nearest reconstructed reference sample value after rounding) and the sample value at its left position (calculated using two-dimensional separable K-tap interpolation).

[0355] At the top boundary sample position of the M×N block, the vertical boundary sample gradient is calculated by taking the difference between the sample value at the position below it (calculated using two-dimensional separable K-tap interpolation) and the sample value at the extended sample position above it (as described above, the nearest reconstructed reference sample value after rounding). Similarly, at the bottom boundary sample position of the M×N block, the vertical boundary sample gradient is calculated by taking the difference between the sample value at the position below it (as described above, the nearest reconstructed reference sample value after rounding) and the sample value at the position above it (calculated using two-dimensional separable K-tap interpolation).

[0356] Example 2

[0357] In this embodiment, when BPOF / BDOF is used and the specific coding unit to which the current sub-block belongs does not use motion vector refinement (MVR), or when the specific coding unit to which the current sub-block belongs uses both MVR and BDOF, but MVR does not generate an integer displacement relative to the initial MV pair during the integer distance refinement stage of MVR, the extended sample position value is derived using the process described in Embodiment 1. In other words, when MVR and BDOF are used simultaneously, and the integer distance refinement stage of MVR generates the initial MV pair, i.e., with zero displacement, the extended sample position value is derived using the process described in Embodiment 1.

[0358] When a specific coding unit to which the current sub-block belongs uses both MVR and BDOF, but the MVR generates an integer shift relative to the initial MV pair during the integer distance correction stage of the MVR (i.e., the corrected MV pair after the integer distance correction stage is different from the initial MV pair), the sample value at the extended sample position (x_spel_pos, y_spel_pos) is obtained as the reconstruction reference sample value at position (x_pos, y_pos), calculated as follows:

[0359] x_pos = x_spel_pos >> shift_val

[0360] y_pos=y_spel_pos>>shift_val

[0361] The `shift_val` parameter is selected based on the precision of the subpixel samples used. For subpixel sample precisions of half a pixel, a quarter pixel, 1 / 8 pixel, and 1 / 16 pixel, `shift_val` is 1, 2, 3, or 4, respectively. In other words, when obtaining the sample position in this case, the fractional parts of the position in the horizontal and vertical directions are ignored.

[0362] Advantages of the proposed method:

[0363] Using the method described above for rounding sub-pixel positions in both the horizontal and vertical directions, the boundary gradients are more accurate. This improves the optical flow calculated based on the sample gradients, and also improves the correction performed using the calculated optical flow. In previous methods, fractional offsets in the horizontal and vertical directions were ignored when selecting integer sample values ​​for the extended sample positions. This method provides consistent decoding gain compared to previous methods. Simultaneously, it does not significantly increase complexity, as complexity is only increased when rounding in the horizontal and vertical directions adds offsets.

[0364] Based on this scheme, a possible embodiment combining standard VVC draft 5 is shown below.

[0365] Process of obtaining integer samples of brightness

[0366] The inputs to this process include:

[0367] – In integer sample units (xInt) L ,yInt L The brightness position is indicated by )

[0368] – in fractional sample units (xFrac) L ,yFrac L The brightness position is indicated by )

[0369] –Luminance reference sample array refPicLX L, .

[0370] The output of this process is the predicted luminance sample value, predSampleLX. L .

[0371] The variable shift is set to equal Max(2, 14-BitDepth). Y ).

[0372] The variable picW is set to equal pic_width_in_luma_samples, and the variable picH is set to equal pic_height_in_luma_samples.

[0373] The brightness position in integer sample units (xInt, yInt) is derived as follows:

[0374] xOffset = xFrac L >7? 1:0

[0375] yOffset = yFrac L >7? 1:0

[0376] xInt=Clip3(0,picW-1,sps_ref_wraparound_enabled_flag?ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,xInt L -1_+xOffset):xInt L -1+xOffset)

[0377] yInt = Clip3(0, picH-1, yInt) L -1+yOffset)

[0378] Predicted brightness sample value predSampleLX L The derivation is as follows:

[0379] predSampleLX L =refPicLX L [xInt][yInt]< <shift3

[0380] It should be noted that these exemplary changes in the VVC draft only apply to the luminance component. The chrominance component can adopt these exemplary changes in the same way, or make some changes to the characteristics of the chrominance component.

[0381] Figure 8 This is a flowchart of a video encoding / decoding method provided in an embodiment of the present invention. The method shown performs bidirectional optical flow prediction based on sample values ​​at sample positions in the extended region of the prediction block used in the current block bidirectional optical flow prediction, as follows: Step 1310: Obtain the motion vector (MV) of the current block. Step 1320: Obtain the sub-pixel sample positions in the extended region of the prediction block in the reference image based on the MV. Step 1330: Process the obtained sub-pixel sample positions (specifically, round them down) to obtain integer pixel sample positions in the reference image. Step 1340: Perform bidirectional optical flow prediction by referring to the reference sample values ​​at the obtained integer pixel sample positions in the reference image.

[0382] Figure 9 This is a block diagram of an exemplary encoding / decoding apparatus provided in an embodiment of the present invention. The encoding apparatus 20 / decoding apparatus 30 includes a prediction unit 1440 for performing bidirectional optical flow prediction of the current block; a first acquisition unit 1410 for acquiring the motion vector (MV) of the current block; and a second acquisition unit 1420 for acquiring the sub-pixel sample positions in the extended region of the predicted block in the reference image based on the MV.

[0383] The processing unit 1430 is used to process the obtained sub-pixel sample positions (specifically, to round them down) to obtain integer pixel sample positions in the reference image; wherein, the prediction unit 1440 is used to perform bidirectional optical flow prediction by referring to the reference sample values ​​at the integer pixel sample positions in the obtained reference image.

[0384] The acquisition units 1410 and 1420, processing unit 1430, and prediction unit 1440 can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions can be stored as one or more instructions or code in a computer-readable medium or transmitted via a communication medium and executed by a hardware-based processing unit. Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the various functions described herein can be provided within dedicated hardware and / or software modules for encoding and decoding, or incorporated into combined codecs. Additionally, these techniques can be fully implemented in one or more circuit or logic elements.

[0385] Some specific embodiments are outlined in the following aspects of the invention.

[0386] In a first aspect, a method is provided to obtain predicted sample values ​​based on sample positions in an extended region of a prediction block used in bidirectional optical flow prediction of a current decoded block. The method includes: obtaining the MV of the current decoded block; obtaining sub-pixel positions in a reference image based on the MV and extended sample positions surrounding the sample positions of the current decoded block in the current image; obtaining integer pixel positions in the reference image by rounding / repositioning the obtained sub-pixel sample positions to / from the nearest integer pixel position; and setting the predicted sample value of the sample position in the extended region of the prediction block as the reference sample value at the obtained integer pixel position.

[0387] Rounding / repositioning the obtained sub-pixel sample position to the nearest integer pixel position may include: rounding the obtained sub-pixel sample position to the nearest integer sample position in the horizontal and vertical directions respectively, and rounding the half-sample position between two integer sample positions to the higher integer sample position in their respective directions.

[0388] Rounding / repositioning the obtained sub-pixel sample position to the nearest integer pixel position may include: comparing the obtained sub-pixel sample position with a threshold in both the horizontal and vertical directions; if the sub-pixel sample position is equal to or greater than the threshold, the nearest higher integer sample position can be obtained as the integer pixel position; if the sub-pixel sample position is less than the threshold, the nearest lower integer sample position can be obtained as the integer pixel position.

[0389] Secondly, a method is provided to obtain predicted sample values ​​based on sample positions in an extended region of a prediction block used in bidirectional optical flow prediction of a current decoded block. The method includes: obtaining a first MV corresponding to a predicted sample in the extended region of the decoded block; obtaining a second MV by discarding a fractional portion of the first MV; obtaining a first integer pixel position of the extended region in a reference image based on the second MV; determining a fractional offset vector based on the first MV; and obtaining a second integer pixel position in the reference image based on the first integer pixel position and the fractional offset vector, wherein the second integer pixel position is set as the first integer pixel position plus a positional offset, the positional offset being determined by comparing each component of the fractional offset vector with a threshold. The method further includes: filling the predicted sample with pixel values ​​at the second integer pixel position in the reference image.

[0390] The position offset component being 1 or 0 can depend on the comparison result between the corresponding component of the fractional offset vector and the threshold, wherein the component includes an X component and a Y component.

[0391] The threshold can be K, where the component of the position offset can be set to 1 when the corresponding component of the fractional offset is equal to or greater than K; otherwise, it is set to 0. K can be equal to 8.

[0392] According to another aspect, an encoder or decoder is provided, including processing circuitry for performing any one of the methods provided in the first and second aspects described above.

[0393] According to another aspect, a computer program product is provided, comprising program code for performing any one of the methods provided in the first and second aspects described above.

[0394] According to another aspect, a decoder or encoder is provided, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein, when the processor executes the program, the decoder performs either of the methods provided in the first and second aspects described above.

[0395] Mathematical operators

[0396] The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are precisely defined here, and operations such as exponentiation and real-value division are also defined. Numbering and counting conventions typically start from zero; for example, "first" corresponds to the 0th, "second" corresponds to the 1st, and so on.

[0397] Arithmetic operators

[0398] The assignment operator is defined as follows:

[0399]

[0400] Logical operators

[0401] Logical operators are defined as follows:

[0402] x&&y Boolean logical AND operation between x and y

[0403] Boolean logical OR operation between x and y || y

[0404] ! Boolean logic "NOT"

[0405] x? y:z If x is TRUE or not equal to 0, then return the value of y; otherwise, return the value of z.

[0406] Relational operators

[0407] Relational operators are defined as follows:

[0408] >greater than

[0409] >= Greater than or equal to

[0410] <less than

[0411] <= Less than or equal to

[0412] == equals

[0413] ! = not equal to

[0414] When a relational operator is used on a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is treated as the unique value of that syntax element or variable. The value "na" is not equal to any other value.

[0415] Bitwise operators

[0416] The bitwise operators are defined as follows:

[0417] &Bitwise "AND". When operating on integer arguments, the two's complement representation of the integer value is used. When operating on binary arguments, if it has fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0418] |Bitwise "OR". When operating on integer arguments, the two's complement representation of the integer value is used. When operating on binary arguments, if it has fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0419] ^Bitwise "XOR". When operating on integer arguments, the two's complement representation of the integer value is used. When operating on binary arguments, if it has fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0420] x >> y algorithmically right-shifts the two's complement integer representation of x by y binary digits. This function is only defined for non-negative integer values of y. The result of the right-shift operation is that the number of bits shifted into the most significant bit (MSB) is equal to the value of the MSB of x before the shift operation.

[0421] x << y algorithmically left-shifts the two's complement integer representation of x by y binary digits. This function is only defined for non-negative integer values of y. The result of the left-shift operation is that the number of bits shifted into the least significant bit (LSB) is equal to the value of the LSB of x before the shift operation.

[0422] Assignment operators

[0423] The assignment operators are defined as follows:

[0424] = Assignment operator

[0425] ++ Increment, i.e., x++ is equivalent to x = x + 1; when used in an array index, it is equal to the value of the variable before the increment operation.

[0426] –– Decrement, i.e., x–– is equivalent to x = x – 1; when used in an array index, it is equal to the value of the variable before the decrement operation.

[0427] += Increment by the specified amount, e.g., x += 3 is equivalent to x = x + 3, x += (–3) is equivalent to x = x + (–3).

[0428] –= Decrement by the specified amount, e.g., x –= 3 is equivalent to x = x – 3, x –= (–3) is equivalent to x = x – (–3).

[0429] Range notation

[0430] The following notation is used to specify a range of values:

[0431] x = y..z x takes integer values ​​from y to z (inclusive), where x, y, and z are integers, and z is greater than y.

[0432] Mathematical functions

[0433] The mathematical function is defined as follows:

[0434]

[0435] Asin(x) is the trigonometric arcsine function. It operates on the parameter x, which is in the range of -1.0 to 1.0 (inclusive). The output value is in the range of -π÷2 to π÷2 (inclusive), and the unit is radians.

[0436] The Atan(x) trigonometric arctangent function operates on the parameter x, and the output value ranges from –π÷2 to π÷2 (inclusive), in radians.

[0437]

[0438] Ceil(x) represents the smallest integer value greater than or equal to x.

[0439] Clip1 Y (x) = Clip3(0, (1< <BitDepth Y )–1,x)

[0440] Clip1 C (x) = Clip3(0, (1< <BitDepth C )-1,x)

[0441]

[0442] Cos(x) is a trigonometric cosine function that operates on the parameter x, with units of radians.

[0443] Floor(x) represents the largest integer value less than or equal to x.

[0444]

[0445] Ln(x) returns the value of x in the natural logarithm (the logarithm to the base e, where e is the base constant of the natural logarithm, 2.718281828...).

[0446] Log2(x) is the logarithm of x to the base 2.

[0447] Log10(x) is the logarithm of x to the base 10.

[0448]

[0449] Round(x)=Sign(x)*Floor(Abs(x)+0.5)

[0450]

[0451] Sin(x) is a trigonometric sine function that performs calculations on the parameter x, with the unit being radians.

[0452]

[0453] Swap(x, y) = (y, x)

[0454] Tan(x) is the trigonometric tangent function that performs operations on the parameter x in radians.

[0455] Operation priority order

[0456] When parentheses are not used to explicitly indicate precedence, the following rules apply:

[0457] - Higher priority operations are evaluated before any lower priority operations.

[0458] Operations of the same priority are evaluated from left to right.

[0459] The table below shows the order of operations from highest to lowest. The higher the position in the table, the higher the priority.

[0460] If these operators are also used in the C programming language, the precedence order used in this article is the same as that used in the C programming language.

[0461] Table: Operation priority from highest (top of table) to lowest (bottom of table)

[0462]

[0463]

[0464] Textual description of logical operations

[0465] In the text, the statements in logical operations are described in mathematical form as follows:

[0466]

[0467] It can be described in the following way:

[0468] ...as follows / ...the following shall prevail:

[0469] - If condition 0, then statement 0.

[0470] Otherwise, if condition 1 is true, then statement 1.

[0471] –……

[0472] - Otherwise (a hint about the remaining conditions), then statement n

[0473] Each "if...otherwise, if...otherwise,..." statement in the text is introduced by "...as follows" or "...as follows" (followed by "if..."). The last condition "if...", "otherwise", "if...otherwise,..." always includes an "otherwise,..." statement. Statements containing "if...otherwise, if...otherwise...", "...as follows" or "...as follows" are identified by ending with "otherwise".

[0474] In the text, the statements in logical operations are described in mathematical form as follows:

[0475]

[0476] It can be described in the following way:

[0477] ...as follows / ...the following shall prevail:

[0478] - Statement 0 is true if all of the following conditions are true:

[0479] -Condition 0a

[0480] -Condition 0b

[0481] Otherwise, if one or more of the following conditions are true, then statement 1:

[0482] -Condition 1a

[0483] -Condition 1b

[0484] –……

[0485] -Otherwise, statement n

[0486] In the text, the statements in logical operations are described in mathematical form as follows:

[0487]

[0488]

[0489] It can be described in the following way:

[0490] If condition 0, then statement 0.

[0491] If condition 1 is true, then statement 1

[0492] Although embodiments of the present invention are primarily described in relation to video decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30 (correspondingly, system 10), as well as other embodiments described herein, can also be used for still image processing or decoding, i.e., processing or decoding a single image in video decoding independent of any previous or consecutive images. Generally, if image processing encoding is limited to a single image 17, inter-frame prediction units 244 (encoder) and 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0493] Embodiments of encoder 20 and decoder 30, and the functions described herein relating to encoder 20 and decoder 30, can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions can be stored as one or more instructions or code in a computer-readable medium or transmitted via a communication medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this manner, a computer-readable medium can generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium, such as a signal or carrier wave. A data storage medium can be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described herein. A computer program product may include a computer-readable medium.

[0494] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection can be referred to as computer-readable medium. For example, the definition of medium includes coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (e.g., infrared, radio, microwave, etc.) if instructions are transmitted from a website, server, or other remote source via coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (e.g., infrared, radio, microwave, etc.). However, it should be understood that computer-readable media and data media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible media. The disks and optical discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The above combinations should also be included within the scope of computer-readable media.

[0495] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the various functions described herein can be provided within dedicated hardware and / or software modules for encoding and decoding, or incorporated into combined codecs. Additionally, these techniques can be fully implemented in one or more circuit or logic elements.

[0496] The techniques of this invention can be implemented in a variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). This invention describes various components, modules, or units to emphasize functional aspects of the apparatus used to perform the disclosed techniques, but these components, modules, or units are not necessarily required to be implemented through different hardware units. Rather, as described above, the various units can be combined with suitable software and / or firmware in a codec hardware unit, or provided as a collection of interoperable hardware units including one or more processors as described above.

Claims

1. A bidirectional optical flow prediction method implemented by an encoding device or a decoding device, wherein the bidirectional optical flow prediction method is based on sample values ​​at sample positions in the extended region of the prediction block used in the current block bidirectional optical flow prediction, characterized in that, The method includes: Obtain the motion vector (MV) of the current block (1310); The sub-pixel sample positions in the extended region of the predicted block in the reference image (1320) are obtained according to the MV; Process the obtained sub-pixel sample positions (1330) to obtain integer pixel sample positions in the reference image; (1340) bidirectional optical flow prediction is performed by referring to the reference sample values ​​at integer pixel sample positions in the obtained reference image.

2. The method according to claim 1, characterized in that, The reference sample value at the integer pixel sample position in the obtained reference image is referred to as the sample value at the sub-pixel sample position in the bidirectional optical flow prediction.

3. The method according to claim 1 or 2, characterized in that, The sub-pixel sample position obtained in the processing (1330) includes: rounding the obtained sub-pixel sample position to obtain the integer pixel sample position in the reference image.

4. The method according to any one of the preceding claims, characterized in that, The sub-pixel sample positions obtained by the processing (1330) include: The obtained sub-pixel sample positions are rounded up in the horizontal and vertical directions to the nearest integer pixel sample positions in the corresponding directions, and the integer pixel sample positions in the reference image are obtained based on the nearest integer pixel sample positions in the horizontal and vertical directions.

5. The method according to any one of the preceding claims, characterized in that, The method further includes: Before performing (1340) bidirectional optical flow prediction, reference sample values ​​at integer pixel sample positions in the obtained reference image are obtained as prediction sample values ​​for bidirectional optical flow prediction, wherein Performing bidirectional optical flow prediction (1340) using reference sample values ​​includes: The predicted sample values ​​are used to perform bidirectional optical flow prediction.

6. The method according to claim 4 or 5, characterized in that, The step of rounding the obtained sub-pixel sample positions in both the horizontal and vertical directions includes: The fractional portion of the obtained sub-pixel sample position in the corresponding direction is compared with a threshold, wherein, If the fraction is equal to or greater than the threshold, the position of the nearest high-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction. If the fraction is less than the threshold, the position of the nearest low-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction.

7. The method according to claim 6, characterized in that, The nearest high-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain the truncated integer pixel sample position in the corresponding direction, and adding an offset of 1 to the truncated integer pixel sample position in the corresponding direction. The nearest low-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain the truncated integer pixel sample position in the corresponding direction, and adding an offset of 0 to the truncated integer pixel sample position in the corresponding direction.

8. The method according to claim 6 or 7, characterized in that, For a sub-pixel precision of 1 / P, the threshold is P / 2.

9. The method according to claim 8, characterized in that, For a sub-pixel precision of 1 / 16, the threshold is 8.

10. The method according to claim 4, characterized in that, The step of rounding the obtained sub-pixel sample positions in both the horizontal and vertical directions includes: Shift and reverse shift operations are applied to the components of the obtained sub-pixel sample positions in the corresponding directions.

11. The method according to claim 10, characterized in that, The integer pixel sample positions (x_pos, y_pos) in the reference image are obtained according to the following equation: x_pos=(x_spel_pos+(1<<(shift_val–1)))>>shift_val y_pos=(y_spel_pos+(1<<(shift_val–1)))>>shift_val Where (x_spel_pos, y_spel_pos) are the obtained sub-pixel sample positions, and shift_val is selected according to the precision of the sub-pixel sample used.

12. The method according to claim 11, characterized in that, For subpixel sample precision of half a pixel, quarter a pixel, 1 / 8 pixel, or 1 / 16 pixel, shift_val is 1, 2, 3, or 4, respectively.

13. The method according to any one of the preceding claims, characterized in that, The MV of the current block is the modified MV obtained by performing motion vector refinement (MVR) on the initial MV.

14. The method according to claim 13, characterized in that, The MVR includes an integer distance correction phase for the initial MV, wherein, When the corrected MV is generated as a non-zero displacement relative to the initial MV during the integer distance correction stage of the MVR, the integer pixel sample position in the reference image is obtained by rounding the obtained sub-pixel sample position.

15. The method according to claim 14, characterized in that, In the case where the corrected MV has zero displacement relative to the initial MV during the integer distance correction stage of the MVR, the integer pixel sample position in the reference image is obtained by discarding a fraction of the obtained sub-pixel sample position.

16. The method according to claim 14, characterized in that, When the integer distance correction phase of the MVR produces a zero displacement of the corrected MV relative to the initial MV, the integer pixel sample positions (x_pos, y_pos) in the reference image are obtained according to the following equation: x_pos = x_spel_pos >> shift_val y_pos=y_spel_pos>>shift_val Where (x_spel_pos, y_spel_pos) are the obtained sub-pixel sample positions, and shift_val is selected according to the precision of the sub-pixel sample used.

17. The method according to claim 16, characterized in that, For subpixel sample precision of half a pixel, quarter a pixel, 1 / 8 pixel, or 1 / 16 pixel, shift_val is 1, 2, 3, or 4, respectively.

18. The method according to any one of the preceding claims, characterized in that, The current block can be the current decoding block, the current decoding sub-block, or the current prediction block.

19. The method according to any one of the preceding claims, characterized in that, The reference sample values ​​at integer pixel sample locations in the obtained reference image are used to calculate the boundary sample gradient corresponding to the boundary of the prediction block in the reference image.

20. An encoder (20), characterized in that, Includes processing circuitry for performing the method according to any one of claims 1 to 19.

21. A decoder (30), characterized in that, Includes processing circuitry for performing the method according to any one of claims 1 to 19.

22. A computer program product, characterized in that, The program includes instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 19.

23. An encoder (20), characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to and storing instructions executable by the one or more processors, wherein, when the one or more processors execute the instructions, the encoder performs the method according to any one of claims 1 to 19.

24. A decoder (30), characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to and storing instructions executable by the one or more processors, wherein, when the one or more processors execute the instructions, the decoder performs the method according to any one of claims 1 to 19.

25. An encoder (20), characterized in that, include: Prediction unit (1440) is used to perform bidirectional optical flow prediction for the current block; The first acquisition unit (1410) is used to acquire the motion vector (MV) of the current block; The second acquisition unit (1420) is used to acquire the sub-pixel sample position in the extended region of the prediction block in the reference image according to the MV; Processing unit (1430) is used to process the obtained sub-pixel sample positions to obtain integer pixel sample positions in the reference image; wherein, The prediction unit (1440) is used to perform bidirectional optical flow prediction by referring to the reference sample value at the integer pixel sample position in the obtained reference image.

26. The encoder (20) according to claim 25, characterized in that, The process of processing the obtained sub-pixel sample position includes: rounding the obtained sub-pixel sample position to obtain the integer pixel sample position in the reference image.

27. The encoder (20) according to claim 25 or 26, characterized in that, The processing of the obtained sub-pixel sample positions includes: The obtained sub-pixel sample positions are rounded up in the horizontal and vertical directions to the nearest integer pixel sample positions in the corresponding directions, and the integer pixel sample positions in the reference image are obtained based on the nearest integer pixel sample positions in the horizontal and vertical directions.

28. The encoder (20) according to claim 27, characterized in that, The step of rounding the obtained sub-pixel sample positions in both the horizontal and vertical directions includes: The fractional portion of the obtained sub-pixel sample position in the corresponding direction is compared with a threshold, wherein, If the fraction is equal to or greater than the threshold, the position of the nearest high-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction. If the fraction is less than the threshold, the position of the nearest low-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction.

29. The encoder (20) according to claim 28, characterized in that, The nearest high-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain the truncated integer pixel sample position in the corresponding direction, and adding an offset of 1 to the truncated integer pixel sample position in the corresponding direction. The nearest low-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain the truncated integer pixel sample position in the corresponding direction, and adding an offset of 0 to the truncated integer pixel sample position in the corresponding direction.

30. A decoder (30), characterized in that, include: Prediction unit (1440) is used to perform bidirectional optical flow prediction for the current block; The first acquisition unit (1410) is used to acquire the motion vector of the current block from the bitstream of the encoded video. MV); The second acquisition unit (1420) is used to acquire the sub-pixel sample position in the extended region of the prediction block in the reference image according to the MV; Processing unit (1430) is used to process the obtained sub-pixel sample positions to obtain integer pixel sample positions in the reference image; wherein, The prediction unit (1440) is used to perform bidirectional optical flow prediction by referring to the reference sample value at the integer pixel sample position in the obtained reference image.

31. The decoder (30) according to claim 30, characterized in that, The process of processing the obtained sub-pixel sample position includes: rounding the obtained sub-pixel sample position to obtain the integer pixel sample position in the reference image.

32. The decoder (30) according to claim 30 or 31, characterized in that, The processing of the obtained sub-pixel sample positions includes: The obtained sub-pixel sample positions are rounded up in the horizontal and vertical directions to the nearest integer pixel sample positions in the corresponding directions, and the integer pixel sample positions in the reference image are obtained based on the nearest integer pixel sample positions in the horizontal and vertical directions.

33. The decoder (30) according to claim 32, characterized in that, The step of rounding the obtained sub-pixel sample positions in both the horizontal and vertical directions includes: The fractional portion of the obtained sub-pixel sample position in the corresponding direction is compared with a threshold, wherein, If the fraction is equal to or greater than the threshold, the position of the nearest high-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction. If the fraction is less than the threshold, the position of the nearest low-order integer pixel sample in the corresponding direction is obtained as the position of the closest integer pixel sample in the corresponding direction.

34. The decoder (30) according to claim 33, characterized in that, The nearest high-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain the truncated integer pixel sample position in the corresponding direction, and adding an offset of 1 to the truncated integer pixel sample position in the corresponding direction. The nearest low-order integer pixel sample position in the corresponding direction is obtained by discarding a fraction of the obtained sub-pixel sample position to obtain the truncated integer pixel sample position in the corresponding direction, and adding an offset of 0 to the truncated integer pixel sample position in the corresponding direction.

35. A storage medium, characterized in that, The bitstream obtained using the decoding / encoding method according to any one of claims 1 to 19 is stored.

Images