Bidirectional optical flow method and device for reference image resampling in video decoding

Abstract

A decoding method implemented by a video decoder, comprising: the video decoder determines whether the resolution of a current image being decoded is the same as the resolution of a reference image identified by a reference image list associated with the current image; when it is determined that the resolution of the current image is the same as the resolution of each reference image in the reference images, the video decoder enables bi-direction optical flow (BDOF) for a current block of the current image; when it is determined that the resolution of the current image is different from the resolution of any reference image in the reference images, the video decoder disables the BDOF for the current block of the current image; when the BDOF is enabled for the current block, the video decoder uses the BDOF to correct a motion vector corresponding to the current block.

Description

[0001] This application is a divisional application. The application number of the original application is 202080035926.8, and the original application date is May 14, 2020. The entire contents of the original application are incorporated into this application by reference.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 848,409, filed by Jianle Chen et al. on May 15, 2019, entitled “Processing of Bidirectional Optical Flow (BIO) Decoding Tools for Reference Image Resampling in Video Decoding,” which is incorporated by reference into this application. Technical Field

[0004] The present application generally describes techniques for supporting bi-direction optical flow (BDOF) in video coding. More specifically, the present application describes that the BDOF can be used in reference picture resampling, but the BDOF can only be disabled for blocks or samples when the spatial resolution of the current picture is different from the spatial resolution of the reference picture. Background Art

[0005] Even if a video is relatively short, a large amount of video data is required to describe it. This can cause difficulties when the data is to be streamed or otherwise transmitted over a communication network with limited bandwidth capacity. Therefore, the video data usually needs to be compressed before being transmitted over modern telecommunications networks. Since memory resources may be limited, the size of the video may also cause problems when storing the video in a storage device. Video compression devices usually use software and / or hardware on the source side to decode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The video decompression device that decodes the video data then receives the compressed data on the destination side. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that have little impact on image quality while increasing the compression ratio. Summary of the invention

[0006] A first aspect relates to a method implemented by a video decoder for decoding an encoded video stream. The method comprises: the video decoder determines whether the resolution of a current image being decoded is the same as the resolution of a reference image identified by a reference image list associated with the current image; when it is determined that the resolution of the current image is the same as the resolution of each reference image in the reference image, the video decoder enables bi-directional optical flow (BDOF) for a current block of the current image; when it is determined that the resolution of the current image is different from the resolution of any reference image in the reference image, the video decoder disables the BDOF for the current block of the current image; when the BDOF is enabled for the current block, the video decoder uses the BDOF to correct a motion vector corresponding to the current block.

[0007] In the technology provided by the method, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when reference picture resampling (RPR) is enabled. Since this method has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to existing codecs. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video.

[0008] Optionally, according to any of the above aspects, in another implementation manner of this aspect, enabling the BDOF includes setting a BDOF flag to a first value, and disabling the BDOF includes setting the BDOF flag to a second value.

[0009] Optionally, according to any of the above aspects, in another implementation manner of this aspect, the method further includes: generating a reference image for the current image according to a reference image list in a bidirectional inter-frame prediction mode.

[0010] Optionally, according to any of the above aspects, in another implementation of this aspect, the method further includes: selectively enabling or disabling the BDOF for blocks in the multiple images based on whether the resolution of each image in the multiple images is the same as or different from the resolution of a reference image associated with the image.

[0011] Optionally, according to any of the above aspects, in another implementation of this aspect, the method further includes: when the BDOF is disabled, enabling reference picture resampling (RPR) for the entire coded video sequence (CVS) including the current image.

[0012] Optionally, according to any one of the above aspects, in another implementation manner of this aspect, the resolution of the current image is set in a parameter set of the encoded video code stream, and the current block is obtained from a strip of the current image.

[0013] Optionally, according to any of the above aspects, in another implementation manner of this aspect, the method further includes: displaying an image generated according to the current block on a display screen of the electronic device.

[0014] The second aspect relates to a method for encoding a video bitstream implemented by a video encoder. The method includes: the video encoder determines whether the resolution of a current image being encoded is the same as the resolution of a reference image identified by a reference image list associated with the current image; when it is determined that the resolution of the current image is the same as the resolution of each reference image in the reference image, the video encoder enables bi-direction optical flow (BDOF) for the current block of the current image; when it is determined that the resolution of the current image is different from the resolution of any reference image in the reference image, the video encoder disables the BDOF for the current block of the current image; when the BDOF is enabled for the current block, the video encoder uses the BDOF to correct the motion vector corresponding to the current block.

[0015] In the technology provided by the method, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when reference picture resampling (RPR) is enabled. Since this method has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to existing codecs. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video.

[0016] Optionally, according to any of the above aspects, in another implementation of this aspect, the method also includes: the video encoder determines the motion vector of the current image based on the reference image; the video encoder encodes the current image based on the motion vector; the video encoder decodes the current image through a hypothetical reference decoder.

[0017] Optionally, according to any of the above aspects, in another implementation manner of this aspect, enabling the BDOF includes setting a BDOF flag to a first value, and disabling the BDOF includes setting the BDOF flag to a second value.

[0018] Optionally, according to any of the above aspects, in another implementation manner of this aspect, the method further includes: generating a reference image for the current image according to a reference image list in a bidirectional inter-frame prediction mode.

[0019] Optionally, according to any of the above aspects, in another implementation of this aspect, the method further includes: selectively enabling or disabling the BDOF for blocks in the multiple images based on whether the resolution of each image in the multiple images is the same as or different from the resolution of a reference image associated with the image.

[0020] Optionally, according to any of the above aspects, in another implementation of this aspect, the method further includes: enabling reference picture resampling (RPR) for the entire coded video sequence (coded videosequence, CVS) including the current image even when the BDOF is disabled.

[0021] Optionally, according to any of the above aspects, in another implementation manner of this aspect, the method further includes: sending a video code stream including the current block to a video decoder.

[0022] The third aspect relates to a decoding device. The decoding device includes: a receiver for receiving an encoded video code stream; a memory coupled to the receiver, wherein the memory stores instructions; a processor coupled to the memory, wherein the processor is used to execute the instructions so that the decoding device performs the following operations: determining whether the resolution of the current image being decoded is the same as the resolution of the reference image identified by the reference image list associated with the current image; when it is determined that the resolution of the current image is the same as the resolution of each reference image in the reference image, enabling bi-direction optical flow (BDOF) for the current block of the current image; when it is determined that the resolution of the current image is different from the resolution of any reference image in the reference image, disabling the BDOF for the current block of the current image; when the BDOF is enabled for the current block, using the BDOF to correct the motion vector corresponding to the current block.

[0023] In the technology provided by the decoding device, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when reference picture resampling (RPR) is enabled. Since this method has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to existing codecs. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video.

[0024] Optionally, according to any of the above aspects, in another implementation of this aspect, when the BDOF is disabled, reference picture resampling (RPR) is enabled for the entire coded video sequence (CVS) including the current picture.

[0025] Optionally, according to any of the above aspects, in another implementation of this aspect, the decoding device also includes a display for displaying an image generated according to the current block.

[0026] The fourth aspect relates to an encoding device. The encoding device includes: a memory storing instructions; a processor coupled to the memory, wherein the processor is used to execute the instructions so that the encoding device performs the following operations: determining whether the resolution of the current image being encoded is the same as the resolution of the reference image identified by the reference image list associated with the current image; when it is determined that the resolution of the current image is the same as the resolution of each reference image in the reference image, enabling bi-direction optical flow (BDOF) for the current block of the current image; when it is determined that the resolution of the current image is different from the resolution of any reference image in the reference image, disabling the BDOF for the current block of the current image; when the BDOF is enabled for the current block, using the BDOF to correct the motion vector corresponding to the current block; a transmitter coupled to the processor, wherein the transmitter is used to send a video code stream including the current block to a video decoder.

[0027] In the technology provided by the encoding device, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when reference picture resampling (RPR) is enabled. Since this method has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to existing codecs. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video.

[0028] Optionally, according to any of the above aspects, in another implementation of this aspect, even when the BDOF is disabled, reference picture resampling (RPR) is enabled for the entire coded video sequence (CVS) including the current image.

[0029] Optionally, according to any one of the above aspects, in another implementation manner of this aspect, before the transmitter sends the video code stream to the video decoder, the memory stores the code stream.

[0030] The fifth aspect relates to a decoding device. The decoding device includes: a receiver, configured to receive an image for encoding or receive a code stream for decoding; a transmitter, coupled to the receiver, wherein the transmitter is configured to send the code stream to a decoder or send the decoded image to a display; a memory, coupled to at least one of the receiver or the transmitter, wherein the memory is configured to store instructions; and a processor, coupled to the memory, wherein the processor is configured to execute the instructions stored in the memory to perform the method disclosed herein.

[0031] In the technology provided by the decoding device, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when reference picture resampling (RPR) is enabled. Since this method has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to existing codecs. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video.

[0032] The sixth aspect relates to a system, which includes: an encoder; and a decoder communicating with the encoder, wherein the encoder or the decoder includes the decoding device, the encoding device or the decoding apparatus disclosed herein.

[0033] In the technology provided by the system, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when reference picture resampling (RPR) is enabled. Since this approach has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to existing codecs. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] For a more complete understanding of the present application, reference is made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like components.

[0035] Figure 1is a block diagram of an exemplary decoding system that may utilize video decoding techniques.

[0036] Figure 2 A block diagram of an exemplary video encoder that can implement video decoding technology.

[0037] Figure 3 A block diagram of an exemplary video decoder that can implement video decoding technology.

[0038] Figure 4 A representation of the relationship between an IRAP picture relative to preceding and succeeding pictures when arranged in decoding order and display order, respectively, is shown.

[0039] Figure 5 An example of multi-layer decoding for achieving spatial hierarchicality is shown.

[0040] Figure 6 A schematic diagram of an example of unidirectional inter-frame prediction.

[0041] Figure 7 A schematic diagram of an example of bidirectional inter-frame prediction.

[0042] Figure 8 A video code stream is shown.

[0043] Fig. 9 A segmentation technique for an image is shown.

[0044] Fig.10 An embodiment of a method for decoding an encoded video stream is shown.

[0045] Fig.11 An embodiment of a method for encoding a coded video stream is shown.

[0046] Fig.12 A schematic diagram of a video decoding device.

[0047] Fig.13 is a schematic diagram of an embodiment of a decoding unit. DETAILED DESCRIPTION

[0048] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or existing. The present application should not be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, and may be modified within the scope of the appended claims and the full scope of their equivalents.

[0049] Resolution as described in this article describes the number of pixels in a video file. That is, the resolution is the width and height of the projected image measured in pixels. For example, the resolution of a video may be 1280 (horizontal pixels) × 720 (vertical pixels). This is often shortened to 1280×720, or abbreviated as 720p. BDOF is a process, algorithm, or decoding tool for modifying the motion or motion vector of a prediction block. In BDOF, the motion vector can be found for a sub-coding unit based on the gradient of the difference between two reference images. The RPR feature refers to the ability to change the spatial resolution of a coded image located in the middle of the bitstream without intra-decoding the image at the location where the resolution is changed.

[0050] Figure 1 FIG. 1 is a block diagram of an exemplary decoding system 10 that may utilize the video decoding techniques described herein. Figure 1 As shown, the decoding system 10 includes a source device 12, wherein the source device 12 provides encoded video data that is then decoded by a destination device 14. Specifically, the source device 12 can provide the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 can include any of a variety of devices, including desktop computers, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as "smart" phones and "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device 12 and the destination device 14 can be equipped for wireless communication.

[0051] The destination device 14 may receive the encoded video data to be decoded via a computer-readable medium 16. The computer-readable medium 16 may include any type of medium or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may include a communication medium so that the source device 12 can send the encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and sent to the destination device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may constitute a portion of a packet network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that facilitates communication from the source device 12 to the destination device 14.

[0052] In some examples, the encoded data may be output from the output interface 22 to a storage device. Similarly, the encoded data may be accessed from the storage device through the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital video disk (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or other intermediate storage device, wherein the other intermediate storage device may store the encoded video generated by the source device 12. The destination device 14 may access the stored video data from the storage device by streaming or downloading. The file server may be any type of server capable of storing encoded video data and sending the encoded video data to the destination device 14. Exemplary file servers include a World Wide Web server (e.g., for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. The destination device 14 may access the encoded video data through any standard data connection, including an Internet connection. The standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both, suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data in the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0053] The techniques of the present application are not necessarily limited to wireless applications or settings. These techniques can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission such as HTTP dynamic adaptive streaming over HTTP (DASH), encoding digital video for storage in a data storage medium, decoding digital video stored in a data storage medium, or other applications. In some examples, the decoding system 10 can be used to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0054] exist Figure 1In the example of , source device 12 includes video source 18, video encoder 20 and output interface 22. Destination device 14 includes input interface 28, video decoder 30 and display device 32. According to the present application, video encoder 20 of source device 12 and / or video decoder 30 of destination device 14 can be used to apply these techniques for video decoding. In other examples, source device and destination device can include other components or devices. For example, source device 12 can receive video data from an external video source such as an external camera. Similarly, destination device 14 can be connected to an external display device instead of including an integrated display device.

[0055] Figure 1 The decoding system 10 shown is only an example. These techniques for video decoding can be implemented by any digital video encoding and / or decoding device. Although the techniques of the present application are generally implemented by a video decoding device, these techniques can also be implemented by a video encoder / decoder (commonly referred to as a "codec"). In addition, the techniques of the present application can also be implemented by a video preprocessor. The video encoder and / or video decoder can be a graphics processing unit (GPU) or a similar device.

[0056] Source device 12 and destination device 14 are merely examples of such decoding devices, where source device 12 generates encoded video data to send to destination device 14. In some examples, source device 12 and destination device 14 may perform substantially symmetrical operations, and thus, both source device 12 and destination device 14 include video encoding and decoding components. Thus, decoding system 10 may support one-way or two-way video transmission between video devices 12 and 14, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0057] Video source 18 in source device 12 may include a video capture device (e.g., a video camera), a video archive including previously captured video, and / or a video feed interface that receives video from a video content provider. In another alternative, video source 18 may generate computer graphics-based data as the source video, or as a combination of live video, archived video, and computer-generated video.

[0058] In some cases, when video source 18 is a video camera, source device 12 and destination device 14 may constitute a camera phone or a video phone. However, as described above, the techniques described in this application may be generally applicable to video decoding and may be applied to wireless and / or wired applications. In each case, captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output to computer-readable medium 16 via output interface 22.

[0059] The computer-readable medium 16 may include transient media, such as wireless broadcast or wired network transmission, and may also include storage media (i.e., non-transient storage media), such as hard disks, flash drives, optical disks, digital video disks, Blu-ray disks, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from the source device 12 and provide the encoded video data to the destination device 14 via network transmission or the like. Similarly, a computing device in a media production facility such as a disc stamping facility may receive encoded video data from the source device 12 and generate a disc including the encoded video data. Therefore, in various examples, the computer-readable medium 16 may be understood to include one or more computer-readable media in various forms.

[0060] The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information of the computer-readable medium 16 may include syntax information defined by the video encoder 20. The syntax information is also used by the video decoder 30 and includes syntax elements for describing characteristics and / or processing of blocks and other coding units (e.g., group of pictures (GOP)). The display device 32 displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0061] The video encoder 20 and the video decoder 30 may operate in accordance with a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, the video encoder 20 and the video decoder 30 may operate in accordance with other proprietary or industry standards, such as the International Telecommunications Union Telecommunication Standardization Sector (ITU-T) H.264 standard, also known as Moving Picture Expert Group (MPEG)-4 Part 10, Advanced Video Coding (AVC), H.265 / HEVC, or extended versions of such standards. However, the technology of the present application is not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although in Figure 1 Not shown in the figure, in some aspects, the video encoder 20 and the video decoder 30 can be integrated with the audio encoder and the audio decoder, respectively, and can include a suitable multiplexer-demultiplexer (MUX-DEMUX) unit or other hardware and software to encode both the audio and video in a common data stream or a separate data stream. If applicable, the MUX-DEMUX unit can comply with the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).

[0062] The video encoder 20 and the video decoder 30 can be implemented as any of various suitable encoder circuits, such as one or more microprocessors, one or more digital signal processors (DSP), one or more application-specific integrated circuits (ASIC), one or more field programmable gate arrays (FPGA), one or more discrete logic, one or more software, one or more hardware, one or more firmware, or any combination thereof. When these techniques are partially implemented in software, the device can store the instructions of the software in a suitable non-transitory computer-readable medium, and execute these instructions through one or more processors in the hardware to implement the technology of the present application. The video encoder 20 and the video decoder 30 can be respectively included in one or more encoders or decoders, that is, either can be integrated as part of a combined encoder / decoder (codec) in the corresponding device. The device including the video encoder 20 and / or the video decoder 30 may include an integrated circuit, a microprocessor and / or a wireless communication device such as a cellular phone.

[0063] Figure 2 1 is a block diagram of an exemplary video encoder 20 that can implement video decoding techniques. The video encoder 20 can perform intra-frame coding and inter-frame coding on video blocks within a video slice. Intra-frame coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or image. Inter-frame coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or images of a video sequence. Intra-frame mode (I mode) can refer to any of several spatial-based coding modes. Inter-frame modes, such as uni-directional prediction (uni-directional / uniprediction) (P mode) or bi-prediction (bi-prediction / bi prediction) (B mode), can refer to any of several temporal-based coding modes.

[0064] like Figure 2 As shown, the video encoder 20 receives a current video block within a video frame to be encoded. Figure 2In the example of FIG. 5 , the video encoder 20 includes a mode selection unit 40, a reference frame memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The mode selection unit 40 further includes a motion compensation unit 44, a motion estimation unit 42, an intra-predition / intra prediction unit 46, and a segmentation unit 48. To reconstruct the video blocks, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and an adder 62. Deblocking filter ( Figure 2 A filter (not shown in the figure) is also included to filter the block boundaries in order to remove the blocking artifacts from the reconstructed video. If necessary, a deblocking filter is typically used to filter the output of adder 62. In addition to the deblocking filter, other (in-loop or post-loop) filters may also be used. For the sake of simplicity, such filters are not shown in the figure, but if necessary, such filters (as in-loop filters) can filter the output of adder 50.

[0065] During the encoding process, video encoder 20 receives a video frame or slice to be encoded. The frame or slice may be divided into a plurality of video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-frame prediction encoding on the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-frame prediction unit 46 may also perform intra-frame prediction encoding on the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be encoded to provide spatial prediction. Video encoder 20 may perform multiple encoding processes, for example, to select an appropriate encoding mode for each block of video data.

[0066] In addition, the segmentation unit 48 can segment the blocks of the video data into sub-blocks based on an evaluation of a previous segmentation scheme in a previous encoding process. For example, the segmentation unit 48 can initially segment a frame or a strip into a plurality of largest coding units (LCUs), and segment each LCU into a plurality of sub-coding units (sub-CUs) based on a rate-distortion analysis (e.g., rate-distortion optimization). The mode selection unit 40 can further generate a quadtree data structure indicating that the LCU is segmented into a plurality of sub-CUs. The leaf node CU of the quadtree may include one or more prediction units (PUs) and one or more transform units (TUs).

[0067] The term "block" is used in this application to refer to any of CU, PU or TU in the HEVC context or similar data structures in other standard contexts (for example, macroblocks and their sub-blocks in H.264 / AVC). A CU includes a coding node, a PU associated with the coding node, and a TU. The size of the CU corresponds to the size of the coding node and is square. The size of the CU can range from 8×8 pixels to a tree block size of 64×64 pixels or larger. Each CU may include one or more PUs and one or more TUs. For example, the syntax data associated with the CU may describe the partitioning of the CU into one or more PUs. The partitioning mode may be different when the CU is encoded in skip mode or direct mode, intra-frame prediction mode or inter-frame prediction mode (inter-predition / inter prediction). The PU can be partitioned into non-square shapes. For example, the syntax data associated with the CU may also describe the partitioning of the CU into one or more TUs according to a quadtree. A TU can be square or non-square (for example, rectangular).

[0068] The mode selection unit 40 may select one of the intra-frame coding mode or the inter-frame coding mode according to the error result, etc., and provide the resulting intra-frame coding block or inter-frame coding block to the adder 50 to generate residual block data, and to the adder 62 to reconstruct the coding block for use as a reference frame. The mode selection unit 40 also provides syntax elements such as motion vectors, intra-frame mode indicators, partition information, and other such syntax information to the entropy coding unit 56.

[0069] Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are described separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, a motion vector may represent the displacement of a PU of a video block within a current video frame or image relative to a prediction block in a reference frame (or other coding unit), the prediction block in the reference frame (or other coding unit) being relative to the current block being encoded in the current frame (or other coding unit). The prediction block is a block that is found to closely match the block to be encoded in terms of pixel difference, which may be determined by the sum of absolute difference (SAD), the sum of square difference (SSD) or other difference metrics. In some examples, video encoder 20 may calculate the values ​​of sub-integer pixel positions of a reference image stored in reference frame memory 64. For example, video encoder 20 may interpolate the values ​​of quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference image. Thus, motion estimation unit 42 may perform motion searches relative to integer pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0070] Motion estimation unit 42 calculates a motion vector for the PU by comparing the position of the PU of the video block in the inter-frame coded slice with the position of the prediction block of the reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), where each list is used to identify one or more reference pictures stored in reference frame memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy coding unit 56 and motion compensation unit 44.

[0071] The motion compensation performed by the motion compensation unit 44 may include extracting or generating a prediction block based on the motion vector determined by the motion estimation unit 42. In addition, in some examples, the motion estimation unit 42 and the motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector of the PU of the current video block, the motion compensation unit 44 may find the prediction block pointed to by the motion vector in one of the reference image lists. The adder 50 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded to obtain pixel difference values, as described below. Typically, the motion estimation unit 42 performs motion estimation on the luma component, and the motion compensation unit 44 uses the motion vector calculated based on the luma component for both the chroma component and the luma component. The mode selection unit 40 may also generate syntax elements associated with the video block and the video slice for use by the video decoder 30 to decode the video block of the video slice.

[0072] Intra-prediction unit 46 may perform intra-prediction on the current block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intra-prediction unit 46 may determine an intra-prediction mode to use for encoding the current block. In some examples, intra-prediction unit 46 may encode the current block using various intra-prediction modes, for example, in a separate encoding process, and intra-prediction unit 46 (or mode selection unit 40 in some examples) may select an appropriate intra-prediction mode to use from the tested modes.

[0073] For example, intra-prediction unit 46 may calculate rate-distortion values ​​using rate-distortion analysis for various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics from the tested modes. Rate-distortion analysis is typically used to determine the amount of distortion (or error) between an encoded block and an original unencoded block (that was once encoded to produce the encoded block) and the code rate (i.e., the number of bits) used to produce the encoded block. Intra-prediction unit 46 may calculate a ratio based on the distortion and rate for each encoded block to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0074] In addition, the intra prediction unit 46 can be used to encode the depth block of the depth map using a depth modeling mode (DMM). The mode selection unit 40 can determine whether the available DMM mode produces better encoding results than the intra prediction mode and other DMM modes (e.g., using rate-distortion optimization (RDO)). The data of the texture image corresponding to the depth map can be stored in the reference frame memory 64. The motion estimation unit 42 and the motion compensation unit 44 can also be used to perform inter-frame prediction on the depth block of the depth map.

[0075] After selecting an intra-prediction mode for a block (e.g., a conventional intra-prediction mode or one of the DMM modes), the intra-prediction unit 46 may provide information to the entropy coding unit 56 indicating the intra-prediction mode selected for the block. The entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. The video encoder 20 may carry configuration data in the transmitted bitstream, the configuration data may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables), definitions of coding contexts for various blocks, indications of the most likely intra-prediction mode to be used for each coding context, intra-prediction mode index tables, and modified intra-prediction mode index tables.

[0076] Video encoder 20 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being encoded. Adder 50 represents one or more components that perform this subtraction operation.

[0077] Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block to produce a video block comprising residual transform coefficient values. Transform processing unit 52 may perform other transforms conceptually similar to the DCT. Wavelet transforms, integer transforms, sub-band transforms, or other types of transforms may also be applied.

[0078] The transform processing unit 52 applies a transform to the residual block, thereby generating a block with residual transform coefficients. The transform process can convert the residual information from the pixel value domain to a transform domain, such as a frequency domain. The transform processing unit 52 can send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameter. Then, in some examples, the quantization unit 54 can perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 can also perform the scan.

[0079] After quantization, entropy coding unit 56 performs entropy coding on the quantized transform coefficients. For example, entropy coding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding techniques. In the case of context-based entropy coding, the context may be based on neighboring blocks. After entropy coding unit 56 performs entropy coding, the encoded bitstream may be sent to another device (e.g., video decoder 30) or archived for subsequent transmission or retrieval.

[0080] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for subsequent use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the frames in reference frame memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to produce a reconstructed video block, which is stored in reference frame memory 64. Motion estimation unit 42 and motion compensation unit 44 may use the reconstructed video block as a reference block for inter-coding blocks in subsequent video frames.

[0081] Figure 3 FIG. 3 is a block diagram of an exemplary video decoder 30 that can implement video decoding techniques. Figure 3 In the example of FIG. 1 , video decoder 30 includes entropy decoding unit 70, motion compensation unit 72, intra prediction unit 74, inverse quantization unit 76, inverse transform unit 78, reference frame memory 82, and adder 80. In some examples, video decoder 30 may perform the same operations as video encoder 20 ( Figure 2 ) is a decoding process that is substantially the reverse of the encoding process performed by the entropy decoding unit 70. The motion compensation unit 72 may generate prediction data based on the motion vector received from the entropy decoding unit 70, and the intra-prediction unit 74 may generate prediction data based on the intra-prediction mode indicator received from the entropy decoding unit 70.

[0082] During the decoding process, the video decoder 30 receives an encoded video bitstream from the video encoder 20, which represents video blocks of an encoded video slice and associated syntax elements. The entropy decoding unit 70 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. The entropy decoding unit 70 forwards the motion vectors and other syntax elements to the motion compensation unit 72. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level.

[0083] When the video slice is encoded as an intra-coded (I) slice, intra-prediction unit 74 may generate prediction data for the video blocks of the current video slice based on the indicated intra-prediction mode and data from previously decoded blocks of the current frame or picture. When the video frame is encoded as an inter-coded (e.g., B, P, or GPB) slice, motion compensation unit 72 generates prediction blocks for the video blocks of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 70. These prediction blocks may be generated based on one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame list 0 and list 1 using a default construction technique based on the reference pictures stored in reference frame memory 82.

[0084] The motion compensation unit 72 determines prediction information for the video blocks of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate a prediction block for the current video block being decoded. For example, the motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video slice, the inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information of one or more reference picture lists for the slice, the motion vector of each inter-coded video block in the slice, the inter prediction state of each inter-coded video block in the slice, and other information to decode the video blocks in the current video slice.

[0085] Motion compensation unit 72 may also interpolate based on interpolation filters. Motion compensation unit 72 may calculate interpolated values ​​for sub-integer pixels of reference blocks using interpolation filters used by video encoder 20 during encoding of the video blocks. In this case, motion compensation unit 72 may determine the interpolation filters used by video encoder 20 based on received syntax elements and use these interpolation filters to produce prediction blocks.

[0086] Data of the texture image corresponding to the depth map may be stored in the reference frame memory 82. The motion compensation unit 72 may also be configured to perform inter-frame prediction on the depth block of the depth map.

[0087] In one embodiment, the video decoder 30 includes a user interface (UI) 84. The user interface 84 is used to receive input from a user (e.g., a network administrator) of the video decoder 30. Through the user interface 84, the user can manage or change settings on the video decoder 30. For example, the user can input or otherwise provide the value of a parameter (e.g., a flag) to control the configuration and / or operation of the video decoder 30 according to the user's preferences. For example, the user interface 84 can be a graphical user interface (GUI) that supports the user to interact with the video decoder 30 through graphical icons, drop-down menus, check boxes, etc. In some cases, the user interface 84 can receive information from the user through a keyboard, a mouse, or other peripheral devices. In one embodiment, the user can access the user interface 84 through a smart phone, a tablet device, a personal computer away from the video decoder 30, etc. The user interface 84 described herein can be referred to as an external input unit or an external unit.

[0088] In view of the above, video compression technology is used to perform spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove the redundancy inherent in the video sequence. For block-based video decoding, a video strip (i.e., a video image or a portion of a video image) can be divided into multiple video blocks, which can also be referred to as tree blocks, coding tree blocks (CTB), coding tree units (CTU), coding units (CU) and / or coding nodes. The video blocks in the intra-frame coding (I) strip of the image are encoded using spatial prediction for reference samples of adjacent blocks of the same image. The video blocks in the inter-frame coding (P or B) strip of the image can use spatial prediction for reference samples of adjacent blocks of the same image, or use temporal prediction for reference samples of other reference images. The image can be called a frame, and the reference image can be called a reference frame.

[0089] Spatial prediction or temporal prediction is used to generate a prediction block for the block to be encoded. The residual data represents the pixel differences between the original block to be encoded and the prediction block. Inter-frame coded blocks are encoded based on motion vectors and residual data, wherein the motion vectors point to the blocks of reference samples that make up the prediction block, and the residual data represents the difference between the coded block and the prediction block. Intra-frame coded blocks are encoded based on intra-frame coding modes and residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to produce residual transform coefficients that can be quantized. The quantized transform coefficients are initially arranged in a two-dimensional array and can be scanned to produce a one-dimensional vector of transform coefficients. Entropy coding can be used to achieve further compression.

[0090] Image and video compression technology has developed rapidly, resulting in various coding standards. Such video coding standards include ITU-T H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) MPEG-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and 3D AVC (3D-AVC) and other extensions. HEVC includes extended versions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC) and 3D HEVC (3D-HEVC).

[0091] Versatile Video Coding (VVC) is a new video coding standard being developed by the joint video experts team (JVET) of ITU-T and ISO / IEC. Although there are several working drafts of the VVC standard, one of the VVC working drafts (WD) (Draft 5), namely the proposal "Versatile Video Coding (Draft 5)" JVET-N1001-v3 by B.Bross, J.Chen and S.Liu at the 13th JVET meeting on March 27, 2019, is incorporated herein by reference in its entirety.

[0092] The description of the techniques disclosed herein is based on the video coding standard being developed by the joint video experts team (JVET) of ITU-T and ISO / IEC, namely versatile video coding (VVC). However, these techniques are also applicable to other video codec specifications.

[0093] Figure 4 A representation 400 of the relationship between an intrarandom access point (IRAP) picture 402 relative to a preceding picture 404 and a succeeding picture 406 when arranged in decoding order 408 and display order 410, respectively, is shown. In one embodiment, the IRAP picture 402 is referred to as a clean random access (CRA) picture or an instantaneous decoder refresh (IDR) picture accompanying a decodable random access leading (RADL) picture. In HEVC, IDR pictures, CRA pictures, and Broken Link Access (BLA) pictures are all considered IRAP pictures 402. For VVC, at the 12th JVET meeting in October 2018, it was agreed that IDR pictures and CRA pictures would be IRAP pictures. In one embodiment, Broken Link Access (BLA) pictures and Gradual Decoder Refresh (GDR) pictures may also be considered IRAP pictures. For a coded video sequence, the decoding process always starts from an IRAP picture.

[0094] like Figure 4 As shown, the preceding pictures 404 (e.g., pictures 2 and 3) are located after the IRAP picture 402 in decoding order 408, and before the IRAP picture 402 in display order 410. The succeeding pictures 406 are located after the IRAP picture 402 in both decoding order 408 and display order 410. Figure 4 Two front images 404 and one rear image 406 are depicted in the figure, but those skilled in the art should understand that in actual applications, there may be more or fewer front images 404 and / or rear images 406 arranged in the decoding order 408 and the display order 410.

[0095] Figure 4The leading picture 404 in HEVC is divided into two types, namely, random access skipped leading (RASL) pictures and RADL leading pictures. When decoding starts from an IRAP picture 402 (e.g., picture 1), a RADL picture (e.g., picture 3) can be decoded normally, while a RASL picture (e.g., picture 2) cannot be decoded normally. Therefore, the RASL picture is discarded. In view of the difference between RADL pictures and RASL pictures, the type of the leading picture 404 associated with the IRAP picture 402 should be identified as RADL or RASL for normal and efficient decoding. In HEVC, when there are RASL and RADL pictures, they are subject to the following restrictions: for RASL pictures and RADL pictures associated with the same IRAP picture 402, the RASL picture should be located before the RADL picture in the display order 410.

[0096] The IRAP picture 402 provides the following two important functions / benefits. First, the presence of the IRAP picture 402 indicates that the decoding process can start from this picture. This function supports the random access feature. During random access, the decoding process does not necessarily start from the beginning of the codestream, but from a certain position in the codestream as long as the IRAP picture 402 exists at that position. Second, the presence of the IRAP picture 402 refreshes the decoding process, so that the coded pictures starting with the IRAP picture 402 (excluding RASL pictures) can be decoded without reference to the previous pictures. Therefore, if the IRAP picture 402 exists in the codestream, any errors that may occur during the decoding of the coded pictures located before the IRAP picture 402 can be prevented from propagating to the IRAP picture 402 and those pictures located after the IRAP picture 402 in the decoding order 408.

[0097] Although the IRAP picture 402 provides important functions, these functions affect compression efficiency. The presence of the IRAP picture 402 causes a sharp increase in bit rate. There are two reasons for the inefficiency in compression efficiency. First, because the IRAP picture 402 is an intra-frame predicted picture, it requires relatively more bits to represent the picture than other pictures that are intra-frame predicted pictures (e.g., the previous picture 404, the subsequent picture 406). Second, because the presence of the IRAP picture 402 interrupts the temporal prediction (this is because the decoder refreshes the decoding process, where one of the actions of the decoding process is to delete the previous reference pictures in the decoded picture buffer (DPB)), the IRAP picture 402 causes the decoding of the pictures that are located after the IRAP picture 402 in the decoding order 408 to be less efficient (i.e., require more bits to represent) because these pictures have fewer reference pictures for inter-frame prediction decoding.

[0098] Among the picture types considered as IRAP pictures 402, IDR pictures in HEVC have different indications and derivations compared to other picture types. Some of the differences are as follows:

[0099] To indicate and derive the picture order count (POC) value of an IDR picture, the most significant bit (MSB) portion of the POC is not derived from the previous key picture but is simply set equal to 0.

[0100] For the indication information required for reference picture management, the slice header of the IDR picture does not include the information that needs to be indicated to assist in reference picture management. For other picture types (i.e., CRA, post-position, temporal sub-layer access (TSA) pictures, etc.), the reference picture identification process (i.e., the process of determining the status of reference pictures used and not used for reference in the decoded picture buffer (DPB)) requires information such as the reference picture set (RPS) described below or other forms of similar information (e.g., reference picture lists). However, for IDR pictures, such information does not need to be indicated because the presence of an IDR picture indicates that the decoding process should only identify all reference pictures in the DPB as not used for reference.

[0101] In HEVC and VVC, the IRAP image 402 and the preceding image 404 may be contained in a single network abstraction layer (NAL) unit, respectively. A set of NAL units may be referred to as an access unit. The IRAP image 402 and the preceding image 404 are assigned different NAL unit types so that these images are easily identified by system-level applications. For example, a video stripper needs to understand the type of coded image, but does not need to understand too many details of the syntax elements in the coded bitstream, in particular, it needs to identify the IRAP image 402 from the non-IRAP image, and identify the preceding image 404 from the following image 406, including determining the RASL image and the RADL image. The following image 406 is an image associated with the IRAP image 402 and located after the IRAP image 402 in the display order 410. An image may be located after a specific IRAP image 402 in the decoding order 408, and before any other IRAP image 402 in the decoding order 408. To this end, assigning NAL unit types to the IRAP picture 402 and the preceding picture 404 respectively facilitates such applications.

[0102] For HEVC, the NAL unit types of IRAP images include the following:

[0103] BLA with preceding picture (BLA_W_LP): NAL unit of a Broken Link Access (BLA) picture that may be followed by one or more preceding pictures in decoding order.

[0104] BLA with RADL (BLA_W_RADL): A NAL unit of a BLA picture that may be followed by one or more RADL pictures but not by RASL pictures in decoding order.

[0105] BLA not accompanied by a preceding picture (BLA_N_LP): A NAL unit of a BLA picture that is not followed by a preceding picture in decoding order.

[0106] IDR with RADL (IDR_W_RADL): A NAL unit of an IDR picture that may be followed by one or more RADL pictures, but not by RASL pictures, in decoding order.

[0107] IDR not accompanied by a preceding picture (IDR_N_LP): A NAL unit of an IDR picture that is not followed by a preceding picture in decoding order.

[0108] CRA: A NAL unit of a Clean Random Access (CRA) picture that may be followed by a preceding picture (ie, a RASL picture or a RADL picture or both).

[0109] RADL: NAL unit for RADL images.

[0110] RASL: NAL unit of RASL image.

[0111] For VVC, the NAL unit types for the IRAP picture 402 and the preceding picture 404 are as follows:

[0112] IDR with RADL (IDR_W_RADL): A NAL unit of an IDR picture that may be followed by one or more RADL pictures, but not by RASL pictures, in decoding order.

[0113] IDR not accompanied by a preceding picture (IDR_N_LP): A NAL unit of an IDR picture that is not followed by a preceding picture in decoding order.

[0114] CRA: A NAL unit of a Clean Random Access (CRA) picture that may be followed by a preceding picture (ie, a RASL picture or a RADL picture or both).

[0115] RADL: NAL unit for RADL images.

[0116] RASL: NAL unit of RASL image.

[0117] The reference picture resampling (RPR) feature refers to the ability to change the spatial resolution of a coded image located in the middle of the bitstream without the need to intra-decode the image at the location where the resolution is changed. To achieve this feature, the image needs to be able to reference one or more reference images with a different spatial resolution than the current image for inter-frame prediction. Therefore, such reference images or parts of them need to be resampled to encode and decode the current image. Hence, it is called RPR. This feature can also be called adaptive resolution change (ARC) or other names. There are some use cases or application scenarios that benefit from the RPR feature, as described below.

[0118] Rate adaptation in video telephony and video conferencing: This is to adapt the encoded video to changing network conditions. When network conditions deteriorate, resulting in lower available bandwidth, the encoder can adapt by encoding smaller resolution images.

[0119] Active speaker changes in multi-party video conferencing: For multi-party video conferencing, the video size of the active speaker is usually larger than the video sizes of other participants. When the active speaker changes, the image resolution of each participant may also need to be adjusted. When the active speaker changes frequently, it is more important to have the ARC feature.

[0120] Fast streaming start: For streaming applications, the application usually buffers a certain length of decoded images before starting to display them. Starting a smaller resolution stream allows the application to have enough images in the buffer to start displaying them more quickly.

[0121] Adaptive stream switching in streaming: The Dynamic Adaptive Streaming over HTTP (DASH) specification includes a feature called @mediaStreamStructureId. This feature enables switching between different representations at an open GOP random access point with an undecodable preceding image, such as a CRA image associated with a RASL image in HEVC. When two different representations of the same video have different bitrates but the same spatial resolution, and the @mediaStreamStructureId values ​​of the two representations are the same, switching between the two representations can be performed at the CRA image associated with the RASL image, and the RASL image associated with the switched CRA image can be decoded with acceptable quality, thereby achieving seamless switching. Using ARC, the @mediaStreamStructureId feature can also be used to switch between DASH representations with different spatial resolutions.

[0122] There are various methods to facilitate the implementation of the underlying technology to support RPR / ARC, such as indicating a list of picture resolutions, placing some constraints on the resampling of reference pictures in the DPB, etc. In addition, during the 14th JVET meeting in Geneva, some contributions were proposed to apply constraints in the VVC standard to support RPR. The proposed constraints include the following.

[0123] When a current picture refers to a reference picture with a different resolution than the current picture, some tools should be disabled when decoding blocks in the current picture. These tools include:

[0124] Temporal motion vector prediction (TMVP) and advanced TMVP (ATMVP), which was proposed by JVET-N0118.

[0125] Decoder side motion vector refinement (DMVR), which was proposed by JVET-N0279.

[0126] Bi-directional optical flow (BIO), which was proposed by JVET-N0279.

[0127] Bidirectional prediction of blocks in reference pictures with a different resolution than the current picture is not allowed, as proposed by JVET-N0118.

[0128] For motion compensation, sample filtering should be applied only once, i.e., if resampling and interpolation is required to obtain a finer pixel resolution (e.g., quarter-pixel resolution), then both filters need to be combined and applied only once. This is proposed by JVET-N0118.

[0129] Scalability in video coding is usually supported by multi-layer decoding technology. The multi-layer code stream includes a base layer (BL) and one or more enhancement layers (EL). Examples of scalability include spatial scalability, quality / signal-to-noise (SNR) scalability, multi-view scalability, etc. When using multi-layer decoding technology, an image or a part of an image can be decoded (1) without using a reference picture, that is, without using intra-frame prediction, (2) by referring to a reference picture in the same layer, that is, using inter-frame prediction, or (3) by referring to a reference picture in another one or more layers, that is, using inter-layer prediction. The reference picture used for inter-layer prediction of the current image is called an inter-layer reference picture (ILRP).

[0130] Figure 5 An example 500 of multi-layer decoding for spatial scalability is shown. The resolution of the image 502 of the Nth layer is different from the resolution of the image 504 of the N+1th layer (e.g., the resolution of the image 502 is lower). In one embodiment, the Nth layer is considered to be the base layer as described above, and the N+1th layer is considered to be the enhancement layer as described above. The image 502 of the Nth layer and the image 504 of the N+1th layer can be decoded by inter-frame prediction (as shown by the solid arrows). The image 502 can also be decoded by inter-layer prediction (as shown by the dashed arrows).

[0131] In the context of performing RPR, the reference image may be resampled by selecting a reference image from a lower layer or by inter-layer prediction to generate an upper layer reference image from the lower layer reference image.

[0132] The previous H.26x video coding standard series supports the scalability of one or more profiles from one or more separate profiles to achieve single-layer decoding. Scalable video coding (SVC) is an extensible version of AVC / H.264 that supports spatial, temporal and quality scalability. For SVC, a flag is indicated in each macroblock (MB) of the EL image to indicate whether the EL Mb is predicted using a collocated block from the lower layer, where the prediction based on the collocated block may include texture, motion vector and / or coding mode. When implementing SVC, the unmodified H.264 / AVC implementation cannot be directly reused in the SVC implementation design. The syntax and decoding process of the EL macroblock in SVC are different from those in H.264 / AVC.

[0133] Scalable HEVC (SHVC) is an extension of the HEVC / H.265 standard that supports spatial and quality scalability; multiview HEVC (MV-HEVC) is an extension of HEVC / H.265 that supports multi-view adaptability; 3D HEVC (3D-HEVC) is an extension of HEVC / H.264 that supports more advanced and efficient three-dimensional (3D) video decoding than MV-HEVC. It should be noted that temporal adaptability is an integral part of the single-layer HEVC codec. The design of the multi-layer extension of HEVC adopts the following concept: the decoded pictures used for inter-layer prediction only come from the same access unit (AU), which is regarded as a long-term reference picture (LTRP) and is assigned a reference index in one or more reference picture lists together with other temporal reference pictures of the current layer. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the value of the reference index to refer to one or more inter-layer reference pictures in one or more reference picture lists.

[0134] It should be noted that both reference image resampling and spatial adaptability features require resampling of the reference image or a portion of the reference image. Reference image resampling can be implemented at the image level or the coding block level. However, when referred to as a decoding feature, RPR is a feature of single-layer decoding. Even so, from the perspective of codec design, it is possible and even preferred to use the same resampling filter to achieve the RPR feature of single-layer decoding and the spatial adaptability feature of multi-layer decoding.

[0135] JVET-N0279 proposes disabling BIO for RPR. More precisely, it proposes disabling BIO (also known as BDOF) for the entire coded video sequence (CVS) when RPR is enabled. It can be seen that even if the RPR feature is enabled, in many cases the current picture does not refer to a reference picture of a different resolution. Therefore, disabling BIO for the entire CVS is an unnecessary restriction and may reduce decoding efficiency.

[0136] In the technology disclosed in this article, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when RPR is enabled. Since this method has the ability to selectively disable BDOF, the decoding efficiency can be improved. Therefore, the processor, memory and / or network resources used on the encoder and decoder can be reduced. Therefore, the encoder / decoder (also known as "codec") in video decoding is improved relative to the existing codec. In fact, the improved video decoding process provides a better user experience when sending, receiving and / or watching video.

[0137] Figure 6 FIG. 6 is a schematic diagram of an example of unidirectional inter-frame prediction 600. The unidirectional inter-frame prediction 600 may be used to determine a motion vector of a coding block and / or a decoding block generated when an image is segmented.

[0138] In the unidirectional inter-frame prediction 600, a reference frame 630 including a reference block 631 is used to predict a current block 611 in a current frame 610. The reference frame 630 may be temporally located after the current frame 610 (e.g., as a next reference frame) as shown in the figure, but in some examples, it may also be temporally located before the current frame 610 (e.g., as a previous reference frame). The current frame 610 is an exemplary frame / image that is encoded / decoded at a specific time. The current frame 610 includes an object in the current block 611, which matches an object in the reference block 631 in the reference frame 630. The reference frame 630 is a frame used as a reference when encoding the current frame 610, and the reference block 631 is a block in the reference frame 630, and the object included in the reference block is also included in the current block 611 in the current frame 610.

[0139] The current block 611 is any coding unit that is encoded / decoded at a specified time point in the decoding process. The current block 611 can be an entire partition block or a sub-block when affine inter-frame prediction is used. The current frame 610 is separated from the reference frame 630 by a certain temporal distance (TD) 633. TD 633 represents the amount of time between the current frame 610 and the reference frame 630 in the video sequence, which can be measured in frames. The prediction information of the current block 611 can refer to the reference frame 630 and / or the reference block 631 through a reference index representing the direction and temporal distance between each frame. During the time period represented by TD 633, the object in the current block 611 moves from one position in the current frame 610 to another position in the reference frame 630 (for example, the position of the reference block 631). For example, the object can move along a trajectory 613, which represents the direction in which the object moves over time. The motion vector 635 describes the direction and magnitude of the object moving along the trajectory 613 within TD 633. Therefore, the encoded motion vector 635 , the reference block 631 , and the residual including the difference between the current block 611 and the reference block 631 provide sufficient information to reconstruct the current block 611 and locate the current block 611 in the current frame 610 .

[0140] Figure 7 FIG. 7 is a schematic diagram of an example of bidirectional inter-frame prediction 700. The bidirectional inter-frame prediction 700 may be used to determine motion vectors of coding blocks and / or decoding blocks generated when an image is segmented.

[0141] Bidirectional inter-frame prediction 700 is similar to unidirectional inter-frame prediction 600, but a pair of reference frames are used in bidirectional inter-frame prediction 700 to predict a current block 711 in a current frame 710. Therefore, the current frame 710 and the current block 711 are substantially similar to the current frame 610 and the current block 611, respectively. The current frame 710 is temporally located between a previous reference frame 720 and a next reference frame 730, wherein the previous reference frame 720 is located before the current frame 710 in the video sequence and the next reference frame 730 is located after the current frame 710 in the video sequence. The previous reference frame 720 and the next reference frame 730 are substantially similar to the reference frame 630 in other respects.

[0142] The current block 711 matches the previous reference block 721 in the previous reference frame 720 and the next reference block 731 in the next reference frame 730. This matching indicates that during the playback of the video sequence, an object moves along the motion trajectory 713 from the position of the previous reference block 721 through the current block 711 to the position of the next reference block 731. The current frame 710 is separated from the previous reference frame 720 by a certain previous time distance (TD0) 723, and is separated from the next reference frame 730 by a certain next time distance (TD1) 733. TD0 723 represents the amount of time between the previous reference frame 720 and the current frame 710 in the video sequence, in units of frames. TD1 733 represents the amount of time between the current frame 710 and the next reference frame 730 in the video sequence, in units of frames. Therefore, the object moves from the previous reference block 721 to the current block 711 along the motion trajectory 713 within the time period represented by TD0 723. The object also moves from the current block 711 to the next reference block 731 along the motion trajectory 713 within the time period represented by TD1 733. The prediction information of the current block 711 can refer to the previous reference frame 720 and / or the previous reference block 721 and the next reference frame 730 and / or the next reference block 731 through a pair of reference indexes representing the direction and time distance between the frames.

[0143] The previous motion vector (MV0) 725 describes the direction and magnitude of the object moving along the motion trajectory 713 within TD0 723 (e.g., between the previous reference frame 720 and the current frame 710). The next motion vector (MV1) 735 describes the direction and magnitude of the object moving along the motion trajectory 713 within TD1 733 (e.g., between the current frame 710 and the next reference frame 730). Therefore, in the bidirectional inter-frame prediction 700, the current block 711 can be decoded and reconstructed by the previous reference block 721 and / or the next reference block 731, MV0 725, and MV1 735.

[0144] In one embodiment, inter-frame prediction and / or bidirectional inter-frame prediction may be performed sample by sample (e.g., pixel by pixel) rather than block by block. That is, a motion vector pointing to each sample in the previous reference block 721 and / or the next reference block 731 may be determined for each sample in the current block 711. In these embodiments, Figure 7 The motion vector 725 and the motion vector 735 depicted in 710 represent a plurality of motion vectors corresponding to a plurality of samples in the current block 711 , the previous reference block 721 , and the next reference block 731 .

[0145] In the merge mode and the advanced motion vector prediction (AMVP) mode, the candidate list is generated by adding the candidate motion vectors to the candidate list in the order defined by the candidate list determination mode. These candidate motion vectors may include motion vectors obtained according to unidirectional inter-frame prediction 600, bidirectional inter-frame prediction 700, or a combination thereof. Specifically, the motion vectors are generated for the adjacent blocks when encoding the adjacent blocks. These motion vectors are added to the candidate list of the current block, and the motion vector of the current block is also selected from the candidate list. Then, the motion vector can indicate the index of the selected motion vector in the candidate list. The decoder can construct the candidate list using the same process as the encoder, and can determine the selected motion vector from the candidate list according to the indicated index. Therefore, the candidate motion vectors include motion vectors generated according to unidirectional inter-frame prediction 600 and / or bidirectional inter-frame prediction 700, depending on which method is used to encode the adjacent blocks.

[0146] Figure 8 A video code stream 800 is shown. The video code stream 800 described herein may also be referred to as an encoded video code stream, a code stream, or a variant thereof. Figure 8 As shown, the code stream 800 includes a sequence parameter set (SPS) 802 , a picture parameter set (PPS) 804 , a slice header 806 , and picture data 808 .

[0147] SPS 802 includes data common to all images in a sequence of picture (SOP), while PPS 804 includes data common to the entire image. Slice header 806 includes information about the current slice, such as slice type, reference image to be used, etc. SPS 802 and PPS 804 can be collectively referred to as parameter sets. SPS 802, PPS 804 and slice header 806 are network abstraction layer (NAL) unit types. NAL unit is a syntactic structure that includes an indication of the type of data to be followed (e.g., encoded video data). NAL units are divided into video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes data representing the value of a sample in a video image, and the non-VCL NAL unit includes any associated additional information, such as parameter sets (important slice header data that can be applied to a large number of VCL NAL units) and supplemental enhancement information (timing information and other supplementary data, wherein the supplementary data can enhance the usability of the decoded video signal, but it is not necessary for decoding the value of the sample in the video image). It should be understood by those skilled in the art that the code stream 800 may include other parameters and information in actual applications.

[0148] Figure 8 The image data 808 shown includes data associated with an image or video being encoded or decoded. The image data 808 can be simply referred to as the payload or data carried in the code stream 800. In one embodiment, the image data 808 includes a CVS 814 (or CLVS), which includes multiple images 810. The CVS 814 is a coded video sequence for each coded layer video sequence (CLVS) in the video code stream 800. It should be noted that when the video code stream 800 includes a single layer, the CVS is the same as the CLVS. Only when the video code stream 800 includes multiple layers, the CVS and the CLVS are different.

[0149] like Figure 8 As shown, the slices of each picture 810 may be contained in its own VCL NAL unit 812 for that picture 810. The collection of VCL NAL units 812 in a CVS 814 may be referred to as an access unit.

[0150] Fig. 9 A segmentation technique 900 for an image 910 is shown. The image 910 may be Figure 8As shown in the figure, the image 910 can be divided into a plurality of slices 912. A slice is a spatially distinct region in a frame (e.g., a picture) that is encoded separately from any other region in the same frame. Fig. 9 Three stripes 912 are shown in FIG. 1 , but more or fewer stripes may be used in actual applications. Each stripe 912 may be divided into a plurality of blocks 914 . Fig. 9 Block 914 in can be used with Figure 7 The current block 711, the previous reference block 721 and the next reference block 731 in FIG. 914 may represent a CU. Fig. 9 Four blocks 914 are shown in FIG. 9 , but more or fewer blocks may be used in actual applications.

[0151] Each block 914 may be divided into a plurality of samples 916 (eg, pixels). In one embodiment, the size of each block 914 is measured in units of luminance samples. Fig. 9 Sixteen samples 916 are shown, but more or fewer samples may be used in actual applications.

[0152] Fig.10 An embodiment of a method 1000 for decoding a coded video stream implemented by a video decoder (e.g., the video decoder 30 described above) is shown. The method 1000 can be performed after receiving the decoded stream directly or indirectly from a video encoder (e.g., the video encoder 20 described above). In the method 1000, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when RPR is enabled, thereby improving the decoding process. Since this method has the ability to selectively disable BDOF, the decoding efficiency can be improved. Therefore, this actually improves the performance of the codec, thereby providing a better user experience.

[0153] Step 1002: The video decoder determines whether the resolution of the current image being decoded is the same as the resolution of the reference image identified by the reference image list. In one embodiment, the video decoder receives an encoded video stream (e.g., the stream 800). The encoded video stream includes a reference image list, which indicates the resolution of the current image and indicates a bidirectional inter-frame prediction mode. In one embodiment, the reference image list is included in a reference image list structure. In one embodiment, the reference image list is used for bidirectional inter-frame prediction. In one embodiment, the resolution of the current image is set in a parameter set of the encoded video stream. In one embodiment, the resolution of the reference image is derived from the current image, inferred from the resolution of the current image, parsed from the stream, or obtained in other ways. In one embodiment, the reference image of the current image is generated according to the bidirectional inter-frame prediction mode and based on the reference image list.

[0154] Step 1004: When it is determined that the resolution of the current image is the same as the resolution of each of the reference images, the video decoder enables BDOF for the current block of the current image. In one embodiment, the video decoder enables the BDOF by setting a BDOF flag to a first value (e.g., true, 1, etc.). In one embodiment, even if the BDOF is enabled, the BDOF is an optional process. That is, even if the BDOF is enabled, it is not necessarily required to perform the BDOF.

[0155] Step 1006: When the resolution of the current image is different from the resolution of any of the reference images, the video decoder disables the BDOF for the current block of the current image. In one embodiment, the video decoder disables the BDOF by setting the BDOF flag to a second value (e.g., false, 0, etc.).

[0156] Step 1008: When the BDOF flag is set to the first value, the video decoder modifies the motion vector corresponding to the current block. In one embodiment, the method 1000 further includes: selectively enabling or disabling BDOF for other blocks in the current image according to whether the resolution of the current image is the same as or different from the resolution of the reference image.

[0157] In one embodiment, the method further comprises: enabling reference picture resampling (RPR) for the entire coded video sequence (CVS) including the current picture even when the BDOF is disabled.

[0158] In one embodiment, the current block is obtained from a slice of the current image. In one embodiment, the current image includes a plurality of slices, and the current block is obtained from one of the plurality of slices.

[0159] In one embodiment, the method further includes: displaying an image generated based on the current image to a user of an electronic device (eg, a smartphone, a tablet computer, a laptop computer, a personal computer, etc.).

[0160] Fig.11 An embodiment of a method 1100 for encoding a video stream implemented by a video encoder (e.g., the video encoder 20 described above) is shown. The method 900 can be performed when an image (e.g., from a video) needs to be encoded into a video stream and sent to a video decoder (e.g., the video decoder 30 described above). In the method 1100, when the spatial resolution of the current image is different from the spatial resolution of the reference image, BDOF can be selectively disabled without disabling BDOF for the entire CVS when RPR is enabled, thereby improving the encoding process. Since this method has the ability to selectively disable BDOF, decoding efficiency can be improved. Therefore, this actually improves the performance of the codec, thereby providing a better user experience.

[0161] Step 1102: The video encoder determines whether the resolution of the current image being encoded is the same as the resolution of the reference image identified by the reference image list. In one embodiment, the reference image list is included in the reference image list structure. In one embodiment, the reference image list is used for bidirectional inter-frame prediction. In one embodiment, the resolution of the current image is encoded into a parameter set of the video bitstream. In one embodiment, the reference image of the current image is generated according to the bidirectional inter-frame prediction mode and based on the reference image list.

[0162] Step 1104: When it is determined that the resolution of the current image is the same as the resolution of each of the reference images, the video encoder enables BDOF for the current block of the current image. In one embodiment, the video encoder enables the BDOF by setting a BDOF flag to a first value (e.g., true, 1, etc.). In one embodiment, even if the BDOF is enabled, the BDOF is an optional process. That is, even if the BDOF is enabled, it is not necessarily required to perform the BDOF.

[0163] In one embodiment, the method includes: determining a motion vector of a current image according to a reference image, encoding the current image according to the motion vector, and decoding the current image by a hypothetical reference decoder (HRD).

[0164] Step 1106: When the resolution of the current image is different from the resolution of any of the reference images, the video encoder disables the BDOF for the current block of the current image. In one embodiment, the video encoder disables the BDOF by setting the BDOF flag to a second value (e.g., false, 0, etc.).

[0165] Step 1108: When the BDOF flag is set to the first value, the video encoder modifies the motion vector corresponding to the current block. In one embodiment, the method 1100 further includes: selectively enabling or disabling BDOF for other blocks in the current image according to whether the resolution of the current image is the same as or different from the resolution of the reference image.

[0166] In one embodiment, the method further comprises: enabling reference picture resampling (RPR) for the entire coded video sequence (CVS) including the current picture even when the BDOF is disabled.

[0167] In one embodiment, the current block is obtained from a slice of the current image. In one embodiment, the current image includes a plurality of slices, and the current block is obtained from one of the plurality of slices.

[0168] In one embodiment, the video encoder generates a video code stream including the current block and sends the video code stream to the video decoder. In one embodiment, the video encoder stores the video code stream to send to the video decoder.

[0169] In one embodiment, a method for decoding a video code stream is disclosed. The video code stream includes at least one image, each image includes multiple slices. Each of the multiple slices includes multiple coding blocks and multiple reference image lists. Each reference image list in the multiple reference image lists includes multiple reference images, which can be used to perform inter-frame prediction on the coding blocks in the slice.

[0170] The method includes: parsing a parameter set to obtain resolution information of a current image; obtaining two reference image lists of a current strip in the current image; determining a reference image for decoding a current coding block in the current strip; determining the resolution of the reference image; determining whether to use or enable bi-direction optical flow (BIO) to decode the current coding block according to the resolution of the current image and the resolution of the reference image; and decoding the current coding block.

[0171] In one embodiment, the method includes: when the resolution of the current image is different from the resolution of the reference image, not using or disabling the BIO to decode the current coding block.

[0172] In one embodiment, a method for decoding a video bitstream is disclosed. The video bitstream includes at least one picture, each picture includes multiple slices. Each of the multiple slices is associated with a slice header including multiple syntax elements. Each of the multiple slices includes multiple coding blocks and multiple reference image lists. Each reference image list in the multiple reference image lists includes multiple reference images, which can be used to perform inter-frame prediction on the coding blocks in the current slice.

[0173] The method comprises: parsing a parameter set to obtain a flag, the flag being used to indicate whether a bi-direction optical flow (BIO) decoding tool / technique can be used to decode an image in a current encoded video sequence; obtaining a current slice in the current image; and when the value of the flag indicating whether a bi-direction optical flow (BIO) decoding tool / technique can be used to decode an image in a current encoded video sequence indicates that BIO can be used, parsing a slice header associated with the current slice to obtain a flag indicating whether a BIO decoding tool can be used to decode a coding block in the current slice.

[0174] In one embodiment, when the value of the flag indicating whether the BIO decoding tool can be used to decode the coded block in the current slice indicates that the decoding tool cannot be used to decode the current slice, the BIO decoding tool is not used or is disabled to decode the current coded block.

[0175] In one embodiment, when the value of the flag indicating whether the BIO decoding tool can be used to decode the coded block in the current slice does not exist, it is inferred that the value is the same as the value of the flag indicating whether the bi-direction optical flow (BIO) decoding tool / technique can be used to decode the image in the current coded video sequence.

[0176] In one embodiment, a method for encoding a video bitstream is disclosed. The video bitstream includes at least one picture, each picture includes multiple slices. Each of the multiple slices is associated with a slice header including multiple syntax elements. Each of the multiple slices includes multiple coding blocks and multiple reference image lists. Each reference image list in the multiple reference image lists includes multiple reference images, which can be used to perform inter-frame prediction on the coding blocks in the current slice.

[0177] The method includes: determining whether a bi-direction optical flow (BIO) decoding tool / technology can be used to encode an image in a current encoded video sequence; parsing a parameter set to obtain resolution information of each image code stream; obtaining two reference image lists of a current stripe in a current image; parsing the reference image list of the current stripe to obtain an active reference image that can be used to decode a coding block in the current stripe; if at least one of the following conditions is met, restricting that the BIO decoding tool cannot be used to encode the coding block in the current stripe: the BIO decoding tool cannot be used to encode an image in the current encoded video sequence, and the resolution of the current image is different from the resolution of at least one reference image.

[0178] In one embodiment, a method for decoding a video bitstream is disclosed. The bitstream includes at least one picture, each picture includes multiple slices. Each of the multiple slices is associated with a slice header including multiple syntax elements. Each of the multiple slices includes multiple coding blocks and multiple reference picture lists. Each reference picture list in the multiple reference picture lists includes multiple reference pictures, which can be used to perform inter-frame prediction on the coding blocks in the current slice.

[0179] The method comprises: parsing a parameter set to obtain a flag indicating whether a bi-direction optical flow (BIO) decoding tool / technique can be used to decode an image in a current encoded video sequence; parsing a parameter set to obtain a flag indicating whether a bi-direction optical flow (BIO) decoding tool / technique can be used to decode an image of a reference parameter set, wherein the parameter set refers to a picture parameter set (PPS).

[0180] In one embodiment, when the value of the flag indicating whether the BIO decoding tool can be used to decode the image of the reference PPS indicates that the decoding tool cannot be used, the BIO decoding tool is not used or is disabled to decode the current coding block.

[0181] In one embodiment, a method for encoding a video bitstream is disclosed. The video bitstream includes at least one picture, each picture includes multiple slices. Each of the multiple slices is associated with a slice header including multiple syntax elements. Each of the multiple slices includes multiple coding blocks and multiple reference image lists. Each reference image list in the multiple reference image lists includes multiple reference images, which can be used to perform inter-frame prediction on the coding blocks in the current slice.

[0182] In one embodiment, the method includes: determining whether a bi-direction optical flow (BIO) decoding tool / technique can be used to encode an image in a current encoded video sequence; determining whether a bi-direction optical flow (BIO) decoding tool / technique can be used to encode an image that references a current PPS; and when the BIO decoding tool cannot be used to encode an image in the current encoded sequence, restricting the BIO decoding tool from being used to encode an image that references the current PPS.

[0183] The following syntax and semantics may be used to implement the embodiments disclosed herein. The following description is expanded relative to the base text, which refers to the latest VVC draft specification. In other words, only the new additions are described below, and the text in the base text applies as is and is not mentioned below. New text relative to the base text is shown in bold, and deleted text is shown in italics.

[0184] The construction process of the reference image list is updated as follows:

[0185] The reference picture lists RefPicList[0] and RefPicList[1] are constructed as follows:

[0186]

[0187]

[0188] The flag that determines whether to use BIO is derived as follows.

[0189] Let predSamplesL0 L 、predSamplesL1 L and predSamplesIntra L Let predSamplesL0 be the (cbWidth(width))x(cbHeight(height)) array of predicted luminance sample values. Cb 、predSamplesL1 Cb 、predSamplesL0 Cr 、predSamplesL1 Cr 、predSamplesIntra Cb and predSamplesIntra Cr is a (cbWidth / 2)x(cbHeight / 2) array of predicted chroma sample values.

[0190] -The variable currPic represents the current image, and the variable bdofFlag is derived as follows:

[0191] - If all of the following conditions are true, bdofFlag is set equal to TRUE.

[0192] -sps_bdof_enabled_flag is equal to 1.

[0193] -predFlagL0[xSbIdx][ySbIdx] and predFlagL1[xSbIdx][ySbIdx] are both equal to 1.

[0194] -DiffPicOrderCnt(currPic, RefPicList[0][refIdxL0])*DiffPicOrderCnt(currPic, RefPicList[1][refIdxL1])

[0195] Less than 0.

[0196] -MotionModelIdc[xCb][yCb] is equal to 0.

[0197] -merge_subblock_flag[xCb][yCb] is equal to 0.

[0198] -BcwIdx[xCb][yCb] is equal to 0.

[0199] - luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are both equal to 0.

[0200] -cbHeight is greater than or equal to 8.

[0201] -cIdx is equal to 0.

[0202] -RefPicSameSizeFlag[0][refIdxL0] is equal to 1.

[0203] -RefPicSameSizeFlag[1][refIdxL1] is equal to 1.

[0204] - Otherwise, bdofFlag is set to FALSE.

[0205] The syntax and semantics of a sequence parameter set are as follows:

[0206]

[0207] sps_bdof_enabled_flag is equal to 0, indicating that bidirectional optical flow inter-frame prediction is disabled. sps_bdof_enabled_flag is equal to 1, indicating that bidirectional optical flow inter-frame prediction is enabled.

[0208] The syntax and semantics of a stripe header are as follows:

[0209]

[0210] slice_bdof_enabled_flag is equal to 0, indicating that bidirectional optical flow inter-frame prediction is disabled for the current slice. slice_bdof_enabled_flag is equal to 1, indicating that bidirectional optical flow inter-frame prediction is enabled for the current slice. When the value of slice_bdof_enabled_flag does not exist, it is inferred that the value is equal to the value of sps_bdof_enabled_flag.

[0211] The flag that determines whether to use BIO is derived as follows.

[0212] Let predSamplesL0 L 、predSamplesL1 L and predSamplesIntra L Let predSamplesL0 be the (cbWidth(width))x(cbHeight(height)) array of predicted luminance sample values. Cb 、predSamplesL1 Cb 、predSamplesL0 Cr 、predSamplesL1 Cr 、predSamplesIntra Cb and predSamplesIntra Cr is a (cbWidth / 2)x(cbHeight / 2) array of predicted chroma sample values.

[0213] -The variable currPic represents the current image, and the variable bdofFlag is derived as follows:

[0214] - If all of the following conditions are true, bdofFlag is set equal to TRUE.

[0215] -sps_bdof_enabled_flag is equal to 1.

[0216] -slice_bdof_enabled_flag is equal to 1.

[0217] -predFlagL0[xSbIdx][ySbIdx] and predFlagL1[xSbIdx][ySbIdx] are both equal to 1.

[0218] -DiffPicOrderCnt(currPic, RefPicList[0][refIdxL0])*DiffPicOrderCnt(currPic, RefPicList[1][refIdxL1])

[0219] Less than 0.

[0220] -MotionModelIdc[xCb][yCb] is equal to 0.

[0221] -merge_subblock_flag[xCb][yCb] is equal to 0.

[0222] -BcwIdx[xCb][yCb] is equal to 0.

[0223] - luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are both equal to 0.

[0224] -cbHeight is greater than or equal to 8.

[0225] -cIdx is equal to 0.

[0226] - Otherwise, bdofFlag is set to FALSE.

[0227] The syntax and semantics of a sequence parameter set are as follows:

[0228]

[0229] sps_bdof_enabled_flag is equal to 0, indicating that bidirectional optical flow inter-frame prediction is disabled. sps_bdof_enabled_flag is equal to 1, indicating that bidirectional optical flow inter-frame prediction is enabled.

[0230] Image parameter set syntax and semantics.

[0231]

[0232] pps_bdof_enabled_flag is equal to 0, indicating that bidirectional optical flow inter-frame prediction is disabled for the picture referring to the PPS. pps_bdof_enabled_flag is equal to 1, indicating that bidirectional optical flow inter-frame prediction is enabled for the picture referring to the PPS.

[0233] When the value of sps_bdof_enabled_flag is equal to 0, the value of pps_bdof_enabled_flag should also be equal to 0. This is a requirement for code stream consistency.

[0234] The flag that determines whether to use BIO is derived as follows.

[0235] Let predSamplesL0 L 、predSamplesL1 L and predSamplesIntra L Let predSamplesL0 be the (cbWidth(width))x(cbHeight(height)) array of predicted luminance sample values. Cb 、predSamplesL1 Cb 、predSamplesL0 Cr 、predSamplesL1 Cr 、predSamplesIntra Cb and predSamplesIntra Cr is a (cbWidth / 2)x(cbHeight / 2) array of predicted chroma sample values.

[0236] -The variable currPic represents the current image, and the variable bdofFlag is derived as follows:

[0237] - If all of the following conditions are true, bdofFlag is set equal to TRUE.

[0238] -sps_bdof_enabled_flag is equal to 1.

[0239] -pps_bdof_enabled_flag is equal to 1.

[0240] -predFlagL0[xSbIdx][ySbIdx] and predFlagL1[xSbIdx][ySbIdx] are both equal to 1.

[0241] -DiffPicOrderCnt(currPic, RefPicList[0][refIdxL0])*DiffPicOrderCnt(currPic, RefPicList[1][refIdxL1])

[0242] Less than 0.

[0243] -MotionModelIdc[xCb][yCb] is equal to 0.

[0244] -merge_subblock_flag[xCb][yCb] is equal to 0.

[0245] -BcwIdx[xCb][yCb] is equal to 0.

[0246] - luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are both equal to 0.

[0247] -cbHeight is greater than or equal to 8.

[0248] -cIdx is equal to 0.

[0249] - Otherwise, bdofFlag is set to FALSE.

[0250] Fig.12 A schematic diagram of a video decoding device 1200 (for example, the above-mentioned video encoder 20 or the above-mentioned video decoder 30) provided in an embodiment of the present application. The video decoding device 1200 is suitable for implementing the disclosed embodiments described herein. The video decoding device 1200 includes: an input port 1210 and a receiving unit (Rx) 1220 for receiving data; a processor, a logic unit or a central processing unit (CPU) 1230 for processing data; a sending unit (Tx) 1240 and an output port 1250 for sending data; and a memory 1260 for storing data. The video decoding device 1200 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component, coupled to the input port 1210, the receiving unit 1220, the sending unit 1240 and the output port 1250, as an outlet or an inlet of an optical signal or an electrical signal.

[0251] The processor 1230 is implemented by hardware and software. The processor 1230 can be implemented as one or more CPU chips, cores (for example, as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 1230 communicates with the input port 1210, the receiving unit 1220, the sending unit 1240, the output port 1250, and the memory 1260. The processor 1230 includes a decoding module 1270. The decoding module 1270 implements the above-mentioned disclosed embodiments. For example, the decoding module 1270 implements, processes, prepares, or provides various encoding and decoding functions. Therefore, the inclusion of the decoding module 1270 substantially improves the function of the video decoding device 1200 and affects the conversion of the video decoding device 1200 to different states. Alternatively, the decoding module 1270 is implemented as instructions stored in the memory 1260 and executed by the processor 1230.

[0252] The video decoding device 1200 may also include an input and / or output (I / O) device 1280 for exchanging data with a user. The I / O device 1280 may include an output device, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O device 1280 may also include an input device, such as a keyboard, a mouse, a trackball, etc., and / or a corresponding interface for interacting with these output devices.

[0253] The memory 1260 includes one or more hard disks, tape drives, and solid-state drives, and can be used as an overflow data storage device to store such programs when such programs are selected for execution, as well as to store instructions and data read during program execution. The memory 1260 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random access memory (SRAM).

[0254] Fig.131 is a schematic diagram of an embodiment of a decoding unit 1300. In one embodiment, the decoding unit 1300 is implemented in a video decoding device 1302 (for example, the above-mentioned video encoder 20 or the above-mentioned video decoder 30). The video decoding device 1302 includes a receiving unit 1301. The receiving unit 1301 is used to receive an image for encoding or receive a code stream for decoding. The video decoding device 1302 includes a sending unit 1307, which is coupled to the receiving unit 1301. The sending unit 1307 is used to send the code stream to a decoder or send the decoded image to a display unit (for example, one of the above-mentioned I / O devices 1280).

[0255] The video decoding device 1302 includes a storage unit 1303. The storage unit 1303 is coupled to at least one of the receiving unit 1301 or the sending unit 1307. The storage unit 1303 is used to store instructions. The video decoding device 1302 also includes a processing unit 1305. The processing unit 1305 is coupled to the storage unit 1303. The processing unit 1305 is used to execute the instructions stored in the storage unit 1303 to perform the method disclosed herein.

[0256] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present application, these methods may include other steps, and certain steps may be omitted or combined.

[0257] Although the present application provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in other various specific forms without departing from the spirit or scope of the present application. The examples of the present application will be considered illustrative rather than restrictive and are not intended to be limited to the details described herein. For example, various elements or components may be combined or integrated in another system, or some features may be omitted or not implemented.

[0258] In addition, without departing from the scope of the present application, the techniques, systems, subsystems and methods described and illustrated as independent or separate in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or described as coupled or directly coupled or communicating with each other may also be coupled or communicated indirectly via an interface, device or intermediate component, either electrically, mechanically or otherwise. Other variations, substitutions and modified examples may be determined by those skilled in the art without departing from the spirit and scope disclosed herein.

Claims

1. A method for decoding a video code stream, characterized in that: The method comprises: Parsing an encoded video bitstream, the video bitstream comprising syntax elements associated with a current picture and quantized transform coefficients of a current block of the current picture; determining whether a resolution of the current picture being decoded is the same as a resolution of a reference picture identified by a reference picture list associated with the current picture; When it is determined that the resolution of the current image is the same as the resolution of each of the reference images, enabling bi-direction optical flow (BDOF) for the current block of the current image; disabling the BDOF for a current block of the current image when it is determined that the resolution of the current image is different from the resolution of any of the reference images; Dequantizing and inverse transforming the quantized transform coefficients to obtain a reconstructed residual block; Generate the reference image for the current image according to the reference image list, and perform inter-frame prediction on the current block according to the reference block of the reference image and the syntax element to obtain a prediction block; A reconstructed block is obtained according to the prediction block and the reconstructed residual block.

2. The method according to claim 1, characterized in that Enabling the BDOF includes setting a BDOF flag to a first value, and disabling the BDOF includes setting the BDOF flag to a second value.

3. The method according to claim 1 or 2, characterized in that: The method further comprises generating the reference image for the current image according to the reference image list in a bidirectional inter-frame prediction mode.

4. The method according to claim 1 or 2, characterized in that: The method further includes selectively enabling or disabling the BDOF for blocks in the plurality of images based on whether a resolution of each image in the plurality of images is the same as or different from a resolution of a reference image associated with the image.

5. The method according to claim 1 or 2, characterized in that: The method further includes enabling reference picture resampling (RPR) for an entire coded video sequence (CVS) including the current picture when the BDOF is disabled.

6. The method according to claim 1 or 2, characterized in that: The resolution of the current image is set in a parameter set of the encoded video code stream, and the current block is obtained from a slice of the current image.

7. The method according to claim 1 or 2, characterized in that: The method further includes: displaying an image generated according to the current block on a display screen of an electronic device.

8. A method for encoding a video stream, characterized in that: The method comprises: determining whether a resolution of a current picture being encoded is the same as a resolution of a reference picture identified in a reference picture list associated with the current picture; When it is determined that the resolution of the current image is the same as the resolution of each of the reference images, enabling bi-direction optical flow (BDOF) for the current block of the current image; disabling the BDOF for a current block of the current image when it is determined that the resolution of the current image is different from the resolution of any of the reference images; Generate the reference image for the current image according to the reference image list, and perform inter-frame prediction on the current block according to the reference block of the reference image to obtain a syntax element associated with the current image and a prediction block; Obtaining a residual block according to the prediction block and the current block; Transforming and quantizing the residual block to obtain quantized transform coefficients; The syntax elements and the quantized transform coefficients are encoded into a video bitstream.

9. The method according to claim 8, characterized in that Generating the reference image for the current image according to the reference image list, and performing inter-frame prediction on the current block according to the reference block of the reference image includes: Determine a motion vector of the current image according to the reference image; encoding the current image according to the motion vector; The method further includes decoding the current image by a hypothetical reference decoder.

10. The method according to claim 8 or 9, characterized in that: Enabling the BDOF includes setting a BDOF flag to a first value, and disabling the BDOF includes setting the BDOF flag to a second value.

11. The method according to claim 8 or 9, characterized in that: The method further comprises generating the reference image for the current image according to a reference image list in a bidirectional inter-frame prediction mode.

12. The method according to claim 8 or 9, characterized in that: The method further includes selectively enabling or disabling the BDOF for blocks in the plurality of images based on whether a resolution of each image in the plurality of images is the same as or different from a resolution of a reference image associated with the image.

13. The method according to claim 8 or 9, characterized in that: The method further includes enabling reference picture resampling (RPR) for an entire coded video sequence (CVS) including the current picture even when the BDOF is disabled.

14. The method according to claim 8 or 9, characterized in that The method further includes: sending a video code stream including the current block to a video decoder.

15. A decoding device, characterized in that: include: A receiver for receiving an encoded video bitstream, the video bitstream representing syntax elements associated with a current picture and quantized transform coefficients of a current block of the current picture; a memory coupled to the receiver, wherein the memory stores instructions; A processor is coupled to the memory, wherein the processor is configured to execute the instructions so that the decoding device performs the following operations: determining whether a resolution of the current picture being decoded is the same as a resolution of a reference picture identified by a reference picture list associated with the current picture; When it is determined that the resolution of the current image is the same as the resolution of each of the reference images, enabling bi-direction optical flow (BDOF) for the current block of the current image; disabling the BDOF for a current block of the current image when it is determined that the resolution of the current image is different from the resolution of any of the reference images; Dequantizing and inverse transforming the quantized transform coefficients to obtain a reconstructed residual block; Generate the reference image for the current image according to the reference image list, and perform inter-frame prediction on the current block according to the reference block of the reference image and the syntax element to obtain a prediction block; A reconstructed block is obtained according to the prediction block and the reconstructed residual block.

16. The decoding device according to claim 15, characterized in that When the BDOF is disabled, reference picture resampling (RPR) is enabled for the entire coded video sequence (CVS) including the current picture.

17. The decoding device according to claim 15 or 16, characterized in that: The decoding device also includes a display for displaying an image generated according to the current block.

18. An encoding device, characterized in that: include: a memory storing instructions; A processor is coupled to the memory, wherein the processor is configured to execute the instructions so that the encoding device performs the following operations: determining whether a resolution of a current picture being encoded is the same as a resolution of a reference picture identified in a reference picture list associated with the current picture; When it is determined that the resolution of the current image is the same as the resolution of each of the reference images, enabling bi-direction optical flow (BDOF) for the current block of the current image; disabling the BDOF for a current block of the current image when it is determined that the resolution of the current image is different from the resolution of any of the reference images; Generate the reference image for the current image according to the reference image list, and perform inter-frame prediction on the current block according to the reference block of the reference image to obtain a syntax element associated with the current image and a prediction block; Obtaining a residual block according to the prediction block and the current block; Transforming and quantizing the residual block to obtain quantized transform coefficients; Encoding the syntax elements and the quantized transform coefficients into a video bitstream; A transmitter is coupled to the processor, wherein the transmitter is used to send the video code stream to a video decoder.

19. The encoding device according to claim 18, characterized in that Even when BDOF is disabled, reference picture resampling (RPR) is enabled for the entire coded video sequence (CVS) including the current picture.

20. The encoding device according to claim 18 or 19, characterized in that Before the transmitter sends the video code stream to the video decoder, the memory stores the code stream.

21. A decoding device, characterized in that: include: A receiver, used for receiving an image for encoding or receiving a code stream for decoding; A transmitter, coupled to the receiver, wherein the transmitter is used to send the code stream to a decoder or send the decoded image to a display; a memory coupled to at least one of the receiver or the transmitter, wherein the memory is used to store instructions; A processor is coupled to the memory, wherein the processor is configured to execute the instructions stored in the memory to perform the method according to any one of claims 1 to 7 and claims 8 to 14.

22. The decoding device according to claim 21, characterized in that: The decoding device also includes a display for displaying images.

23. A coding and decoding system, characterized in that: include: Encoder; A decoder that communicates with the encoder; wherein the encoder or the decoder comprises a decoding device, an encoding device or a decoding device according to any one of claims 15 to 22.

24. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed on a computer, the computer executes the video code stream decoding method as described in any one of claims 1-7.

25. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed on a computer, the computer executes the method for encoding a video code stream as described in any one of claims 8 to 14.

26. A method for storing a video code stream, characterized in that: include: receiving an encoded video bitstream, the video bitstream comprising a syntax element associated with a current image, the syntax element comprising a bi-direction optical flow (BDOF) flag and a resolution of the current image, wherein: when the resolution of the current image is the same as a resolution of each of a plurality of reference images, the BDOF flag is enabled, the plurality of reference images being a plurality of reference images identified by a reference image list associated with the current image; and when the resolution of the current image is different from a resolution of any of the reference images, the BDOF flag is disabled; The video code stream is stored in a storage medium.

27. The method according to claim 26, characterized in that When the BDOF flag is disabled, reference picture resampling (RPR) of the entire coded video sequence (CVS) including the current picture is enabled.

28. The method according to claim 26 or 27, characterized in that The resolution of the current image is set in a parameter set of the encoded video code stream, and the current block is obtained from a slice of the current image.

29. A device for storing video code streams, characterized in that: include: A receiver, configured to receive an encoded video stream, the video stream comprising a bi-direction optical flow (BDOF) flag and a resolution of a current image, wherein: when the resolution of the current image is the same as the resolution of each of a plurality of reference images, the BDOF flag is enabled, the plurality of reference images being a plurality of reference images identified by a reference image list associated with the current image; and when the resolution of the current image is different from the resolution of any of the reference images, the BDOF flag is disabled; A memory is used to store the video code stream.

30. The device according to claim 29, characterized in that When the BDOF flag is disabled, reference picture resampling (RPR) of the entire coded video sequence (CVS) including the current picture is enabled.

31. The device according to claim 29 or 30, characterized in that The resolution of the current image is set in a parameter set of the encoded video code stream, and the current block is obtained from a slice of the current image.

32. A method for transmitting a code stream, characterized in that: include: Obtaining an encoded video stream from a memory, the video stream comprising a bi-direction optical flow (BDOF) flag and a resolution of a current image, wherein: the BDOF flag is enabled when the resolution of the current image is the same as the resolution of each of a plurality of reference images, the plurality of reference images being a plurality of reference images identified by a reference image list associated with the current image; When the resolution of the current image is different from the resolution of any of the reference images, the BDOF flag is disabled; Send the video code stream.

33. The method according to claim 32, characterized in that When the BDOF flag is disabled, reference picture resampling (RPR) of the entire coded video sequence (CVS) including the current picture is enabled.

34. The method according to claim 32 or 33, characterized in that The resolution of the current image is set in a parameter set of the encoded video code stream, and the current block is obtained from a slice of the current image.

35. A device for transmitting a code stream, characterized in that: include: a processor, configured to obtain an encoded video stream from a memory, the video stream comprising a bi-direction optical flow (BDOF) flag and a resolution of a current image, wherein: when the resolution of the current image is the same as the resolution of each of a plurality of reference images, the BDOF flag is enabled, the plurality of reference images being a plurality of reference images identified by a reference image list associated with the current image; and when the resolution of the current image is different from the resolution of any of the reference images, the BDOF flag is disabled; A transmitter is used to send the video code stream.

36. The device according to claim 35, characterized in that When the BDOF flag is disabled, reference picture resampling (RPR) of the entire coded video sequence (CVS) including the current picture is enabled.

37. Apparatus according to claim 35 or 36, characterised in that The resolution of the current image is set in a parameter set of the encoded video code stream, and the current block is obtained from a slice of the current image.

Images