Prediction for video encoding and decoding using external reference

Abstract

Various embodiments relate to a video coding system in which some elements required for decoding are generated according to an unspecified process within the video coding system. This process is hereinafter referred to as an "external" process. This external process can generate "external" reference pictures to be used by a decoder adapted to use these external pictures. Coding methods, decoding methods, coding devices, and decoding devices based on this post-processing approach are proposed.

Description

Technical Field

[0001] The present disclosure is in the field of video compression, and at least one embodiment more particularly relates to video prediction using an external reference. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transforms to exploit spatial and temporal redundancies in video content. Generally, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, and the difference between the original image block and the predicted image block (typically represented as a prediction error or prediction residual) is then transformed, quantized, and entropy coded. During encoding, the original image block is typically partitioned / separated into sub-blocks using various partitioning schemes, such as quadtrees. To reconstruct the video, the compressed data is decoded through the inverse processes of prediction, transform, quantization, and entropy coding. Summary of the Invention

[0003] In at least one embodiment, it is proposed to perform video prediction based on reference elements that are generated according to a process that is not specified within the video coding system itself and is therefore outside the scope of conventional codec specifications.

[0004] A first aspect relates to a method for decoding video data. For a current picture of the video, the method includes: obtaining information representing an external reference picture, and reconstructing the current picture based on the external reference picture.

[0005] A second aspect relates to an apparatus for decoding video data, the apparatus comprising a decoder. For a current picture of the video, the decoder is configured to: obtain information representing an external reference picture, and reconstruct the current picture based on the external reference picture.

[0006] In variations of the first and second aspects, the video is multi-view video, the external reference picture includes texture of the first view and a motion vector map representing disparity information between the first view and the second view, and reconstruction of the second view is performed based on motion compensation using the texture and disparity information based on the first view. Further variations include: copying the first view into a decoded picture buffer of the second view; associating the first view with a picture order count; copying the disparity information into a motion information map of a concatenated reference picture; and predicting the second view based on the copied information. In further variations, prediction is performed using a temporal motion vector prediction mode.

[0007] A third aspect relates to a method for encoding video data. For a current picture of a video, the method includes: generating information representing an external reference picture, and encoding the information representing the external reference picture using the encoded video.

[0008] A fourth aspect relates to an apparatus for encoding a video, the apparatus comprising a decoder, wherein for a current picture of the video, the decoder is configured to: obtain information representing an external reference picture, and reconstruct the current picture based on the external reference picture.

[0009] In variations of the third and fourth aspects, the video is a multi-view video, the external reference picture includes a texture of a first view and a motion vector map representing disparity information between the first view and the second view, and reconstruction of the second view is based on motion compensation using the texture and disparity information based on the first view.

[0010] A fifth aspect relates to a video signal comprising a bitstream having video content and high-level syntax information, wherein the bitstream is generated according to the third aspect or the fourth aspect.

[0011] One or more embodiments of the present invention further provide a non-transitory computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to at least a portion of any of the above methods. This embodiment further provides a computer program product including instructions for performing at least a portion of any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A block diagram of a video encoder according to an embodiment is shown.

[0013] Figure 2 A block diagram of a video decoder according to an embodiment is shown.

[0014] Figure 3 A block diagram is shown of an example of a system in which various aspects and embodiments may be implemented.

[0015] Figure 4 The principle of a decoded picture buffer for a conventional block-based video coding standard is shown.

[0016] Figure 5 The principle of the temporal motion vector prediction mode of conventional block-based video coding standards is shown.

[0017] Figure 6 The principle of adaptive motion vector prediction mode of conventional block-based video coding standards is shown.

[0018] Figure 7 The principle of temporal scalability of conventional block-based video coding standards is shown.

[0019] Figure 8 An example of spatial scalability of conventional block-based video coding standards is shown.

[0020] Figure 9 An example implementation of using external reference pictures in a multi-view application is shown.

[0021] Figure 10 An example implementation of a multi-view application using the TMVP mode is shown.

[0022] Figure 11 A variant implementation of a multi-view application using the TMVP mode is shown.

[0023] Figure 12 A variant implementation is shown where the "external reference picture" is not an ILRP.

[0024] Figure 13 An embodiment is shown in which an "external reference picture" is generated by an external process. DETAILED DESCRIPTION

[0025] Various embodiments relate to a video coding system in which some elements required for decoding, and more specifically, some elements required for prediction, are generated according to an unspecified process within the video coding system. This process is hereinafter referred to as an "external" process. This external process can generate "external" reference pictures to be used by a decoder adapted to use these external pictures. Coding methods, decoding methods, coding devices, and decoding devices based on this post-processing approach are proposed.

[0026] In addition, although the present invention describes principles related to specific drafts of VVC (Versatile Video Coding) or HEVC (High Efficiency Video Coding) specifications, it is not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether previously developed or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0027] Figure 1 A video encoder 100 is shown. Variations of this encoder 100 are contemplated, but for clarity, the following describes encoder 100 without describing all contemplated variations. Prior to encoding, a video sequence may undergo pre-encoding processing (101), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0028] In an encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded, for example, using intra mode or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use for encoding the unit and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0029] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0030] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering, adaptive loop filter (ALF) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0031] Figure 2 A block diagram of a video decoder 200 is shown. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs a decoding step that is the reverse of the encoding step. The encoder 100 also typically performs video decoding as part of encoding the video data. Specifically, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition (235) the picture based on the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0032] The decoded picture may also undergo post-decoding processing (285), such as an inverse color transform (e.g., a transform from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0033] Figure 3 Shown is a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 can be embodied as the equipment comprising the various components described below, and is configured to perform one or more aspects described in this document. Examples of such equipment include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 can be embodied in single integrated circuit (IC), multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing element and encoder / decoder elements of system 1000 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0034] System 1000 includes at least one processor 1010 configured to execute instructions loaded therein to implement, for example, various aspects described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0035] System 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded or decoded video, for example, and may include its own processor and memory. Encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. In addition, encoder / decoder module 1030 can be implemented as a separate component of system 1000, or can be incorporated into processor 1010 as a combination of hardware and software known to those skilled in the art.

[0036] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0037] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Team (JVET)).

[0038] Inputs to the elements of system 1000 may be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 3 Other examples not shown include composite video, which is not shown.

[0039] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as are known in the art. For example, the RF section may be associated with elements adapted to: (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) again band-limit to a narrower frequency band to select a signal band, which in some embodiments may be referred to as a channel, (iv) demodulate the downconverted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select the desired packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and by filtering, down-conversion and filtering to required frequency band again to perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some elements in these elements, and / or add other elements of execution similar or different functions.Adding element can be included in and inserts element between existing element, for example, inserts amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0040] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processor 1010, as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 1010, as desired. The demodulated stream, error-corrected stream, and demultiplexed stream are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0041] The various components of system 1000 may be disposed within an integrated housing. Within the integrated housing, the various components may interconnect and transmit data therebetween using a suitable connection arrangement 1140 (e.g., an internal bus as known in the art, including an inter-IC (I2C) bus, wiring, and printed circuit boards).

[0042] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0043] In various embodiments, a wireless network such as a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to transmit data streams or otherwise provide the system 1000. The Wi-Fi signals of these embodiments are received by a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other cloud communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which delivers data via the HDMI connection of an input block 1130. Still other embodiments use the RF connection of an input block 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0044] System 1000 can provide output signals to various output devices (including display 1100, speaker 1110 and other peripheral devices 1120). The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display and / or a foldable display. The display 1100 can be used for a television, a tablet, a notebook, a cellular phone (mobile phone) or other device. The display 1100 can also be integrated with other components (for example, as in a smart phone), or it can be separate (for example, an external monitor for a laptop computer). In various examples of the embodiments, other peripheral devices 1120 include one or more of a stand-alone digital video disc (or digital versatile disc, both terms are DVR), a disc player, a stereo system and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0045] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speakers 1110 can be integrated into a single unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0046] Alternatively, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 are optionally separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection (including, for example, an HDMI port, a USB port, or a COMP output).

[0047] These embodiments may be executed by the processor 1010 or by computer software implemented in hardware or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0048] The technical field of the invention relates to reference picture buffers (180, 280), also called decoded picture buffers, and motion estimation (175) and motion compensation (170, 275) elements of a video encoder or decoder.

[0049] Figure 4 The principle of a decoded picture buffer for a conventional block-based video coding standard is shown. For example, in a conventional block-based hybrid video coding standard such as VVC, a decoded picture buffer (DPB) stores different possible reference pictures that can be used to decode the current picture. Pictures are displayed according to a picture order count (POC) that provides a temporal reference for the pictures. The DPB can contain reconstructed texture and motion information maps (MV-maps aka MV-info, aka collocated motion) associated with each picture. The motion information contains motion vectors, reference indices, and reference POCs (pocRef) that are typically used to decode blocks of the current picture. Typically, the motion information is stored at a block-based resolution (e.g., a 4x4 resolution).

[0050] During the decoding process of the current picture, the reconstructed motion information used to construct the (current) block is stored in the MV-map for possible further use in predicting the MV of other blocks of the current picture or in case the current picture is stored in the DPB (marked as "used for reference").

[0051] Reference picture list 0 (L0 shown to the left of the current picture) contains a list of reference pictures used to decode P pictures. Reference picture list 0 and reference picture list 1 (L1 shown to the right of the current picture) contain lists of reference pictures used to decode B pictures.

[0052] Figure 5The principle of the temporal motion vector prediction mode of a conventional block-based video coding standard is shown. In practice, in conventional video codecs, motion information can be used in the so-called temporal motion vector prediction mode (TMVP). In this mode, for each slice or picture, one reference of the DPB is signaled as a collocation ("coLoc") shown in the figure by a dotted rectangle. We denote pocCol as the POC of the "coLoc" reference picture. In the current picture, during reconstruction of the current block, the current block can use the collocated MV in the coLoc reference MV map to build a motion compensated prediction (Equation 2). For example, the TMVP predictor is built from the collocated MV (mvCol) by a rescaling process (Equation 1). We denote pocRef as the POC of the reference picture and pocColRef as the POC of the reference picture corresponding to mvCol ( Figure 2 ) and is used to reconstruct the collocated block in the “coLoc” picture:

[0053] TMVP = MV × (pocCur - pocRef) / (pocCol - pocColRef) (Equation 1)

[0054] pred-Ima[pocCur][x]=Ima[pocRef][x+TMVP] (Equation 2)

[0055] pred-Ima[pocCol][x]=ima[pocColRef][x+mvCol] (Equation 3)

[0056] In the specific case where (pocCur - pocRef) equals (pocCol - pocColRef), TMVP = MV regardless of the value of (pocCur - pocRef), thus avoiding the undefined result of Equation 1 if pocCol equals pocColRef.

[0057] Figure 6 The principle of the adaptive motion vector prediction mode of a conventional block-based video coding standard is shown. In practice, motion information can also be used in (another) so-called adaptive motion vector prediction mode (AMVP). In the AMVP mode, a motion vector (MV) predictor list is constructed from the reconstructed neighborhood. The reference index (pocRef) and the index of the MV predictor in the MV predictor list are signaled in the bitstream. In the figure, A denotes a neighboring reconstructed candidate of the current block, mvA denotes the reconstructed MV of block A, and refPocA denotes the POC of the reference picture of the reference index of A. Depending on the video codec, the motion vector predictor mvP can be equal to mvA or can be constructed from mvA by rescaling (Equation 4).

[0058] mvP = mvA × (pocCur - pocRef) / (pocCur - pocRefA) (Equation 4)

[0059] Figure 7 The scalability principle of conventional block-based video coding standards is shown. When a video codec uses scalability, the video encoder can generate several bitstreams (named layers) that allow encoding of video sequences with enhanced quality or additional features with respect to the base layer stream or another layer. For example, the reconstructed image can have enhanced resolution (spatial scalability), quality (SNR scalability), frame rate (temporal scalability), color gamut (color gamut scalability, high dynamic range scalability), bit depth (bit depth scalability), additional views (multi-view scalability), etc. Scalable video codecs exploit the ability to predict blocks with reconstructed images from another bitstream / layer on which it depends. For example, the figure depicts temporal scalability, where it is possible to encode additional temporal pictures of layer 1 so that the reconstructed reference frames can be predicted from the base layer 0 stream (the reference frames replicated in the figure).

[0060] Figure 8 An example of spatial scalability of conventional block-based video coding standards is shown. In this example, a reconstructed picture from the base layer (layer-0) can be rescaled (e.g., upsampled) and used as an additional reference frame for constructing inter-frame prediction for the current layer (layer-1). Such additional reference frames are called inter-layer reference pictures (ILRPs) and are stored in sub-segments (sub-DPBs) of the decoded picture buffer. Inter-layer reference pictures (ILRPs) are temporally collocated with the current picture of the current layer; in other words, they have the same POC.

[0061] It is anticipated that future systems and / or applications will need to be able to use reference texture and / or motion that may be available through other (external) means, i.e. outside the scope of current video standards. For example:

[0062] - A picture of the current layer can be predicted using a reconstructed picture decoded from another layer coded with a different standard (e.g.: layer-1 is VVC coded, layer-0 is AVC coded).

[0063] - One additional reference picture generated by an application-based module can be used. For example, an additional "virtual reference" frame can be used, which is calculated based on the two reference pictures of the DPB using a deep learning-based frame rate up-conversion algorithm.

[0064] - A reference picture can be replicated with different weighted prediction parameters to manage local brightness compensation or improve motion accuracy. This principle can be extended with a more sophisticated brightness variation model.

[0065] Multi-view content can be used to provide disparity for immersive content. There is sufficient redundancy between views to allow for this disparity. When views are compressed, one view can be a good predictor of the next. VVC lacks a mechanism to compute a prediction of a view based on the texture and depth of another view. To exploit this potential for predicting one view from another, a method is needed to signal that a view in the DPB has been generated by an external device acting as a predictor.

[0066] Different video codecs support different mechanisms to achieve scalability. SHVC proposes the use of multi-layer coding and advanced syntax, and defines the external layer as the base layer with nuh_layer_id equal to 0. However, only the decoding of pictures with nuh_layer_id greater than 0 can use the "external base layer". This is signaled in the Video Parameter Set (VPS) with "vps_base_layer_internal_flag = 0".

[0067] In conventional scalable codecs, an "inter-layer reference picture" (ILRP) has the same POC (same access unit) as the current picture. The ILRP is placed in a sub-DPB that contains pictures of the DPB that have the same POC as the current picture.

[0068] For example, in case of multi-view scalability, one may have two "inter-layer reference pictures" corresponding to two different views.

[0069] Currently, contrary to what HEVC (3D-HEVC and SHVC extensions / annexes) has done, VVC has not been extended to support process-specific scalability features, nor has it been specifically optimized for compression of multi-view and depth content. In VVC, scalability is envisioned to be supported using only high-level syntax signaling.

[0070] The VVC draft specification describes a "reference picture list structure" for signaling the composition of the DPB used to decode the current picture. The "reference picture list structure" syntax allows signaling information for constructing a reference picture POC list. This syntax is shown in Table 1.

[0071]

[0072] Table 1

[0073] The process used to construct the reference image POC consists of the following steps:

[0074] abs_delta_poc_st[listIdx][rplsIdx][i] specifies the value of the variable AbsDeltaPocSt[listIdx][rplsIdx][i] as shown below:

[0075] if(sps_weighted_pred_flag||sps_weighted_bipred_flag)

[0076] AbsDeltaPocSt[listIdx][rplsIdx][i]

[0077] =abs_delta_poc_st[listIdx][rplsIdx][i] (7-121)

[0078] else

[0079] AbsDeltaPocSt[listIdx][rplsIdx][i]

[0080] =abs_delta_poc_st[listIdx][rplsIdx][i]+1

[0081] In fact, it is stated in the specification that when weighted prediction is not enabled, no repeated reference pictures are required. Thus, when weighted prediction is not enabled, the ΔPOC value is not allowed to be zero.

[0082] Additionally, DeltaPocValSt is derived from AbsDeltaPocSt as follows:

[0083]

[0084] And the reference picture list derived from DeltaPocValSt is determined as follows:

[0085]

[0086]

[0087]

[0088] The embodiments described below are designed with the foregoing in mind. Figure 1 Encoder 100, Figure 2 The decoder 200 and Figure 3The system 1000 is suitable for implementing at least one of the embodiments described below, and more particularly for implementing a motion estimation (175) element and a motion compensation (170, 275) element of a video encoder or decoder using a reference picture buffer (180, 280).

[0089] In at least one embodiment, the video coding system uses external reference pictures to reconstruct the video. These external reference pictures are generated locally by an external process alongside the video encoding / decoding process and are not included in the coded stream. In at least one embodiment, this principle is applied to multiple views and allows the use of disparity equations to predict the current view from another view.

[0090] In at least one embodiment, external reference pictures are signaled using syntax elements that describe the reference picture list used to reconstruct the current picture (e.g., grouped with a "reference picture list structure" in the example of VVC syntax). For this purpose, an "external_ref_pic_flag" is defined to signal that the reference picture is provided by "external means not specified in the current specification."

[0091] An example of the corresponding high-level syntax is shown in Table 2. In all additional syntax elements described herein, new elements are identified by underlined text.

[0092] <![CDATA[ if(enabled_external_ref_pic_flag) ]]> <![CDATA[ external_ref_pic_flag[listIdx][rplsIdx][i] ]]> <![CDATA[ u(1) ]]>

[0093] Table 2

[0094] In a variant implementation, an additional flag "available_ref_pic_flag" is defined to signal that a reference picture is not available (neither present in the bitstream nor provided by an external device), but the VPS includes the information as if the reference picture was present in the bitstream.

[0095] In another variant, the flag is encoded only for inter-layer reference pictures, as shown in the high-level syntax example of Table 3.

[0096]

[0097] Table 3

[0098] In a variant implementation, the presence of the flags "external_ref_pic_flag" and / or "available_ref_pic_flag" depends on whether the reference picture is an ILRP and / or on whether the reference picture is short-term or long-term. For example, if it is a long-term reference picture, the flags are not present.

[0099] In at least one embodiment, the concept of "external reference pictures" is extended to any layer. This means:

[0100] - "External picture reference" can have nuh_layer_id not equal to 0

[0101] - and / or decoding of pictures with nuh_layer_id equal to 0 can use "external picture reference"

[0102] - and / or "external image references" may not be considered ILRP.

[0103] For example, in the example of VVC high-level syntax, this can be achieved by signaling whether the bitstream contains "external reference pictures" (enabled_external_ref_pic_flag) that may be added in the sub-DPB. For example, this flag can be encoded in the SPS or VPS, DPS, PPS or slice header. Table 4 shows such a flag in the SPS.

[0104]

[0105] Table 4

[0106] In a variant embodiment, for each layer, the VPS includes a new flag named vps_external_layer[i] to indicate whether the layer is external, as shown in Table 5.

[0107]

[0108]

[0109] Table 5

[0110] In at least one embodiment, the external reference pictures have a different POC (different access unit) than the current POC. In this case, ie when at least one external reference picture has a different POC than the current POC, the MV rescaling process may be applied.

[0111] In at least one embodiment, when the DPB contains several external reference pictures, two external reference pictures can be used for bidirectional prediction. This is independent of the POC value of the external reference pictures. In a variant, the external reference pictures used for bidirectional prediction should have a POC different from the current POC.

[0112] In at least one embodiment, several external reference pictures have the same POC as the current POC.

[0113] In at least one embodiment, several "external reference pictures" have the same POC (pocExt), which is different from the current POC.

[0114] To enable these implementations, the following modifications need to be made to the VVC syntax identified by the underlined text.

[0115] abs_delta_poc_st[listIdx][rplsIdx][i] specifies the variable

[0116] The value of AbsDeltaPocSt[listIdx][rplsId][i] is as follows:

[0117]

[0118] Regarding motion vectors, in SHVC, when the reference is IRLP, the value of the reconstructed motion vector is required to be equal to 0. However, in the case of multi-view prediction, this requirement is counterproductive because the reconstructed samples of the current view may not be spatially aligned with the reference view. Therefore, in at least one embodiment, a flag (e.g., named zero_mv_ref_flag) indicates whether this requirement applies. For example, the flag zero_mv_ref_flag can be encoded in the reference picture list structure or VPS.

[0119] In a variant, another flag or the same flag indicates whether the MV-map should be set to zero. In a variant implementation, this flag is only coded for "external reference pictures". In a variant implementation, this flag is only coded in the case of ILRP, so if inter_layer_ref_pics_flag[listIdx][rplsIdx][i]=0, this flag is not present (and therefore inferred to be false). In a variant implementation, to ensure the same behavior as in SHVC, this flag is not coded in the case of ILRP, and if inter_layer_ref_pics_flag[listIdx][rplsIdx][i]=1, this flag is inferred to be true.

[0120] Figure 9 An example implementation of using external reference pictures in a multi-view application is shown. In fact, in the case of a multi-view system, if the viewport information (sensor position in the scene, field of view, etc.) and view depth (depth information of each sample of the texture view) of the two views VA and VB are known, the disparity DAB between the two views VA and VB can be derived. The disparity information DAB allows the projection of VB texture samples into VA (e.g. Figure 9As shown in , and expressed by Equation 6). In this case, the model is the same as the classic motion compensation applied on one reference picture (Equation 4). Therefore, the VA texture can be predicted from the VB texture using disparity as motion information (MV), where the VB texture is the reference picture (Equation 5):

[0121] pred-Ima[pocCur][x]=Ima[pocRef][x+mv] (Equation 5)

[0122] pred-view-A[pocCur][x]=view-B[pocRef][x+DAB] (Equation 6)

[0123] Figure 10 Example implementations of multi-view applications using the TMVP mode are shown. At least one implementation involves a process for generating and signaling a "collocated" reference (texture and MV-information) that can be used via the TMVP mode to predict the current view (view-A) from another view (view-B) using a disparity equation.

[0124] To this end, the following operations are performed:

[0125] - copy view-B into view-A's DPB and associate "view-B" with porref,

[0126] - copy the disparity DAB into the motion information (MV map) of the "collocated" reference picture (pocCol),

[0127] - Then, you can use the TMVP pattern introduced earlier, using Figure 10 The disparity information shown is used to predict the current block of view-A according to view-B.

[0128] Figure 11 A variant embodiment of a multi-view application using the TMVP mode is shown. In this embodiment, a reference picture marked as "collocated" is generated, and the picture consists of the texture of view-B and the MV-map constructed with the disparity DAB (e.g. Figure 9 The values ​​of pocCol, pocRef, and pocColRef are set equal to pocCur, so (pocCur - pocRef) and (pocCol - pocColRef) are equal to zero and TMVP is equal to MVP as previously explained.

[0129] In a variant implementation, when using syntax VVC-6.0, the ILRP MV-map is not copied to the child DPB, but is set to zero. Whether the ILRP MV-map is copied is signaled in the bitstream. For example, a flag may be signaled in the SPS, slice header, DPS, and / or VPS.

[0130] In a variant implementation of multi-view applications, in the case of AMVP with MV rescaling, if AMVP is used to predict a neighboring block A, then pocRefA is equal to pocCur, while pocRef may be different from pocCur. Therefore, the denominator of (Equation 4) may be zero. To cope with this uncertainty, the AMVP mode is modified as follows:

[0131] - If AMVP uses MV rescaling and pocCur is equal to pocRefA, then set mvP equal to mvA.

[0132] In another variant embodiment, when AMVP uses MV rescaling and pocCur is equal to pocRefA, then mvP is set equal to zero.

[0133] Figure 12 A variant implementation is shown where the "external reference picture" is not ILRP. In this implementation, a dependency on a reconstructed picture of the current layer ("use_rec_ref_pic_flag") is signaled. This indicates whether the process for constructing the "external reference picture" requires some reconstructed pictures of the current layer. If the "external reference picture" has the same POC as the current picture, it is placed in a child DPB. Two alternatives are possible: dependency on a reference picture of the current layer or dependency on a non-reference picture of the current layer.

[0134] Table 6 shows an example of the corresponding syntax.

[0135]

[0136] Table 6

[0137] Conventional scalable codec specifications describe the process of deriving inter-layer reference pictures (ILRPs) from source pictures for inter-layer prediction. Typical examples include algorithmic upscaling or color gamut scalability described in the specification. However, some applications require a different process while maintaining the same inter-layer dependency information.

[0138] Figure 13An embodiment is shown in which an "external reference picture" is generated by an external process. In this context, the external process refers to a process not defined in the video codec specification. The external process can use the current picture, but it can also use external data to generate reference pictures and motion vectors. In at least one embodiment, a flag, such as "vps_external_process_flag," is used to signal whether the reference picture uses the external process to construct an inter-layer reference picture.

[0139] In a variant implementation, the information vps_external_process_flag is not present if vps_external_layer[i] = 1. In another variant, if vps_external_process_flag is not present and vps_external_layer[i] = 1, then vps_external_process_flag is inferred to be equal to 1. In another variant, if vps_external_process_flag is not present and vps_external_layer[i] = 0, then vps_external_process_flag is inferred to be equal to 0.

[0140] Table 7 shows an example of the syntax of a reference picture list, which is modified to implement at least one of the embodiments described above. Modified elements are identified by underlined text.

[0141]

[0142]

[0143] Table 7

[0144] Table 8 shows an example of the syntax of a reference picture list, which is modified to implement at least one of the embodiments described above. Modified elements are identified by underlined text.

[0145]

[0146]

[0147] Table 8

[0148] This application describes various aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described in detail, and at least individual characteristics are shown, often in a manner that may sound limited. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous submissions.

[0149] The aspects described and contemplated in this patent application can be implemented in many different forms. Figure 1 、 Figure 2 and Figure 3 Some embodiments are provided, but other embodiments are contemplated, and the discussion of these figures does not limit the breadth of the embodiments. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0150] Various methods are described herein, and each method includes one or more steps or actions for achieving the method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0151] The various methods and other aspects described in this patent application may be used to modify modules of the video encoder 100 and decoder 200 (e.g., the reference picture buffers (180, 280), the motion compensation module, and the motion estimation module (170, 175, 275)), such as Figure 1 and Figure 2 Furthermore, aspects of the present invention are not limited to VVC or HEVC and may be applied, for example, to other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application may be used alone or in combination.

[0152] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes, and the aspects are not limited to these specific values.

[0153] Various implementations involve decoding. As used herein, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder. In various embodiments, such a process also includes or alternatively includes a process performed by a decoder of the various implementations described in this application.

[0154] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" refers specifically to a subset of operations or broadly to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0155] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used in this application, may encompass, for example, all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder. In various embodiments, such processes also include or alternatively include processes performed by the encoder of the various implementations described in this application.

[0156] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" refers specifically to a subset of operations or broadly to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0157] Note that the grammatical elements used herein are descriptive terms, and therefore, they do not preclude the use of other grammatical element names.

[0158] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0159] Various implementations refer to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is often formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all coding options (including all considered modes or coding parameter values) and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to reduce coding complexity, particularly for calculating approximate distortion based on predictions or prediction residual signals rather than reconstructed residual signals. A hybrid of these two approaches may also be used, such as by using approximate distortion for only some of the possible coding options and full distortion for others. Other approaches only evaluate a subset of the possible coding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization does not necessarily involve a complete evaluation of both coding costs and associated distortion.

[0160] This application describes various aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described in detail, and at least individual characteristics are shown, often in a manner that may sound limited. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous submissions.

[0161] The specific implementations and aspects described herein can be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of specific implementation (e.g., discussed only as a method), the specific implementation of the features discussed can also be implemented in other forms (e.g., an apparatus or program). The apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, computers, tablet computers, smart phones, mobile phones, portable / personal digital assistants, and other devices that facilitate the transmission of information between end users.

[0162] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0163] Additionally, this application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0164] Furthermore, this application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0165] Additionally, this application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0166] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," "frame," "slice," and "tile" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoding side, while "decoding" is used on the decoding side.

[0167] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first-listed option and the second-listed option (A and B), or only the first-listed option and the third-listed option (A and C), or only the second-listed option and the third-listed option (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0168] Furthermore, as used herein, the term "signaling" means (among other things) indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a specific one of the brightness compensation parameters. Thus, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder may transmit specific parameters to the decoder (explicit signaling) so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and others, signaling may be used without transmitting them (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding transmitting any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form of the word "signal" is mentioned above, the word "signal" may also be used as a noun herein.

[0169] It will be apparent to one of ordinary skill in the art that a specific implementation may generate various signals formatted to carry, for example, storable or transmittable information. The information may include, for example, instructions for performing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiment. Such signals may be formatted, for example, as electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

Claims

1. A method for decoding a multi-view video, the method comprising: obtaining information representing an external reference picture, wherein the external reference picture includes a texture of a first view and a motion vector map representing disparity information between the first view and the second view; as well as Reconstructing the current picture of the second view based on the external reference picture, wherein reconstructing the current picture comprises: copying the first view into a decoded picture buffer of the second view; associating the first view with a picture order count, copying the disparity information into a motion information map of a reference picture collocated with the current picture of the second view; and The second view is predicted based on the replication information.

2. The method of claim 1, wherein the prediction is performed using a temporal motion vector prediction (TVMP) mode.

3. The method of claim 2, wherein a picture order count (pocRef) of the first view and a picture order count (pocCol) of the collocated reference picture are set equal to a picture order count (pocCur) of the current picture of the second view.

4. The method according to claim 1, wherein The prediction is performed using an adaptive motion vector prediction (AMVP) mode, and wherein, if the picture order count (pocCur) of the current picture is equal to the picture order count (pocRefA) of the reference picture of the neighboring block, the motion vector predictor (mvP) is set equal to the motion vector (mvA) of the neighboring block. 5 . The method of claim 1 , wherein obtaining information representing an external reference picture comprises decoding a flag (external_ref_pic_flag) from a reference picture list structure in a bitstream, the flag signaling that the reference picture is an external reference picture.

6. The method according to claim 1, wherein The information representing the external reference picture also includes a flag (zero_mv_ref_flag) indicating whether the motion vector map of the external reference picture is to be set to zero.

7. An apparatus for decoding a multi-view video, the apparatus comprising a decoder configured to perform the following operations on a current picture of a second view: obtaining information representing an external reference picture, wherein the external reference picture includes a texture of a first view and a motion vector map representing disparity information between the first view and the second view; as well as reconstructing the current picture of the second view based on the external reference picture, Reconstructing the current picture includes: copying the first view into a decoded picture buffer of the second view; associating the first view with a picture order count, copying the disparity information into a motion information map of a reference picture collocated with the current picture of the second view; and The second view is predicted based on the replication information.

8. A non-transitory computer-readable medium comprising program code instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images