Video codecs using multi-resolution reference picture management

Abstract

A video encoding and decoding method and apparatus are provided. A set of two or more pictures is encoded and decoded. The set includes at least a first picture encoded and decoded at a first spatial resolution and a second picture encoded and decoded at a second spatial resolution different from the first spatial resolution. Based on the corresponding spatial resolutions, the set of two or more pictures is stored in a decoded picture buffer. The set of two or more pictures stored in the decoded picture buffer is used as a reference picture set for encoding and decoding one or more consecutive pictures after the set of two or more pictures. The set of two or more pictures is incorporated into a reference picture list, which contains at least pictures with two different spatial resolutions.

Description

[0001] This application is a divisional application of a patent application with an application date of December 9, 2019, application number 201980080833, and invention name “Video Codec Using Multi-Resolution Reference Picture Management”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 766,997 filed on December 7, 2018. The entire contents of the above application are incorporated herein by reference. Technical Field

[0004] The present disclosure relates generally to video coding and compression. Specifically, the present disclosure relates to systems and methods for performing video coding using reference picture management. Background Art

[0005] This section provides background information related to the present disclosure. The information contained in this section should not necessarily be construed as prior art.

[0006] Any of a variety of video codec techniques can be used to compress video data. Video coding can be performed according to one or more video codec standards. Some illustrative video codec standards include Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), and Moving Picture Experts Group (MPEG) codecs. Video codecs typically utilize prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that make full use of the redundancy inherent in video images or sequences. One goal of video codec technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0007] According to many video codec specifications, for various purposes, it is necessary to identify and manage pictures stored in a decoded picture buffer (DPB). For example, these pictures can be used as reference pictures for performing inter-frame prediction. Additionally or alternatively, these pictures can be used as output pictures for display. At the same time, the information associated with these pictures can also be used in operations such as scaling of motion vectors in temporal motion vector (MV) derivation and / or scaling of sample values ​​in weighted prediction, etc. Summary of the invention

[0008] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0009] According to a first aspect of the present disclosure, a video encoding and decoding method is performed at a computing device having one or more processors and a memory, the memory storing a plurality of programs to be executed by the one or more processors. The method includes: encoding and decoding a set of two or more pictures, the set including at least a first picture encoded and decoded at a first spatial resolution and a second picture encoded and decoded at a second spatial resolution different from the first spatial resolution. The set of two or more pictures is stored in a decoded picture buffer operatively coupled to a decoder. Based on their corresponding picture resolutions, the set of two or more pictures is stored in a decoded picture buffer. The set of two or more pictures stored in the decoded picture buffer is used as a reference picture set for encoding and decoding one or more consecutive pictures after the set of two or more pictures. The set of two or more pictures is incorporated into a reference picture list, the reference picture list containing at least pictures with two different spatial resolutions.

[0010] According to a second aspect of the present disclosure, a video encoding and decoding device is provided. The device includes a codec configured to encode and decode a set of two or more pictures. The set includes at least a first picture encoded and decoded at a first spatial resolution and a second picture encoded and decoded at a second spatial resolution different from the first spatial resolution. The set of two or more pictures is stored in a decoded picture buffer operatively coupled to a decoder. Based on their corresponding picture resolutions, the set of two or more pictures is stored in the decoded picture buffer. The decoded picture buffer includes a computer-readable storage device configured to store the set of two or more pictures. Based on the first and second spatial resolutions, the set of two or more pictures is stored in the decoded picture buffer. The set of two or more pictures is stored in the decoded picture buffer operatively coupled to the decoder. Based on their corresponding picture resolutions, the set of two or more pictures is stored in the decoded picture buffer. The set of two or more pictures stored in the decoded picture buffer is used as a reference picture set for encoding and decoding one or more consecutive pictures after the set of two or more pictures. A set of two or more pictures is incorporated into a reference picture list, the reference picture list containing at least pictures having two different spatial resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Hereinafter, a set of illustrative, non-limiting embodiments of the present disclosure will be described in conjunction with the accompanying drawings. A person of ordinary skill in the relevant art may implement variations of structure, method, or function based on the examples presented herein, and these variations are all included within the scope of the present disclosure. In the absence of conflicts, the teachings of different embodiments may, but need not, be combined with each other.

[0012] Figure 1 is a block diagram illustrating an illustrative encoder that can be used in conjunction with many video codec standards.

[0013] Figure 2 is a block diagram illustrating an illustrative decoder that can be used in conjunction with many video codec standards.

[0014] Figure 3 is an example of a parameter set that can be used in conjunction with many video codec standards.

[0015] Figure 4 is a diagram of motion vector scaling used to derive the temporal motion vector predictor (TVMP).

[0016] Figure 5 An example of a scalable video codec architecture is shown.

[0017] Figure 6 A set of pictures encoded and decoded at two different spatial resolutions is shown. DETAILED DESCRIPTION

[0018] The terms used in this disclosure are intended to illustrate specific examples rather than to limit the disclosure. The singular forms "a", "a", and "the" as used in this disclosure and the appended claims also refer to the plural forms unless the context clearly implies otherwise. It should be understood that the term "and / or" as used herein refers to any or all possible combinations of one or more associated listed items.

[0019] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of protection of the present disclosure, the first information may be referred to as the second information; and similarly, the second information may also be referred to as the first information. As used herein, the term "if" may be understood to mean "when", "at", or "in response to", depending on the context.

[0020] Singular or plural references throughout this specification to "one embodiment," "another embodiment," and the like mean that one or more specific features, structures, or characteristics described in connection with one embodiment are included in at least one embodiment of the present disclosure. Thus, the phrases "in one embodiment" or "in an embodiment," "another embodiment," and the like, appearing in singular or plural form in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable manner.

[0021] Conceptually, many video codec standards are similar, including those mentioned previously in the Background section. For example, almost all video codec standards use block-based processing and share similar video codec block diagrams to achieve video compression.

[0022] Figure 1 A block diagram of an illustrative encoder 100 that can be used in conjunction with many video codec standards is shown. In encoder 100, video frames are divided into multiple blocks for processing. For each given video block, a prediction is formed based on an inter-frame prediction method or an intra-frame prediction method. In inter-frame prediction, one or more predictors are formed through motion estimation and motion compensation based on pixels from a previously reconstructed frame. In intra-frame prediction, a predictor is formed based on reconstructed pixels in the current frame. Through mode decision, the best predictor can be selected to predict the current block.

[0023] The prediction residual representing the difference between the current block and its predictor is sent to the transform 102 module. The transform coefficients are then sent from the transform 102 module to the quantization 104 module for entropy reduction. The quantized coefficients are then fed to the entropy codec 106 module to generate a compressed video bitstream. Figure 1 As shown in , prediction-related information 110 (e.g., block partition information, motion vector, reference picture index, intra-frame prediction mode, etc.) from the inter-frame prediction module and / or the intra-frame prediction 112 module is also fed through the entropy coding and decoding 106 module and saved in the compressed video bit stream 114.

[0024] In the encoder 100, a decoder-related module is also required to reconstruct pixels for prediction purposes. First, the prediction residual is reconstructed by the inverse quantization 116 block and the inverse transform 118 module. The reconstructed prediction residual is combined with the block predictor 120 to generate the unfiltered reconstructed pixels of the current block.

[0025] In order to improve the efficiency of coding and decoding and visual quality, loop filters are usually used. For example, deblocking filters are provided in AVC, HEVC and the current version of VVC. In HEVC, an additional loop filter called SAO (Sample Adaptive Offset) is defined to further improve the efficiency of coding and decoding. In the current version of the VVC standard, another loop filter called ALF (Adaptive Loop Filter) is being actively studied and it is likely to be included in the final standard.

[0026] These loop filter operations are optional. Performing these operations helps improve codec efficiency and visual quality. They can also be turned off, with the encoder 100 making this decision to save computational complexity.

[0027] It should be noted that intra prediction is usually based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels (if the encoder 100 has these filter options turned on).

[0028] Figure 2 is a block diagram of an exemplary decoder 200 that can be used in conjunction with many video codec standards. The decoder 200 is similar to the one located at Figure 1 The part of the encoder 100 related to reconstruction. In the decoder 200 ( Figure 2 ), the incoming video bitstream 201 is first decoded by the entropy decoding 202 module to derive the quantized coefficient levels and information related to the prediction. The quantized coefficient levels are then processed by the inverse quantization 204 block and the inverse transform 206 module to obtain the reconstructed prediction residuals. The block predictor mechanism implemented in the intra / inter mode selector 212 block is configured to perform an intra prediction 208 process or a motion compensation 210 process based on the decoded prediction information. The reconstructed prediction residuals from the inverse transform 206 module are added to the prediction output generated by the block predictor mechanism using a summer 214 to obtain a set of unfiltered reconstructed pixels. With the loop filter turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video for output.

[0029] Parameter sets were introduced in the H.264 / AVC video codec standard due to the destructive effects of the loss of certain header information. The header information may be in the form of a sequence header and / or a picture header. For example, in many video codec applications, a picture may be divided into multiple segments (e.g., slices), each of which is transmitted in its own transmission unit (e.g., RTP packet). Typically, the above-mentioned header information is transmitted together with the first segment of the picture. In this case, due to the lack of this header data, the loss of the first packet of the picture may result in a completely incorrect picture.

[0030] Parameter sets can be part of the video bitstream ( Figure 1 114 or Figure 2 201), or may be transmitted by the decoder in other ways (e.g., out-of-band transmission using a reliable channel, or hard decoding in the encoder 100 ( Figure 1 ) and decoder 200( Figure 2 ) to receive. Figure 3 , parameter sets 300 may each contain different types of data with different scopes. For example, a sequence parameter set (SPS) 302 is defined to carry video sequence level information. A picture parameter set (PPS) 304 is also defined to carry picture level information. A parameter set contains an identifier that can be referenced directly or indirectly from a slice header 308.

[0031] In HEVC, another type of parameter set called video parameter set (VPS) 306 is introduced to convey information applicable to multiple layers as well as sub-layers to support scenarios with capability exchange and session negotiation. Each layer of a given video sequence (regardless of whether they have the same or different SPS 302) points to the same VPS 306.

[0032] The information conveyed by VPS 306 includes: 1) common semantic elements shared by multiple layers or operation points to avoid unnecessary repetition; 2) basic information of the operation point required for session negotiation, such as profiles and levels; and 3) other operation point-specific information that does not belong to a single SPS 302, such as hypothetical reference decoder (HRD) parameters for layers or sublayers. H.264 / AVC does not have a comparable parameter set, so the information listed above must usually be repeated in each SPS 302.

[0033] like Figure 3 As shown in , the slice header 308 may reference the PPS 304 by the identifier pps_id. Likewise, the PPS 304 may reference the SPS 302 by the identifier sps_id, and the SPS may reference the VPS by the identifier vps_id.

[0034] For inter prediction, a motion model is defined to describe how to form a prediction block from a given reference picture.

[0035] In H.265 / HEVC, H.264 / AVC and previous video codec standards, only the translational motion model is defined and used for block-based motion compensation (MC). Based on this model, the reference pixel position (x r ,y r ) is derived as:

[0036] x r =x+a

[0037] y r =y+b

[0038] Where (x, y) is the position of the current pixel. Parameters a and b are the horizontal and vertical components of the motion vector (a, b). Such a motion model is simple, but it cannot be used to model non-translational motion such as rotation, scaling, or shear mapping.

[0039] In order to achieve more efficient inter-frame prediction, a more complex affine motion model is used in the latest VVC standard. Currently, both 4-parameter and 6-parameter affine models are supported. The general 6-parameter affine model can be expressed as:

[0040] x r =a*x+b*y+c

[0041] y r =d*x+e*y+f

[0042] Among them, (a, b, c, d, e, f) are parameters to be determined and transmitted from the encoder to the decoder.

[0043] As a simplification, VVC also supports the 4-parameter affine model shown below to classify motions limited to combinations of rotation, scaling, and translation.

[0044] x r =a*x+b*y+c

[0045] y r =-b*x+a*y+f

[0046] Although the 4-parameter affine model is not as general as the 6-parameter affine model, it has fewer parameters to be encoded, decoded and transmitted. In the case where the motion is limited to rotation, scaling and translation, the 4-parameter affine model is usually preferred from a rate-distortion (RD) perspective.

[0047] In HEVC, the motion vector of the current block can be predicted using the motion vector in one of its reference pictures. Such a motion vector predictor is called a temporal motion vector predictor (TMVP). A series of processes for deriving the TMVP for a given current block are defined in HEVC.

[0048] Specifically, in the derivation of the TMVP, a scaled motion vector is derived from the MV of a co-located picture, which is a previously encoded and decoded picture in a reference picture list. In the derivation of the TMVP, an explicit flag (collocated_from_l0_flag) in the slice header is first sent to the decoder to indicate whether the co-located picture is selected from a first reference picture list, referred to as List 0, rather than from a second reference picture list, referred to as List 1. In the following description of the present disclosure, for simplicity, List 0 is also referred to as L0; similarly, List 1 is also referred to as L1. A co-located reference index (collocated_ref_idx) is further sent to indicate which picture in the list is selected as the co-located picture for deriving the TMVP. According to the predefined order of MVs of different lists in the co-located blocks of the co-located picture, the L0MV and L1MV of the temporal motion candidate are independently derived, as shown in Table 1.

[0049] Table 1 shows an exemplary algorithm for deriving temporal MVs from co-located blocks for TMVP in HEVC.

[0050]

[0051] Table 1

[0052] Figure 4 404. FIG. 405 is an illustration of a motion vector scaling process for deriving a temporal motion vector predictor (TVMP). As shown by dashed line 401, a scaled motion vector for deriving the TMVP is obtained by scaling a selected motion vector of a co-located prediction unit (PU) 402 using a set of picture order count (POC) distances tb 403 and td 404, where tb 403 is defined as the POC difference (also called POC distance) between the reference picture of the current picture and the current picture, and td 404 is defined as the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero. In the HEVC specification, the actual implementation of the scaling process is described. For a B slice, two motion vectors are obtained. The first motion vector corresponds to reference picture list 0 and the second motion vector corresponds to reference picture list 1. The first motion vector and the second motion vector are combined to form a bi-directionally predicted TMVP.

[0053] As explained in the previous section, previously decoded / reconstructed pictures are saved in a decoded picture buffer (DPB). These pictures are managed based on certain rules so that they are available when needed and permanently deleted from the DPB when no longer needed. Pictures in the DPB are used for various purposes, such as being used as reference pictures for inter-frame prediction and / or for output for display. They also provide information for motion vector scaling when deriving the temporal motion vector predictor (TMVP) and for sample value scaling in weighted prediction.

[0054] In both AVC and HEVC, pictures stored in the DPB are managed by a decoded reference picture marking process to determine whether the picture is still used as a reference picture for decoding consecutive pictures in decoding order, and / or to determine whether the picture needs to be output (or displayed). If the picture is not needed in either case, it is removed from the DPB and the corresponding storage space can be used for other pictures. In both standards, reference pictures are divided into two types: short-term reference pictures and long-term reference pictures.

[0055] The reference picture marking process is implemented by different mechanisms in AVC and HEVC. In H.264 / AVC, there are two mechanisms for decoding reference picture marking: implicit sliding window processing and explicit memory management control operation (MMCO) processing. When the number of reference frames is equal to a given maximum number (max_num_ref_frames) defined in the SPS, the sliding window processing marks the short-term reference picture as "unused for reference". Short-term reference pictures are stored in a first-in-first-out manner so that the most recently decoded short-term picture is saved in the DPB.

[0056] Explicit MMCO processing may include multiple MMCO commands. The MMCO command may mark one or more short-term or long-term reference pictures as "unused for reference", mark all pictures as "unused for reference", or mark the current reference picture or an existing short-term reference picture as long-term and assign a long-term reference picture index to the long-term reference picture.

[0057] In H.265 / HEVC, a completely different approach to reference picture management is introduced, namely through Reference Picture Sets (RPS). For each specific slice, the RPS defines the complete set of reference pictures used by the current picture or any subsequent picture. This reference picture set is all kept in the DPB. Unlike the mechanism in H.264 / AVC, with the RPS concept, no information from earlier pictures in the decoding order is required to maintain the correct state of reference pictures in the DPB.

[0058] It may be desirable to change the video resolution of the bitstream in order to accommodate temporary changes in the available bandwidth of the communication channel. This can be achieved through simulcast or scalable video codecs.

[0059] In the case of simulcast, bitstreams for the same video but with different spatial resolutions can be generated separately and sent simultaneously in a broadcast manner. The receiving device can adaptively select between bitstreams of different resolutions based on the bandwidth of the receiving device. This resolution switching usually occurs at the location of a key picture (e.g., an IDR picture) that can be used as a random access point.

[0060] In scalable video coding, bitstreams with different spatial resolutions for the same video are jointly generated in a layered manner, where each layer corresponds to a different video resolution. Through this joint coding, certain correlations between videos of different resolutions can be further exploited to reduce the bit rate.

[0061] Figure 5 An example of a scalable video codec structure is shown. In this example, there are two layers, which can be referred to as a base layer 501 and an enhancement layer 502. In this example, the base layer 501 has a lower spatial resolution and a lower temporal resolution than the enhancement layer 502. Inter-layer prediction (indicated by vertical arrows) can be supported to improve the codec efficiency of the enhancement layer 502 compared to simulcast.

[0062] According to the scalable video codec profile of H.264 / AVC, any of several items can be used for inter-layer prediction from the co-located base layer 501 block. These items include using reconstructed pixels, motion vectors, and / or block texture prediction residuals in the intra-frame prediction area. In addition, when the base layer 501 and the enhancement layer 502 have different spatial resolutions, these items from the co-located base layer 501 block need to be appropriately upsampled and / or scaled. In this case, additional buffers may be required to store the upsampled / scaled intermediate data. In addition to any additional buffers required for inter-layer prediction, each layer has its own DPB.

[0063] As in the case of simulcast, bitstream switching from lower spatial resolution to higher spatial resolution in scalable video codecs can occur at the location of a key picture (e.g., an IDR picture) that can be used as a random access point. Since the base layer 501 is always required, switching from enhancement layer 502 to base layer 501 can occur at any picture location.

[0064] AOMedia Video 1 (AV1) is an open, royalty-free video codec format developed by the Alliance for Open Media (AOMedia), a coalition of companies from the semiconductor industry, video-on-demand providers, and web browser developers formed in 2015.

[0065] In AV1, a feature called "frame super-resolution" is supported. According to this codec mode, a picture can be encoded or decoded at a lower spatial resolution and then super-resolved to full resolution in a standard way before updating the reference buffer set. Such an approach is known to provide perceptual advantages at very low bitrates. In order to make the operation computationally tractable, the super-resolution process is decomposed into a linear upscaling followed by the application of a Wiener filter-based loop recovery tool at a higher spatial resolution. Furthermore, in order to achieve a cost-effective hardware implementation without overhead in the line buffer, the upscaling / downscaling is restricted to operate only horizontally.

[0066] In existing video codec standards such as H.264 / AVC and H.265 / HEVC, videos are encoded and decoded at a constant spatial resolution. In cases where the spatial resolution needs to be reduced to accommodate lower bit rate scenarios and / or lower available bandwidth, pictures that can support random access (e.g., IDR pictures) and new parameter set information (e.g., SPS, PPS, etc.) need to be marked together with the IDR pictures. The problem with this mechanism is that since the IDR picture is intra-coded, sending the picture is usually very expensive. If a lower bit rate is imposed on the IDR picture, its visual quality may be greatly affected.

[0067] With the scalable extended profiles of AVC and / or HEVC, the spatial resolution can be reduced by keeping only the base layer 501 picture bitstream to accommodate lower bitrate scenarios. However, scalable video codecs involve multiple layers. Its codec efficiency is generally not as efficient as encoding and decoding a single layer. The decoder is more complex because it must support multi-layer decoding (and in some cases, multi-layer motion compensation). In fact, on the decoder side, this additional complexity is very undesirable.

[0068] In AV1, this is better supported through the frame super-resolution codec mode. In this case, frame resolution reduction can occur at any given picture position (which includes inter-predicted pictures). However, before the reconstructed picture is saved to the DPB for future use, it does need to be frame-upscaled and then super-resolved to ensure that the pictures in the DPB all have the same spatial resolution.

[0069] According to the present disclosure, pictures can be encoded and decoded at different spatial resolutions at any picture position. Changes in spatial resolution can occur at any picture position, and do not have to occur on pictures that support random access (e.g., IDR pictures). In addition, there is no need for an upscaling process and a super-resolution process to scale the reconstructed picture back to the original video resolution. Instead, no matter what image resolution the picture is encoded and decoded at, the reconstructed image is saved directly to the DPB.

[0070] Image resolution degradation is not limited to the horizontal direction. It can occur in both horizontal and vertical directions.

[0071] Figure 6 A set of pictures encoded and decoded at two different spatial resolutions is shown. For example, based on the teachings of the present disclosure, the DPB buffer may contain pictures with at least two different spatial resolutions. Figure 6 As shown in , assuming that the pictures are encoded and decoded according to their time order K, K+1, ...K+6, ..., the pictures can be spatially downgraded to save bit rate when needed. In this example, the first picture 601 represented as K and the second picture 602 represented as K+1 are both at the first spatial resolution. However, the third picture 603 represented as K+2 is downgraded to a second spatial resolution lower than the first resolution to adapt to the lower bit rate. For example, this may be due to a reduction in available bandwidth on the communication channel. The fourth picture 604 (K+3), the fifth picture 605 (K+4) and the sixth picture 606 (K+5) are all encoded and decoded at the second (lower) spatial resolution. At the seventh picture 607 (K+6), the communication channel bandwidth is fully improved, and the seventh picture 607 can be encoded and decoded at the first (higher) resolution applied to the first picture 601 and the second picture 602.

[0072] In the present disclosure, pictures are reconstructed to their coded resolution and saved directly into the DPB without any further upscaling and / or super-resolution. Figure 6In the example shown, the third picture 603, the fourth picture 604, the fifth picture 605 and the sixth picture 606 (K+2, K+3, K+4 and K+5) are all saved in the DPB at the second (lower) spatial resolution, while the first picture 601, the second picture 602 and the seventh picture 607 (K, K+1 and K+6) are saved in the DPB at the first (higher) resolution. In other words, the pictures with the first spatial resolution in the set are directly stored in the decoded picture buffer at the first spatial resolution, and the pictures with the second spatial resolution in the set are directly stored in the decoded picture buffer at the second spatial resolution. As a result, the decoded picture buffer stores pictures with different spatial resolutions. In one or more examples, the decoded picture buffer stores pictures at their original spatial resolution based on different available bandwidths in the communication channel.

[0073] Therefore, according to the present disclosure, the reference picture list can contain pictures with different spatial resolutions. It is worth mentioning that this feature is orthogonal to the basic decoded reference picture marking process. For example, it can be used with sliding window or MMCO processing in H.264 / AVC. It can also be used with RPS in H.265 / HEVC and its associated reference picture list generation process.

[0074] The main motivation for allowing different spatial resolution pictures in the reference picture list is the availability of more sophisticated motion models in VVC. For example, with affine prediction, the scaling effects required to achieve good prediction are automatically handled when the reference picture has a different spatial resolution than the current picture being encoded or decoded.

[0075] In HEVC, in SPS 302 ( Figure 3 ) to indicate picture width and height at the video sequence level, and assume that each picture in the video sequence has the same size. According to the present disclosure, picture width and picture height are indicated directly and / or indirectly at the picture level (e.g., in PPS 304). The original video resolution can still be indicated at the sequence level (e.g., in SPS 302).

[0076] Such signaling can take various forms. For example, in PPS 304, the picture width and height can be indicated according to how many luma sample values ​​there are along each dimension. Alternatively or additionally, in PPS 304, they can also be indicated as ratio values ​​relative to the original video picture width and height. For example, such ratio values ​​can be 1 / 4, 1 / 2, 3 / 4, and 1.0, etc. These ratio values ​​can be indicated as denominators and numerators. They can also be indicated as index values ​​for a set of predefined denominators and numerators.

[0077] According to the present disclosure, in addition to the picture POC difference, the motion vector scaling can be based on the picture resolution. For example, according to the relative ratio of the picture spatial resolution, the motion vector scaling can be adjusted accordingly.

[0078] According to the present disclosure, when a reference picture and a current picture being encoded and decoded have different spatial resolutions, a global motion vector may be derived based on the two spatial resolutions to at least resolve a scaling effect between the two pictures caused by the different spatial resolutions.

[0079] For the current picture to be coded, a global motion vector can be derived for each reference picture. It is assumed that the motion vector decoded and / or indicated for each block in the current picture has been predicted and / or indicated using the global motion vector associated with the corresponding reference picture for the motion vector decoder. As a result, the actual motion vector of the current block is jointly derived based on the motion vector of the indicated block and the global motion vector.

[0080] In the case where the current picture to be encoded and decoded has the same spatial resolution as the reference picture, the global motion vector used with the reference picture may not be needed.

[0081] The global motion vector can use various motion models. For example, it can use an affine motion model to convey the scaling effect between different pictures mentioned above. Other motion models can also be used to convey scaling effects.

[0082] According to the present disclosure, such global motion vectors do not need to be explicitly signaled to the decoder in the bitstream. Instead, they can be derived based on the same set of rules at both the encoder and decoder ends. These same set of rules can at least include: taking the picture spatial resolution as one of the factors in determining the global motion vector.

[0083] According to another embodiment of the present disclosure, the global motion vector of each reference picture may be signaled to a decoder in a bitstream. Such signaling may also take various forms. For example, each of the global motion vectors may be directly indicated as a motion vector. Each of the global motion vectors may also be indicated as a scaling factor along the horizontal and vertical dimensions. Alternatively, each of the global motion vectors may also be indicated as an index value for a set of predefined motion vectors and / or scaling factors.

[0084] According to the current disclosure, although the lower / higher resolution pictures in the DPB do not need to be upsampled / downsampled to the spatial resolution of the current picture, a set of filters can still be sent to the decoder to indicate the preferred picture upsampling / downsampling filters to be used for better motion compensation prediction or better display. Such signaling can be placed in the header information, or in the SPS 302 and PPS 304. It can also be marked as SEI (Supplemental Enhancement Information).

[0085] In some examples, a set of two or more pictures does not include any pictures that support random access.

[0086] In some examples, the second spatial resolution is less than the first spatial resolution, including resolution downgrading from the first spatial resolution to the second spatial resolution, and the method further includes: downgrading the resolution only in the horizontal direction, downgrading the resolution only in the vertical direction, or downgrading the resolution in both the horizontal and vertical directions.

[0087] In some examples, the method further includes: sending a coded set comprising two or more pictures over a communication channel having a dynamically changing bandwidth capacity; and setting the second spatial resolution to be smaller than the first spatial resolution in response to a decrease in the bandwidth capacity of the communication channel.

[0088] In some examples, the method further includes: sending a codec set comprising two or more pictures over a communication channel having a dynamically changing bandwidth capacity; and setting the second spatial resolution to be greater than the first spatial resolution in response to an increase in the bandwidth capacity of the communication channel.

[0089] In some examples, the set of two or more pictures includes a reference picture list containing pictures having different spatial resolutions.

[0090] In some examples, affine prediction is used to provide an enhanced scaling effect for the set of two or more pictures, wherein the reference picture list includes two or more pictures, each of which has a different spatial resolution than a new picture currently being encoded or decoded.

[0091] In some examples, the method further includes: using a picture parameter set to indicate a picture width and a picture height of each picture in the set of two or more pictures.

[0092] In some examples, the method further includes indicating the picture width and the picture height by indicating a first number of luma samples in a first dimension of the set of two or more pictures, and indicating a second number of luma samples in a second dimension of the set of two or more pictures.

[0093] In some examples, the method further includes indicating the picture width and the picture height by indicating a ratio value for the set of two or more pictures relative to a previous picture width and a previous picture height. In some examples, the ratio value is indicated as a numerator and a denominator. In some examples, the ratio value is indicated as an index value into a set of predefined numerators and denominators.

[0094] In some examples, the method further includes performing motion vector scaling, and adjusting the motion vector scaling in response to a difference between the first spatial resolution and the second spatial resolution.

[0095] In some examples, the method further includes deriving a global motion vector based on the first spatial resolution and the second spatial resolution to account for a scaling effect between the first picture and the second picture.

[0096] In some examples, the method further includes: deriving a respective global motion vector for each corresponding picture in the set of two or more pictures, the set of two or more pictures including a reference picture set; each picture in the set of two or more pictures including a plurality of blocks; and decoding a motion vector indicated for each of the plurality of blocks in the set of two or more pictures. The indicated motion vector is predicted using a respective global motion vector associated with a corresponding picture in the set of two or more pictures. An actual motion vector for a current block of a current picture is jointly derived based on the indicated motion vector and the global motion vector.

[0097] In some examples, the method further includes: using an affine motion model to convey a zoom effect between the first image and the second image.

[0098] In some examples, the method further includes deriving each corresponding global motion vector at the decoder based on the same set of rules applied at both the decoder and the encoder. The same set of rules considers at least one of the first spatial resolution and / or the second spatial resolution.

[0099] In some examples, the method further includes labeling each corresponding global motion vector using a first scaling ratio along a horizontal dimension and a second scaling ratio along a vertical dimension.

[0100] In some examples, the method further includes: identifying each corresponding global motion vector using a set of index values ​​that index into a set of predefined motion vectors or scaling ratios.

[0101] In some examples, the method further includes: indicating the selected filter by using at least one of: header information, supplemental enhancement information, a picture parameter set, or a sequence parameter set.

[0102] In some examples, the set of two or more pictures does not include any pictures that support random access.

[0103] In some examples, the second spatial resolution is less than the first spatial resolution, including a resolution downgrade from the first spatial resolution to the second spatial resolution, and the device is also configured to downgrade the resolution only in the horizontal direction, downgrade the resolution only in the vertical direction, or downgrade the resolution in both the horizontal and vertical directions.

[0104] In some examples, the apparatus further includes a data transmitter configured to transmit a codec set comprising two or more images on a communication channel having a dynamically changing bandwidth capacity, and configured to set a second spatial resolution to be smaller than the first spatial resolution in response to a decrease in the bandwidth capacity of the communication channel.

[0105] In some examples, the apparatus further includes a data transmitter configured to transmit a codec set comprising two or more images on a communication channel having a dynamically changing bandwidth capacity, and configured to set a second spatial resolution to be greater than the first spatial resolution in response to an increase in the bandwidth capacity of the communication channel.

[0106] In some examples, the set of two or more pictures includes a reference picture list containing pictures having different spatial resolutions.

[0107] In some examples, an affine prediction process is performed to provide an enhanced scaling effect for the set of two or more pictures, wherein the reference picture list includes two or more pictures, each of which has a different spatial resolution than a new picture currently being encoded or decoded.

[0108] In some examples, the apparatus further includes a transmitter configured to indicate, using a picture parameter set, a picture width and a picture height of each picture in the set of two or more pictures. In some examples, the transmitter is further configured to indicate the picture width and the picture height by indicating a first number of luma samples in a first dimension of the set of two or more pictures, and indicating a second number of luma samples in a second dimension of the set of two or more pictures. In some examples, the transmitter is further configured to indicate the picture width and the picture height by indicating a ratio value relative to a previous picture width and a previous picture height for the set of two or more pictures.

[0109] In some examples, the ratio values ​​are indicated as a numerator and a denominator.

[0110] In some examples, the ratio value is indicated as an index value into a set of predefined numerators and denominators.

[0111] In some examples, the apparatus is further configured to perform motion vector scaling and adjust the motion vector scaling in response to a difference between the first spatial resolution and the second spatial resolution.

[0112] In some examples, the apparatus is further configured to derive a global motion vector based on the first spatial resolution and the second spatial resolution to account for a scaling effect between the first picture and the second picture.

[0113] In some examples, the apparatus is further configured to: derive a respective global motion vector for each corresponding picture in the set of two or more pictures, the set of two or more pictures comprising a reference picture set; each picture in the set of two or more pictures comprising a plurality of blocks; and decode a motion vector indicated for each of the plurality of blocks in the set of two or more pictures. The indicated motion vector is predicted using a respective global motion vector associated with a corresponding picture in the set of two or more pictures. An actual motion vector for a current block of a current picture is jointly derived based on the indicated motion vector and the global motion vector.

[0114] In some examples, the apparatus is further configured to use an affine motion model to convey a zoom effect between the first picture and the second picture.

[0115] In some examples, the apparatus further comprises a decoder configured to derive each corresponding global motion vector based on a same set of rules applied at both the decoder and the encoder. The same set of rules considers at least one of the first spatial resolution or the second spatial resolution.

[0116] In some examples, the apparatus is further configured to label each corresponding global motion vector using a first scaling ratio along a horizontal dimension and a second scaling ratio along a vertical dimension.

[0117] In some examples, the apparatus is further configured to identify each corresponding global motion vector using a set of index values ​​that index into a set of predefined motion vectors or scaling ratios.

[0118] In some examples, the apparatus is further configured to indicate the selected filter by using at least one of: header information, supplemental enhancement information, a picture parameter set, or a sequence parameter set.

[0119] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes, and may be executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, wherein the computer-readable storage medium corresponds to a tangible medium such as a data storage medium or a communication medium, the communication medium including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, a computer-readable medium may generally correspond to: (1) a non-temporary tangible computer-readable storage medium; or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures to implement the embodiments described in the present application. A computer program product may include a computer-readable medium.

[0120] In addition, the above method can be implemented using a device including one or more circuits, including application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. The device can use these circuits in combination with other hardware or software components to perform the methods described above. Each module, submodule, unit, or subunit disclosed above can be implemented at least in part using one or more circuits.

[0121] By considering the specification and practice of the invention disclosed herein, other embodiments of the invention will be apparent to those of ordinary skill in the art. This application is intended to cover any variation, use or modification of the invention that follows the general principles of the invention, and includes deviations from the disclosure known in the art or customary practice. The specification and examples are considered to be exemplary only, and the true scope and spirit of the invention are indicated by the appended claims.

[0122] It should be understood that the present invention is not limited to the exact examples described above and illustrated in the accompanying drawings and that various modifications and changes may be made without departing from the scope of protection of the present invention, which is defined solely by the appended claims.

Claims

1. A video encoding method, comprising: encoding a set of two or more pictures to obtain corresponding coded pictures of the set of two or more pictures, the set comprising at least a first picture coded at a first spatial resolution and a second picture coded at a second spatial resolution different from the first spatial resolution; Decoding the coded picture of the set of two or more pictures based on the corresponding spatial resolution and storing the decoded picture based on the corresponding spatial resolution in a buffer; as well as A reference picture list is obtained from the decoded pictures based on the corresponding spatial resolution stored in the buffer, wherein the reference picture list is used to perform inter-frame prediction on the pictures after the set including two or more pictures without changing the corresponding spatial resolution of the decoded pictures, without ensuring that the pictures have the same spatial resolution as the reference pictures to be used in performing inter-frame prediction, and the reference picture list includes at least decoded pictures with the first spatial resolution and the second spatial resolution.

2. The video encoding method according to claim 1, wherein: The second spatial resolution is smaller than the first spatial resolution, including resolution downgrading from the first spatial resolution to the second spatial resolution, and the method further includes: downgrading the resolution only in the horizontal direction, downgrading the resolution only in the vertical direction, or downgrading the resolution in both the horizontal and vertical directions.

3. The video encoding method according to claim 1, wherein: The method further includes: using a picture parameter set, indicating a picture width and a picture height for each picture in the set of two or more pictures.

4. The video encoding method according to claim 3, wherein: The marking further includes: marking a first number of luma samples in a first dimension of the set of two or more pictures, and marking a second number of luma samples in a second dimension of the set of two or more pictures.

5. The video encoding method according to claim 3, wherein: The marking further includes: marking a ratio value relative to a previous picture width and a previous picture height for the set of two or more pictures.

6. The video encoding method according to claim 5, wherein: The ratio value is indicated using a numerator and a denominator.

7. The video encoding method according to claim 5, wherein: The ratio values ​​are referenced as index values ​​to a set of predefined numerators and denominators.

8. The video encoding method according to claim 1, wherein: The picture following the set of two or more pictures includes a current picture encoded at a third spatial resolution based on a reference picture in the reference picture list, and wherein a ratio for encoding the current picture is determined based on the third spatial resolution and a spatial resolution of the reference picture.

9. A video encoding device, comprising: an encoder configured to encode a set of two or more pictures, the set comprising at least a first picture encoded at a first spatial resolution and a second picture encoded at a second spatial resolution different from the first spatial resolution, to obtain a corresponding coded picture of the set of two or more pictures; a decoder configured to decode the coded pictures of the set of two or more pictures based on respective spatial resolutions; as well as a buffer configured to store decoded pictures based on corresponding spatial resolutions; The encoder is further configured to obtain a reference picture list from the decoded pictures based on the corresponding spatial resolution stored in the buffer, wherein the reference picture list is used to perform inter-frame prediction on the pictures after the set of two or more pictures without changing the corresponding spatial resolution of the decoded pictures, without ensuring that the pictures have the same spatial resolution as the reference pictures to be used in performing inter-frame prediction, and the reference picture list includes at least decoded pictures with the first spatial resolution and the second spatial resolution.

10. The video encoding apparatus according to claim 9, wherein: The second spatial resolution is smaller than the first spatial resolution, including a resolution downgrade from the first spatial resolution to the second spatial resolution, and the device is further configured to: downgrade the resolution only in the horizontal direction, downgrade the resolution only in the vertical direction, or downgrade the resolution in both the horizontal and vertical directions.

11. The video encoding apparatus according to claim 9, wherein: The video encoding apparatus is further configured to: use a picture parameter set to indicate a picture width and a picture height for each picture in the set of two or more pictures.

12. The video encoding device according to claim 11 is further configured to: indicate a first number of luma samples in a first dimension of the set including two or more pictures, and indicate a second number of luma samples in a second dimension of the set including two or more pictures. 13 . The video encoding apparatus according to claim 11 , further configured to: indicate a ratio value relative to a previous picture width and a previous picture height for the set of two or more pictures.

14. The video encoding apparatus according to claim 13, wherein: The ratio value is indicated using a numerator and a denominator.

15. The video encoding apparatus according to claim 13, wherein: The ratio values ​​are referenced as index values ​​to a set of predefined numerators and denominators.

16. The video encoding apparatus according to claim 9, wherein: The picture following the set of two or more pictures includes a current picture encoded at a third spatial resolution based on a reference picture in the reference picture list, and wherein a ratio for encoding the current picture is determined based on the third spatial resolution and a spatial resolution of the reference picture.

Images