Picture / video coding that supports variable resolution and / or efficiently handles area-based packing.

Abstract

The present invention provides a video encoding / decoding method, electronic device, and non-temporary computer-readable medium that support variable resolution and / or efficiently handle region-based packing. [Solution] The video decoding method includes decoding a flag from the bitstream of encoded video data that indicates whether reference picture resampling is possible and whether the reference picture has parameters that can differ from the corresponding parameters of the current picture; determining, based on the flag, that reference picture resampling is possible and that the parameters of the reference picture are different from the corresponding parameters of the current picture; resampling a portion of the reference picture to generate a resampled portion of the reference picture; and predicting the current picture using the resampled portion of the reference picture.

Description

[Technical Field]

[0001] This application relates to video / picture coding with improved coding efficiency, which involves supporting variable resolution and / or efficiently handling region-based packing. [Background technology]

[0002] The single-layer-based version of HEVC does not allow changes in picture resolution within an encoded video sequence. This is only permitted in a random access point RAP (for example, in an IDR RAP that completely resets the decoded picture buffer), where the resolution change can occur at the start of a new encoded video sequence, including flushing the decoded picture buffer (DPB). However, flushing the DPB significantly reduces the achievable coding efficiency due to the fact that it interrupts the utilization of previously coded references.

[0003] Therefore, there is a need for improved video / picture codecs that can more efficiently utilize resolution variations to improve coding efficiency.

[0004] There are also tools that enable the transport of scenes, such as panoramic scenes, via coded pictures or coded videos, a technique called region-based packing, where the picture / video codec does not care how this scene is mapped onto one or more pictures on a region-by-region basis. Mapping is merely pre-processing on the encoder side and post-processing on the decoder side, but the codec operates on the packed one or more pictures without any awareness of region-based packing. MPEG OMAF provides a framework for such transport of scenes via packed one or more pictures, for example. For example, scene divisions / mappings that define the divisions of coded one or more pictures to picture regions, each of which is mapped onto its respective scene region of the scene, are signaled as secondary information via SEI messages, in order to control post-processing on the decoder side to remap the coded one or more pictures onto scene regions. Such a solution would be efficient in terms of reusing existing coding tools, i.e., existing picture / video codecs, but it would be advantageous to have a temporary concept that would allow region-based packing to be handled more efficiently. [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, the object of this application is to provide an encoding / decoding tool that improves coding efficiency when using fluctuating scene resolutions and / or enables more efficient processing of region-based packing of coded single / multiple pictures.

[0006] This objective is achieved by the subject matter of the independent claims enclosed with this specification. [Means for solving the problem]

[0007] According to a first aspect of this application, a more efficient video codec is achieved by supporting variations in the target spatial resolution in which video is coded within a data stream, on a time scale finer than the random access distance in which video coding is performed. The temporal variation in the target spatial resolution can be controlled as part of the rate control of the video encoder. When the variation in target spatial resolution occurs on a time scale finer than the random access distance, coding efficiency can be improved by allowing the encoder to adapt the target spatial resolution to the current video coding complexity. Different possibilities exist regarding the variation in target spatial resolution, i.e., the number of samples per corresponding picture content in the picture of the video. For example, the video encoder and video decoder may use a prediction loop in which a DPB is connected in series, and already processed pictures are buffered in the DPB at the varying target spatial resolution. In this case, if there is a difference in target spatial resolution between the referenced picture and the reference picture that references the referenced picture, resampling of the pictures in the DPB may occur on the fly, and the resampling provides a motion-compensated predictor. Alternatively, the prediction loop may be fed by pictures at a reference spatial resolution, i.e., at some fixed spatial resolution which may be greater than any other possible value the target spatial resolution can take. In such a case, resampling may be performed to fill the gap between the target spatial resolution and the reference spatial resolution. For example, prediction residual coding at the target spatial resolution is performed while resampling of the prediction residual toward the reference spatial resolution is performed in order to perform prediction correction of predictors obtained from pictures in the DPB at the reference spatial resolution rather than the target spatial resolution. Alternatively, resampling of predictors obtained from pictures in the DPB toward the target spatial resolution may be performed in order to perform prediction correction at the target spatial resolution. Switching between these options may be supported by the encoder and decoder, respectively.The variation in target spatial resolution may occur at the picture level, i.e., for each picture, the target spatial resolution may be set by the encoder and signaled to the decoder in the data stream. According to one embodiment, the signaling overhead associated with continuous changes in target spatial resolution and associated picture size is mitigated by dividing the video pictures according to the change in target spatial resolution in a default manner, for example, in a way that the divisions are proportionally constant, i.e., the pictures are proportional to each other despite their varying sizes, i.e., the divisions correspond when scaled to the same spatial resolution. According to a further embodiment, the relative scaling of the divisions at the end may be presented in the data stream as one option, while the other option involves signaling divisions at a given time scale that is finer than or equal to the time scale at which the target spatial resolution varies. The distinctions just described may be used to encode videos such as videos in which the selection between interprediction mode and intraprediction mode results in blocks that are performed in that unit, videos in which the 0 prediction parameters for interprediction blocks and intraprediction blocks result in blocks that are performed in that unit, or videos in which the prediction residuals in the encoder and decoder, respectively, result in blocks that are subdivided in that unit into tiles that are coded / decoded independently of each other, i.e., without intrapicture coding interdependence.

[0008] In a further aspect of this application, the video codec involves coding / decoding, depending on the scene divisions, pictures of video divided according to the scene divisions into picture regions, each of which maps each scene region of a scene onto the picture region. The video encoder supports adjustment of the scene divisions, such as a kind of user setting, and signals these scene divisions in the data stream from which the video decoder derives them. By rendering the coding / decoding of video in the scene division-dependent data stream, the scene divisions are not only a pre-processing / post-processing for the video codec, but also enable improved coding efficiency by considering the scene divisions that define the divisions into picture regions, and their mapping onto the scene regions. For example, rather than signaling the scene divisions as secondary information in addition to coding packed pictures, i.e., pictures divided into picture regions, scene division signaling can be used to at least partially save signaling overhead that would be redundant for the scene divisions. For example, coding a packed picture may involve coding divisions into blocks to which the packed picture is coded in units, such as coding units in which inter or intra predictive decisions are made in units, predictive units in which the corresponding predictive parameters for the aforementioned blocks are made in units, and / or transform units in which the predictive residuals are transformed and coded in units, and / or tiles in which the packed pictures are coded in units in a manner independent of each other. The division decisions related to the multi-tree subdivision involved in the coding divisions may be inferred, for example, based on scene divisions, thereby saving signaling overhead. Alternatively, the video encoder may simply follow the limitations in the coding divisions with respect to scene divisions. Furthermore, some coding / decoding tools of the video encoder and video decoder may be made to operate depending on these scene divisions. For example, predictor derivation may be performed depending on scene divisions.Scene divisions can result in picture regions within a single picture where the scene sampling resolution—that is, the number of samples per unit area or unit field of view of the scene—is different from the ratio between the picture region size, measured in units of samples, and the corresponding scene region size, measured, for example, as a fraction of the entire scene. Such differences in scene sampling resolution may be taken into account by video encoders and video decoders when performing motion-compensated prediction. For example, a predictor may be determined depending on the difference in scene sampling resolution between the picture region where the interprediction block is located and the picture region of the reference picture pointed to by the motion vector of each interprediction block. Furthermore, motion vector prediction may be performed in a manner that takes into account the difference in scene sampling resolution. For example, instead of immediately adopting such motion vectors of adjacent blocks, candidate motion vectors may be derived based on the motion vectors of adjacent blocks via modifications, and these motion vectors may be duplicated in such a way that a version of this motion vector has its initial point set at the reference point of the adjacent block, and another version of this motion vector has its initial point set at the reference point of the interprediction block, and both have endpoints, and the images of such endpoints in the scene have the same distance from each other but different from the images of the reference points of adjacent blocks and interprediction blocks in the scene via scene divisions. Thus, motion vector prediction is performed via detours across the scene region and thereby responds to scene divisions.

[0009] Furthermore, video coding / decoding may respond to scene divisions in terms of setting one or more parameters related to coding divisions. For example, such parameters may be set and signaled individually in the data stream for each picture region. Alternatively or additionally, filtering of a given picture in a video may be performed using a filter in which at least one of its parameters is varied across the given picture depending on the scene division. For example, at least one parameter may be set at the boundary between adjacent picture regions depending on the difference in scene sampling resolution between adjacent / contacting picture regions. In this way, the filter characteristics can operate more precisely by taking the scene sampling resolution into account. Moreover, video encoders and video decoders can identify the locations of boundary surfaces between picture regions that do not touch each other in a scene, i.e., scene discontinuity boundaries, based on the scene divisions. Coding and decoding may respond to these scene discontinuity boundaries; that is, for example, an encoder may treat these scene discontinuity boundaries as picture boundaries. Decoders and encoders can perform filtering differently at scene discontinuity boundaries compared to the boundaries between picture regions that are adjacent to each other in the scene. Alternatively, signaling of at least one parameter on a picture region basis may be supported. Alternatively or additionally, quantization parameters controlling the spectral shape of the spatial frequency spectrum of quantization noise may be set in particular on a picture region basis, i.e., on a group of one or more picture regions. Variations may be signaled within the data stream.

[0010] The latter aspect, namely the variation of at least one quantization parameter that controls the spectral shape of the spatial frequency spectrum of the quantization noise of quantization, may be used independently of whether or not scene divisions are used, according to further aspects of this application. In particular, when there is locally or spatially varying resolution, coding efficiency can be improved by adapting the quantization parameter to the local characteristics of the picture.

[0011] In a further aspect of this application, improved coding efficiency is achieved by having a video codec that allows the target spatial resolution at which the video is coded in the data stream to be varied in units of the picture region of the picture, which is smaller than the picture, while signaling variations in the data stream. This aspect may be combined with the first and third aspects described above, namely, quantization parameter variations. The prediction loop may be fed, for example, a picture at a reference spatial resolution. Again, the variation of the target spatial resolution within the picture may be a task performed by rate control of the encoder.

[0012] The advantageous aspects of this application are the subject matter of the dependent claims, and preferred embodiments of this application are described below with reference to the drawings. [Brief explanation of the drawing]

[0013] [Figure 1a] This is a schematic block diagram of a video encoder according to one embodiment, which uses video coding that depends on the temporal variation of the target spatial resolution. [Figure 1b] This is a block diagram of a decoder compatible with the video encoder shown in Figure 1a, according to one embodiment. [Figure 2a] This is a schematic diagram showing the difference in target spatial resolution according to the embodiments of Figure 1a and Figure 1b. [Figure 2b] Figure 1b is a schematic diagram of the reconstructed video output by the video decoder shown. [Figure 3]Schematic diagram of a coded video for explaining the time scale at which the target spatial resolution varies compared to the random access distance / pitch. [Figure 4a] Schematic diagram of a decoder operating according to different deformation modes that can be operated by the decoder of FIG. 1b, where the deformation modes with different resolutions at which pictures are buffered in the DPB are shown. [Figure 4b] Schematic diagram of a decoder operating according to different deformation modes that can be operated by the decoder of FIG. 1b, where the deformation modes with different resolutions at which pictures are buffered in the DPB are shown. [Figure 5a] Block diagram of an encoder that forms a possible implementation form of the encoder of FIG. 1a and operates according to different deformation modes in which pictures are buffered in the decoded picture buffer at the reference spatial resolution, thereby conforming to the decoder of FIG. 4b, but the side or region where prediction correction is executed is different, that is, at a varying target spatial resolution or reference spatial resolution. [Figure 5b] Block diagram of an encoder that forms a possible implementation form of the encoder of FIG. 1a and operates according to different deformation modes in which pictures are buffered in the decoded picture buffer at the reference spatial resolution, thereby conforming to the decoder of FIG. 4b, but the side or region where prediction correction is executed is different, that is, at a varying target spatial resolution or reference spatial resolution. [Figure 6] Diagram showing a possible syntax for signaling a varying target spatial resolution. [Figure 7] Diagram showing an example for the SPS syntax including a flag for switching on and off the possibility of varying the target spatial resolution. [Figure 8] Diagram showing a syntax example for the SPS for transmitting different target spatial resolution settings on a larger scale, such as for a sequence of pictures, which enables pictures to reference any of these settings in the data stream by indexing. [Figure 9]Figure 8 shows an example of slice segment header syntax, where default target spatial resolution settings, such as those signaled, are referenced by index syntax elements. [Figure 10] This figure shows an example syntax for signaling the output aspect ratio to a certain resolution, such as a reference spatial resolution or one of the target spatial resolutions. [Figure 11] This figure shows an example of syntax for sending output aspect ratios for all possible target spatial resolution settings. [Figure 12] A figure showing an alternative syntax in which the number of different output aspect ratio settings can be signaled, where the number is equal to or greater than the number of possible target spatial resolution settings. [Figure 13] This is a schematic diagram illustrating the possibility of signaling the relative tiling of pictures based on their fluctuating target spatial resolutions using a single flag. [Figure 14a] This is a schematic diagram illustrating an example of scene division or area-based packing. [Figure 14b] This is a schematic diagram illustrating an example of scene division or area-based packing. [Figure 15a] This is a schematic block diagram showing a video encoder according to one embodiment, in which each video encoder operates in response to scene division or region-based packing. [Figure 15b] This is a schematic block diagram of a video decoder that fits the video encoder in Figure 15a, which operates similarly in response to or performs video decoding that depends on the scene segmentation. [Figure 16] Figures 15a and 15b are schematic diagrams illustrating the coding segments used for video coding and video decoding in the video encoder and decoder, as well as the corresponding coding segment information transmitted within the data stream, in order to explain the potential dependencies between video coding and video decoding in scene segments. [Figure 17]This schematic diagram illustrates an example of a coding division using a sequence of normal tree block divisions into an array of tree blocks, followed by a hierarchical multitree subdivision of the tree root block, in order to explain the parameterization of such coding divisions and to illustrate the possibility of considering scene divisions within such coding divisions. [Figure 18] This figure shows an example of syntax for transmitting coding partitioning parameterization within a data stream, specifically within SPS, as illustrated in Figure 17. [Figure 19a] This is a schematic diagram illustrating different possibilities for using scene segmentation to essentially improve the coding segmentation transmitted within a data stream. [Figure 19b] This is a schematic diagram illustrating different possibilities for using scene segmentation to essentially improve the coding segmentation transmitted within a data stream. [Figure 19c] This figure shows a syntax example for a CU partition syntax for partially partitioning a block extending from its own RWP region into an adjacent RWP region, while avoiding signaling partition indicators for partition decisions that are redundant or that could otherwise be inferred by examining the location of each region to be partitioned with respect to the boundary between the current RWP region and the adjacent RWP region. [Figure 19c-1] This figure shows a syntax example for a CU partition syntax for partially partitioning a block extending from its own RWP region into an adjacent RWP region, while avoiding signaling partition indicators for partition decisions that are redundant or that could otherwise be inferred by examining the location of each region to be partitioned with respect to the boundary between the current RWP region and the adjacent RWP region. [Figure 20a] This schematic diagram shows that the current interpretation block references a portion of the reference picture within the picture region corresponding to dual scene sample density / resolution. [Figure 20b] This is a schematic diagram showing the opposite situation compared to Figure 20a. [Figure 21] This is a schematic diagram showing the current interpretation block and the derivation of the motion vector predictor for this block, based on adjacent blocks existing within adjacent picture regions with different scene sampling densities / resolutions. [Figure 22] This schematic diagram shows the scaling of motion vector predictors derived from adjacent blocks that both reside within the same picture region, but where the motion vectors of the adjacent blocks point to picture regions with different scene sampling resolutions / densities. [Figure 23] This is a schematic diagram illustrating the application of a filter across picture regions with different scene sampling resolutions / densities. [Figure 24a] The figure shows example syntax for syntax that signals different quantization parameters to control noise colorness in order to vary this quantization parameter within the picture and within the picture region. [Figure 24b] Figure 24a is a schematic diagram of transformation blocks, such as the predictive residual transformation block, to illustrate how the quantization parameters can affect noise colorness. [Figure 25a] This schematic block diagram shows an encoder according to one embodiment, i.e., a possible video encoder, in which the encoder locally varies quantization parameters that affect the noise colorness within the coded picture. [Figure 25b] This is a schematic block diagram of a decoder compatible with the encoder in Figure 25a. [Figure 26] This is a schematic diagram showing the coded picture and the modified form of the target spatial resolution within its sub-regions. [Figure 27] This figure shows an example syntax for signaling a fluctuating target spatial resolution. [Figure 28]This is a schematic diagram illustrating the variation in target spatial resolution when implementing or modifying the video encoder and decoder in Figures 1a and 1b to operate with a spatially varying target spatial resolution, with or without the temporal change in target spatial resolution described in Figure 2a. [Figure 29] Figures 26-28 illustrate syntax examples to show that spatial differences in target spatial resolution within several pictures can be handled using syntax elements, resampling indicators, or other indicators that show several target spatial resolution values ​​for each block into which each picture is divided. [Figure 30] This figure shows a syntax example that differs from Figure 29 in that the transmission of syntax elements is performed at the coding unit level, i.e., for the block on which the prediction mode determination is performed, rather than for the transformation block referenced in the exemplary Figure 20. [Figure 31] This figure shows a syntax example for SPS, which suggests the possibility of restricting the transmission of syntax elements shown in Figures 29 and 30 to blocks within some default block size ranges, while assuming that a reference spatial resolution must be used for encoding / decoding blocks outside that range. [Figure 32] This figure shows a syntax example that differs from Figure 29 in that the transmission of syntax elements is limited to a predetermined block size. [Figure 33] This figure shows an example syntax for SPS signaling a quantization step size offset, which should be used for resampled blocks of a picture, i.e., blocks that have not been coded / decoded at the reference spatial resolution. [Modes for carrying out the invention]

[0014] This description begins with embodiments relating to the first aspect of this application. Several further aspects will be described later. Note that the aspects of this application may be combined as outlined below in more detail.

[0015] A first aspect of this application relates to the dynamic variation of target spatial resolution on a time scale finer than the random access pitch or distance at which video coding is performed. Further aspects of this application will be described later, and it should be noted that these aspects of this application may be used together to form further embodiments beyond those described herein, and such combinations will also be described below.

[0016] For example, an embodiment of the present application relating to a first aspect may enable a hybrid block-based motion-compensated video encoder to dynamically change the spatial resolution, which is measured, for example, as the amount of samples per field of view in which a picture is depicted, and the change relates, for example, to the input spatial resolution. Coding takes advantage of this fact in terms of coding efficiency. Later, there are also embodiments and aspects described in which such a hybrid block-based motion-compensated video encoder is given the opportunity to dynamically change the spatial resolution with respect to picture subsections as well.

[0017] To date, the signaling needs of such codec systems remain unresolved. For example, the single-layer-based version of HEVC does not allow picture resolution to change within the coded video sequence. This is only permissible at random access points without a leading picture, i.e., in IDR RAPs that completely reset the decoded picture buffer, where the resolution change can occur at the start of a new coded video sequence, including flushing the decoded picture buffer. However, there are scenarios where a resolution change is desirable at a location in the bitstream different from that of an IDR / BLA-based RAP. Initially, an open GOP coding structure would use a RAP type with greater coding efficiency, i.e., a reference picture preceding the RAP, accompanied by a leading picture following the RAP in bitstream order. Here, the RAP is, for example, a CRA picture. This would require resampling of the reference picture preceding the RAP, which is not foreseen in known codecs. Going forward, to avoid the coding efficiency degradation due to predictive reservoir loss associated with such RAPs, the RAP rate should be kept above certain limits. Therefore, it may be desirable to change the spatial resolution within an encoded video sequence without involving the occurrence of RAP, such as within the highest time layer of a hierarchical GOP or within any other subset of encoded pictures.

[0018] The embodiments described herein enable resolution changes within a coded video sequence on a time scale finer than the random access pitch / distance. For example, there are three conceivable variations for performing resolution changes in the middle of such a CVS. Here, the reference spatial resolution may define the highest spatial resolution, such as the maximum value of the target spatial resolution at which the video is coded in the data stream. However, before describing that, the general structure of the encoder and decoder is described with respect to Figures 1a and 1b. Figure 1a shows a video encoder 10 that receives a source video 12 and encodes the source video 12 into a data stream 14 at the target spatial resolution in such a way that the target spatial resolution varies on a time scale finer than the random access distance at which the source video is coded into the data stream 14. Figure 2a shows, for example, a subsequence of pictures from the video 12, here exemplifyingly, three pictures 161, 162, and 163. All of these pictures show the same scene 18 and correspond within the field of view of scene 18, for example, indicated by a rectangle 20 surrounding scene 18. In other words, pictures 161-163 of video 12 may correspond within the scene sections depicted in these pictures. However, as illustrated by the dots in pictures 161-163 in Figure 2a, which represent a kind of spatial sampling for each of the pictures 161-163, the pictures of video 12 are coded into the data stream 14 by encoder 10 with a fluctuating target spatial resolution, that is, with a fluctuating number of samples per scene section or region 20, or, to put it another way, with several different samples per unit area of ​​scene 18, such as a unit field of view. The following description of various implementation variations clarifies what "fluctuating target spatial resolution" may refer to. In detail, the fluctuating target spatial resolution may relate to the predicted residuals into which video 12 is coded into the data stream 14.That is, encoder 10 may be a predictive coder, such as a hybrid video coder that supports inter-predictive mode and intra-predictive mode, and target spatial resolution variation may refer to the resolution at which the predictive residual is coded into the data stream 14. That is, source video 12 has several spatial resolutions, i.e., its pictures at the time it enters encoder 10, but the coding conditions at which encoder 10 performs the transition from input resolution to target spatial resolution may depend on the chosen implementation. In detail, as outlined in more detail below, resampling may be performed as an initial process that naturally precedes all other coding tasks of encoder 10, so that all such other coding tasks operate at this target spatial resolution. These tasks may include, for example, predictive and residual coding. The decoded picture buffer of encoder 10, shown exemplary in 22 in Figure 1a, will thus buffer pictures 161-163 of video 12 at the varying target spatial resolution. This variant will be referred to as variant a below. Alternatively, the decoded picture buffer 22 may buffer the picture of video 12 at a reference spatial resolution, i.e., at some fixed spatial resolution such as the input resolution of video 12, along with a transition between the target spatial resolution at which residual coding is performed and the reference spatial resolution at which it is performed with respect to the predicted or residual signal, and these variations are referred to below as b and c.

[0019] Figure 1b shows a video decoder compatible with the encoder of Figure 1a. The video decoder of Figure 1b is schematically shown using reference numeral 30 and is configured to decode video 32 from data stream 14, i.e., for the reconstruction of video 12. The video decoder 30 is configured to derive from the data stream 14 a variation in the target spatial resolution in which video 32 is encoded, the variation being defined on a time scale finer than the random access distance in which video 32 is encoded therein. Refer to Figure 2b, which shows the reconstructed or decoded video 32 and its pictures 341-343 in a manner corresponding to the method shown in Figure 2a, when the source video 12 entering the encoder 10 is internally resampled in the encoder 10 for encoding. All of the pictures 341-343 show or represent scene 18, and may all have this same scene section encoded therein, for example. In contrast, depending on the implementation configurations described in more detail below, the sample resolution of pictures 341-343 output by the decoder 30 may correspond to the target spatial resolution shown exemplarily in Figure 2a, or it may be constant in time, for example, to the aforementioned reference sample resolution to which these pictures are buffered in the decoded picture buffer 36 of the decoder 30, just as they were in the DPB 22 of the encoder 10. These examples will become clear from the following descriptions of several variations for implementing the encoder and decoder, respectively.

[0020] As explained above, variations in target spatial resolution occur on a time scale finer than the random access distance / pitch at which video coding is performed. As outlined in more detail below and shown in Figure 2a, the time scale at which the target spatial resolution varies may be, for example, picture-unit variation. That is, the target spatial resolution can vary on a picture-by-picture basis. Therefore, encoder 10 can determine how the target spatial resolution should be set on a picture-by-picture basis by signaling the variation in target spatial resolution at the picture level. However, this is not the only possibility. Figure 3 shows a time axis t, with hatching used to show pictures of video 12 at random access points. They appear at several temporal distances Δt, which may be constant as shown in Figure 3, or may vary over time, in which case the random access distance may represent, for example, the average distance between random access point pictures of video 12. The time scale at which the target spatial resolution occurs indicates the minimum duration during which the target spatial resolution can vary between them. Therefore, as shown in the examples in Figures 1a and 1b, the random access distance Δt, 38 is greater than the minimum time distance δt, 40 at which the target spatial resolution can change. Figure 3 shows an example where the target spatial resolution can change from one picture to another in video 12.

[0021] As already shown above, there are different possibilities for implementing the encoder and decoder in Figures 1a and 1b, respectively. These possibilities are the variants a, b, and c shown below. Figures 4a and 4b show variant a, and variants b and c, respectively. Although Figures 4a and 4b focus on the decoder 30, it is clear how the encoder 10 should be adapted to each concept / variant. According to all variants, the predicted residual for each picture is coded in the data stream 14 at a fluctuating target spatial resolution, as illustrated by rectangles of various sizes on the left side of Figures 4a and 4b. According to variant a, each picture is buffered at the target spatial resolution in a reconstructed form. This is also illustrated by showing the pictures of the video in the form of rectangles of various sizes in the DPB 36. The DPB 36 is connected in series into the prediction loop 42, which is also present in the encoder, to maintain synchronization with the decoder 30. According to variant form a, when this reference picture with target resolution A is used as a reference picture by another picture currently being decoded having target spatial resolution B ≠ A, resampling 44 of the reference picture in DPB 36 is performed on the fly. As shown in Figure 4a, the decoder 30 may be a block-based video decoder that supports, for example, inter-prediction mode and intra-prediction mode, and performs transform coding to code the prediction residual, i.e., residual transform coding, and in-loop filtering. As already mentioned, the resolution of the video output, i.e., the picture of video 32, varies. A post-processor, not shown in Figure 4a and not shown in Figure 1b, may be connected to the output of the decoder 30 in order to resample the picture 34 of video 32 to exhibit a constant output sample rate for video presentation.

[0022] Figure 4b shows variant forms b and c. According to both variant forms, the video pictures are buffered in DPB36 at the reference spatial resolution. In other words, each coded picture is reconstructed in DPB36, and then the overall picture is resampled in DPB36 at the highest resolution, if the highest reference spatial resolution is selected. According to variant form b, if necessary, on-the-fly resampling is performed when referenced by pictures of different resolutions, such as lower resolutions, if a resolution greater than or equal to any of the fluctuating target spatial resolution values ​​is used as the reference target spatial resolution. The stream of output pictures, i.e., video 32, is output at the reference target spatial resolution, such as the highest resolution. According to variant form c, the pictures are also stored in DPB36 at the reference target spatial resolution. If the picture resolution of the picture currently being decoded is not equal to the reference spatial resolution, the output picture stream, i.e., video 32, is output at the reference spatial resolution as it was in the case of variant form b, and the residual signal or predicted residual is resampled during reconstruction, such as by bilinear filtering. Thus, in variant form a, DPB 36 is filled with pictures of varying resolutions, and each reference picture is resampled to the resolution of the picture currently being coded / decoded during their reconstruction, whereas in variant forms b and c, DPB 36 is filled only with the highest resolution picture, which is achieved in two different ways.

[0023] Figures 5a and 5b focus on the differences between modified forms b and c. Figures 5a and 5b focus on the encoder side. As already mentioned with respect to Figures 4a and 4b, the prediction loop 42 coexists in the decoder and encoder, and therefore all the tasks and steps described with respect to the decoder are also performed on the encoder side, the difference being that the encoder also selects or chooses the prediction mode, i.e., the parameter setting and coding parameters such as inter-prediction mode or intra-prediction mode, for example, prediction parameters for each mode, quantization of the transform prediction residual, etc. The encoder may do this by performing some rate control and, more specifically, by performing some cost minimization, i.e., aiming to minimize costs depending on the strain and coding rate.

[0024] As shown in Figures 5a and 5b, the encoder 10 is implemented as a block-based predictive video coder. That is, a subtractor 48 subtracts the predictive signal 50 from the source signal 52 to derive an undistorted predictive residual signal 54, which then receives a predictive residual signal 58, which is finally coded into the data stream 14 via several stream multiplexers or modules 60 that perform formatting, entropy coding, and header setup for this data, after undergoing transformation, scaling, and quantization in module 56. To maintain synchronization with the decoder side, the encoder 10 reconstructs the predictive residual signal in the spatial domain by applying the predictive residual 58, which can be derived from the data stream 14, to a scaling and inverse transformation module or an inverse quantization and inverse transformation module 62. One input of an adder 64, which adds this predicted residual signal to the predicted signal 50, is connected to the output of this module 62, and the thus modified predicted signal is subjected to filtering by a filter-controlled analysis module 66 and the subsequent in-loop filter 68. The result is buffered in a DPB 22 to buffer the reconstructed picture and to form a reservoir for a reference picture that will be referenced by the inter-prediction block of the picture to be coded. The motion estimation module 70 has access to the source signal 52 to estimate the motion field of the source video and thus determine the motion vector used to code the video. The picture stored in the DPB 22 is made accessible for the inter-prediction module 72, which generates predictors, i.e., 74, of the inter-prediction block of the picture currently being coded. In the case of an intra-prediction block, there is an intra-prediction module 76 that has access to the modified predicted signal output by the adder 64. The prediction parameters, i.e., the intra-prediction mode, are determined by the intra-picture estimation module 78 based on an analysis of the corrected prediction signal, i.e., the signal that has been pre-reconstructed in the vicinity of the block currently being coded.Prediction parameters for inter-prediction blocks, such as motion vectors, i.e. 80, and prediction parameters for intra-prediction blocks, i.e., for angular modes such as DC mode or planar mode, or intra-prediction modes such as non-angular mode, i.e. 82, are forward transmitted in the data stream 14 to the multiplexer 60 to signal these prediction parameters together with the selected prediction mode, i.e., the inter / intra-prediction mode determination. The selector 84 constitutes the prediction signal 50 by selecting an inter-prediction predictor 74 or an intra-prediction predictor 86 according to the selected prediction mode for various blocks of the picture currently being coded, and thus forms the prediction signal 50. The encoder 10 also includes a general-purpose coder control 88 that undertakes tasks such as rate control and cost minimization.

[0025] Figure 5a focuses on variant b. As shown in Figure 5a, the picture 90 of the inbound video signal 12 is resampled in the initial resampling stage 92 to become a picture 94 of the fluctuating target spatial resolution, and then these pictures 94 enter the encoder loop, i.e., they are subject to subtraction by the subtractor 48. It should be noted that with respect to modules 62 and 64, 66, 68, the DPB 22 for which reference numeral 36 is used as far as the decoder side is concerned, and modules 72, 76, and 84 having subsequent modules that form a prediction loop 42 or are connected in series within the prediction loop 42, this task is transparent to the decoder side corresponding to encoder 10 in its internal structure. The target spatial resolution fluctuation can be controlled by a general-purpose coder control 88 as shown in Figure 5a, depending on rate control and / or cost minimization. Since the picture 94 that forms the source signal 52 for the encoder 10 in Figure 5a has a fluctuating spatial resolution, the encoder 10 in Figure 5a includes a resampler that performs resampling to or from the target spatial resolution to a reference spatial resolution. For example, the resampler 96 is connected into the forward prediction loop 42 of the DPB22 to resample the picture before buffering it into the DPB22 at the reference spatial resolution. That is, resampling is performed whenever the target spatial resolution of the picture currently being coded is not equal to the reference spatial resolution. Furthermore, the resampler 98 is connected between the DPB22 and the interprediction module 72, together with the motion estimator 70 which also has access to the sampled result, to resample the reference picture referenced by the picture currently being coded. That is, the resampler 98 resamples the buffered picture in the DPB22 from the reference spatial resolution to the target spatial resolution. Other blocks and modules shown using hatching are also configured to handle the fluctuating target spatial resolution of the inbound picture 94.

[0026] Figure 5b focuses on variant form c, where the source signal 52 is directly formed by the video 12. In the example of Figure 5b, the input resolution of the source video 12, i.e., the source resolution, is used as the reference spatial resolution, although this example may differ in that a resampler, such as a resampler 92, performs the initial resampling from the source resolution to the reference spatial resolution. According to variant form c, as shown in Figure 5b, a resampler 100 is connected between the output of the subtractor 48 and the input of the transform, scaling, and quantization module 56 to resample the undistorted predicted residual signal 54 to a fluctuating target spatial resolution before the transform and quantization in module 56. A resampler 102 performs the reverse process, i.e., resampling from the target spatial resolution to the reference spatial resolution, by being connected on the one hand to the output of the inverse transform module 62 and on the other hand to the first input of the adder 64, i.e., the predicted correction performed by the adder 64. That is, according to variant form c shown in Figure 5b, the resampling step is performed on the predicted residual signal, thereby enabling reconstruction at the highest resolution.

[0027] The following explanation focuses on several signaling possibilities to address the needs arising according to variant forms a-c.

[0028] As already explained above, the target spatial resolution may vary on a time scale corresponding to the picture pitch or frame rate. Figure 6 shows, for example, that the target spatial resolution signaling 104 may be included in the picture parameter set of the data stream 14, i.e., PPS 106. This may be done according to variations a to c. The signaling 104 in the example of Figure 6 indicates the picture size horizontally 108 and vertically 110. The possibility of using a variable target spatial resolution may be a mode signaled to be switched on or off via a flag 112 included in PPS 106, so that when activated, the target spatial resolution of the corresponding picture is indicated by using syntax elements 108 and 110 indicating the corresponding picture size, measured in units of the number of samples in x and y, or not when not activated.

[0029] Sequence range constraint flags can trigger whether dynamic resolution is allowed or not. Such flags may be included in the SPS (Sequence Parameter Set) or in larger range parameter sets, such as those for the entire video, as in the VUI. Figure 7 shows an example of such a flag, namely flag 116, being included in the SPS 114. The latter flag 116, namely SPS_dynamic_resolution_enabled_flag, is set to 0 to signal that the value of flag 112, namely PPS_dynamic_resolution_enabled_flag, should not be set to 1.

[0030] An alternative example would be that the picture size set (Pie Size Set) is defined within a parameter set with a larger range than PPS, such as SPS or VPS, and that an index to that set is included in or associated with every video slice NAL unit in the slice header. Examples are shown in Figures 8 and 9. In this case, SPS 114 includes a target spatial resolution signaling 118 if subsequent flags are set beyond flag 116, and the target spatial resolution signaling 118 signals the picture size or target spatial resolution once for each possible value or state of the target spatial resolution via syntax elements 108 and 110, an example of which is shown in Figure 9. A slice header 120 references one of these possible values ​​or states via index 122, thereby indicating a reference to the respective target spatial resolution indicated via the signaling 118 for each picture to which the slices of the slice header 120 belong. It may be necessary for all slices within a single picture to indicate the same target spatial resolution or to be indexed to the same target spatial resolution.

[0031] According to one embodiment, it is possible to signal within the data stream 14 which possibility is being used to allow variation in the target spatial resolution. For example, the flag output_scaling_flag or some other signaling may indicate whether resampling is performed as a pre-processing / post-processing step or as a normative in-loop resampling step. This makes it possible to distinguish between variant forms b and c, where resampling is performed within the decoding loop 42, and variant form a, i.e., resampling outside the decoding loop. In this case, multiple modes, i.e., variant form a on the one hand and at least one of variant forms b and c on the other, are implemented in the codec and encoder, decoder, respectively. In this way, the decoder is made able to constitute further lower elements of the multimedia processing chain, down to an indication of whether dynamic adjustment of the resolution of the decoded video signal should be performed before presentation.

[0032] Further signaling, which may be included in or inserted into data stream 14, may include aspect ratio signaling. In detail, HEVC has several bitstream signaling modes, for example, that are sequence-specific and not permitted to be changed within the CVS. As in the embodiments described so far, the resolution, i.e., the target spatial resolution, may be allowed to change within the coded video sequence CVS, and such signaling should be dynamic, i.e., these modes should be described dynamically. For example, taking the sample aspect ratio signaling mode, reducing the resolution of the picture in source signal 12 can be achieved in several ways, i.e., by resampling horizontally, or vertically, or by resampling in both directions. All three of the options just mentioned require different resampling ratios for a given target resolution, which will result in different sample aspect ratios. Thus, changing the resolution will result in changing the coded sample aspect ratio. One option is to specify a default way in which the aspect ratio applies to one of the picture sizes, such as the first or largest picture size specified in the SPS across all possible picture sizes. Another possibility is to specify a reference picture size to which the signaled aspect ratio applies in order to derive which is the sample aspect ratio for other formats, and leave it to the decoder 30 or some renderer connected to the decoder's output. See, for example, Figure 10. Figure 10 shows that the data stream may contain information or signaling indicating the output aspect ratio in 126. Figure 10 shows that this information 126 may be contained in vui124, but this is just an example.In other words, the signaling 126 indicates an aspect ratio 128 in x (horizontal) and an aspect ratio 130 in y (vertical) for one possible setting of the target spatial resolution, such as the reference spatial resolution, and a decoder or some device receiving the video 32 uses this aspect ratio to present the video 32 to a picture at the respective spatial resolution. If the picture size of the video 32 is varied, such as in variant form a, the aspect ratio signaling 126 is used for the picture at the corresponding spatial resolution, such as the reference spatial resolution, and for the picture at a different resolution, the aspect ratio for presentation is determined based on the signaling 126 and the respective target spatial resolution or picture size of each picture. That is, for a different picture size with pic_width_in_luma_samples[i] and pic_height_in_luma_samples[i], the aspect ratio for that picture is... sar_width[i]=sar_width_default / pic_width_in_luma_samples_default* pic_width_in_luma_samples[i], and sar_height[i]=sar_height_default / pic_height_in_luma_samples_default*pic_height_in_luma_samples[i] It can be derived as follows.

[0033] Note that the example calculation above assumes there are no cropping parameters. `pic_width_in_luma_samples[i]` and `pic_height_in_luma_samples[i]` correspond to the entire decoded picture, while the cropped picture is only a portion of the output. Two different resolutions can use completely different cropping windows, and therefore the above calculation requires performing cropping before calculating the relationship between `sar_width` and `sar_height` at different resolutions. That is, croppedWidth[i]=pic_width_in_luma_samples[i]-(conf_win_right_offset[i]+conf_win_left_offset[i]), croppedHeight[i]=pic_height_in_luma_samples[i]-(conf_win_top_offset[i]+conf_win_left_offset[i]), therefore, sar_width[i]=sar_width_default / croppedWidth_default*croppedWidth[i], and sar_height[i]=sar_height_default / croppedHeight_default* croppedHeight[i] That is the case.

[0034] However, these sample aspect ratio values ​​are typically used at a clear boundary for post-processing, where the direct values ​​from the VUI are used to constitute the post-processing step. While the above calculation is straightforward and easily performed, a simpler approach that would allow the use of existing boundaries and APIs with minimal changes would be to directly signal different aspect ratios for each resolution, as is done today.

[0035] For example, the first option would be to describe several aspect ratios, one for each signaled picture size, as shown in Figure 11.

[0036] Here, signaling 126 indicates the aspect ratio for outputting the video 32 picture for each possible setting of the target spatial resolution, the number of which is indicated by num_pic_in_luma_samples_minus1. Figure 11 shows that an alternative method or alternative option for indicating the aspect ratio might be to index some of the default aspect ratios by index 132.

[0037] The signaling example in Figure 11 can be further improved and may be more efficient for some cases. It should be understood that in the example in Figure 11, there are as many described sample aspect ratios as there are picture sizes. Alternatively, Figure 11 shows the sample aspect ratios for each possible setting of the target spatial resolution. However, it may occur that the sample aspect ratios are equal for all picture sizes or all possible settings of the target spatial resolution, or that the number of assumed sample aspect ratios is at least less than the number of picture sizes or target spatial resolutions. Therefore, aspect ratio signaling may be performed as shown in Figure 12, where flag 134 can indicate whether all aspect ratios are equal, i.e., all pictures in video 32 have the same aspect ratio. If yes, signaling 126 only needs to be signaled once for all pictures. Otherwise, the number of different aspect ratios is signaled in 136, and the signaling 126 is then included in the data stream as frequently as indicated by 136. For each picture, then it is indicated with respect to which target spatial resolution will be used, i.e., via and indirectly through the picture size indication 104 or 122, this selection of the target spatial resolution to be used also indicates the aspect ratio to be applied, i.e., by associating the correct aspect ratio with each target spatial resolution, and the presentation of the video 32 or the respective picture.

[0038] As explained previously, the following parameters may be included in data stream 14.

[0039] Similarly, signaling for the following parameters may exist within the data stream. min_spatial_segmentation_idc ue(v)

[0040] `min_spatial_segmentation_idc` indicates the parallel processing capability. For example, (when tiles are not used and wavefront processing is not enabled - when `entropy_coding_sync_enabled_flag` is equal to 0) the maximum number of rumor samples contained within a slice is constrained by it. If tiles are used instead, the maximum number of rumor samples contained within a tile is constrained by it. Another parallel processing mode is when `entropy_coding_sync_enabled_flag` is equal to 1 (wavefront processing), where `CtbSize` and picture size determine the number of "wavefront" rows, and the value of `min_spatial_segmentation_idc` sets a constraint on the relationship between `CtbSize` and picture size. For any of these parallel processing modes (slice, tile, wavefront), the value of `min_spatial_segmentation_idc` sets the picture size-dependent constraints described above. The original HEVC specification follows for better understanding.

[0041] - If not (min_spatial_segmentation_idc is not equal to 0), then the bitstream conformance requirement is that exactly one of the following conditions must be true:

[0042] - In each PPS activated within CVS, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0, (4*PicSizelnSamplesY) / minSpatialSegmentationTimes More than 4 spatial samples There are no slices in the CVS that includes this item.

[0043] - In each PPS activated within CVS, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0, (4*PicSizelnSamplesY) / minSpatialSegmentationTimes More than 4 spatial samples There are no tiles in the CVS that includes this.

[0044] - In each PPS activated within CVS, tiles_enabled_flag is equal to 0, entropy_coding_sync_enabled_flag is equal to 1, and the syntax elements pic_width_in_luma_samples, pic_height_in_luma_samples, and the variable CtbSizeY satisfy the following constraint: (2*pic_height_in_luma_samples+pic_width_in_luma_samples)*CtbSizeY≦(4*PicSizelnSamplesY) / minSpatialSegmentationTimes4.

[0045] Considering that the picture size or target spatial resolution can vary per picture, min_spatial_segmentation_idc can be signaled in a more flexible way. Similar to the aspect ratio, the signaling of min_spatial_segmentation_idc could be: The same constraints regarding lumens per slice / tile apply to other picture sizes, for a specific picture size, Using min_spatial_segmentation_idc[i] for each picture size, It is possible.

[0046] Another example is, max_bytes_per_pic_denom ue(v) It is related to.

[0047] max_bytes_per_pic_denom indicates the number of bits that cannot be exceeded by the total size of the VCL NAL units of the CVS-encoded picture ((PicSizelnMinCbsY*RawMinCuBits) / (8*max_bytes_per_pic_denom)). It also depends on the picture size.

[0048] Therefore, similar to the above, for some picture sizes it is as follows, namely, The same constraints apply to any picture size, for a specific picture size: Using min_spatial_segmentation_idc[i] for each picture size, It should be shown.

[0049] Another example is, max_bits_per_min_cu_denom ue(v) It is related to.

[0050] max_bits_per_min_cu_denom indicates an upper limit on the number of coded bits in coding_unit() within CVS. It depends on the Ctb size and minCbLog2SizeY, as follows: (128+RawMinCuBits) / max_bits_per_min_cu_denom*(1<<(2*(log2CbSize-MinCbLog2SizeY)))

[0051] In the multi-resolution coding schemes discussed here, it can be assumed that the Ctb size and minimum coding block size (minCbLog2SizeY) depend on the picture size. Considering this, different values ​​may be expected for different picture sizes.

[0052] Another example is, log2_max_mv_length_horizontal ue(v) log2_max_mv_length_vertical ue(v) It is related to.

[0053] Similarly, constraints on two syntax elements for MV can be used for parallelization. They represent constraints in MV that apply to the coded picture. This information allows the decoder to know whether enough data has been reconstructed from the (still being decoded) picture in order to begin decoding another picture that references it. Since picture sizes can vary, for the previous example, the following option can be used, namely: • For specific picture sizes, the MV is scaled to correspond to each picture size. Using log2_max_mv_length_horizontal[i] and log2_max_mv_length_vertical[i] for each picture size, Displaying images according to their respective picture sizes is required.

[0054] (For example, in some scenarios such as tile streaming where the same content is presented to the client with a varying overall resolution (to handle devices with, for example, a FullHD or 4K video decoder), it may be desirable to maintain a fixed tile structure with respect to the picture plane regardless of the picture size or aspect ratio. One embodiment is that, as illustrated in the upper part of FIG. 13 below, a flag (relative_tiling_flag) indicating the tiled structure signaled at the highest reference spatial resolution (resolution A) (for example, the default structure signaled with a certain sequence range) will be used in the bitstream for pictures with a lower resolution (resolution B) as well, and the tiling structure is scaled to a lower resolution compared to each picture size. In the lower part of FIG. 13, an example for non-relative tiling is shown, where, illustratively, a picture at the resampled resolution B (<A) fits entirely within a single tile of the original resolution. In this embodiment, the tiling structure is signaled only for one resolution, but within a sequence range, for example, in the SPS. Other syntaxes with picture ranges, for example, PPS, omit each syntax.)

[0055] First, this may enable bit savings by avoiding redundant tiling signaling for each resolution. Second, the fact that the amount and relative dimensions of the tiles remain constant over the course of the coded video sequence allows for more efficient resource planning on the decoder side, where the decoder can avoid re-evaluating the degree of parallelization to be used in the decoding process for each individual picture. In particular, the fact that the relative dimensions of the tiles remain constant over pictures of different resolutions can help simplify decoder resource planning and enable an estimate of the decoder execution time for each individual tile.)

[0056] In another embodiment, the signaling may consist of sequence-level guarantees included in the bitstream that indicate that tiles will be aligned for all pictures within the same spatial layer even if the spatial resolution changes within the layer. The decoder may use this guarantee to reserve parallel decoding resources accordingly and to determine whether the decoder is able to decode the bitstream through parallel processing. Multiple tile structures may be sent in the bitstream, all of which are identical or subsampled equivalents of the same tile structure.

[0057] A further embodiment consists of a flag in the bitstream that guarantees that the tile structure will be aligned within the same spatial layer when the resolution changes within that layer, such that for all tile boundaries present at lower resolutions, the aligned tile boundaries will also exist at higher resolutions. Pictures coded at higher resolutions may also contain additional tile boundaries. Multiple tile structures may be sent in the bitstream, but all structures at lower resolutions are identical, and uniform tile structures exist at higher resolutions.

[0058] When the decoder performs random access (RA) to the encoded bitstream, i.e., when decoding begins from any RAP in the middle of the stream, reading pictures belonging to open GOP RAPs such as CRAs are usually discarded, and decoding / presentation begins from the RAP picture. Depending on system variant a, b, or c, the decoder's knowledge of the resolution in the bitstream differs.

[0059] In variant a), the decoder may be notified of the resolution of the current picture that will be present in the DPB at a specific resolution after decoding, via picture-level range syntax in the RAP. Each reference picture in the DPB at that specific resolution, which is required for decoding the current picture at a different resolution, is resampled to the resolution of the current picture.

[0060] In variants b) and c), in RAP, the sequence-level range syntax may inform the decoder about the highest resolution in the bitstream, but the picture-level range syntax informs the decoder about the resolution of the current picture to be decoded, in order to enable the decoder to resample all pictures to the highest resolution.

[0061] The following description relates to an appended aspect of this application, namely enabling video coding for region packing, i.e., scene-level subdivision of a picture into picture regions, each relating to several scene sections or scene regions of a scene. According to this aspect, the video codec responds to region packing or scene packing / subdivision. Scene packing / subdivision may be performed in such a way that different scene regions are transmitted in the data stream at different scene sampling densities or different scene resolutions, thereby resulting in sub-picture scene resolution changes. There are various reasons why it may appear beneficial to vary the spatial scene resolution within a picture of a video sequence during coding, i.e., to represent individual subsections of a scene at different spatial resolutions. Such means may be used either to improve the coding efficiency of the codec, i.e., as another means to reduce the bitrate span for a given picture area, or to reduce the codec memory requirements for processing samples of a decoded picture by reducing the amount of samples before processing the decoded picture to a post-decoder resampling engine. However, video codecs available to date do not care about such measures and system standards that rely on subpicture resampling, such as MPEG OMAF, which perform it along with pre- and post-processing without codec recognition. Here, the picture is divided into subpictures or regions, and these regions are rearranged or packed within the coded picture. In current technology codecs, only the metadata of these packing steps is carried within the video bitstream, i.e., in the form of so-called region-unit packing SEI messages, in order to reverse the processing before presenting the video. According to embodiments further described below, the idea is to facilitate knowledge about the packing format in the codec tool determination and coding process.For example, a normative region-wise packing (RWP) syntax that defines the so-called unpacked picture, i.e., the region of the original picture, may be introduced, for example, into PPS, where all regions are placed with the correct adjacent context at equal spatial resolution and in correspondence with the regions in the packed picture. The image transformations described between the unpacked picture and the regions in the packed picture may include translation, scaling, mirroring, etc.

[0062] Figures 14a and 14b provide examples of two exemplary RWP scenarios. In these figures, the left side shows the unpacked picture 200, i.e., the scene, and the right side shows the packed picture 202, i.e., the coded picture. The unpacked picture 200 and the packed picture 202 are shown at a uniform sample resolution. The packed picture 202 is subdivided into picture regions 204. Each picture region 204 has a corresponding scene section 206 associated with it from the unpacked picture 200, and the correspondence or association is indicated by the use of the letters a, b, and c. The division of picture 202 into picture region 204, indicated by arrow 208, and the mapping of these picture regions 204 into scene section 206 are signaled in the data stream via RWP information or scene packing / division information, and this mapping 208 may be accompanied by different resampling or spatial scaling of the picture region 204, as is evident from Figures 14a and 14b.

[0063] Therefore, according to embodiments further described below, RWP syntax may be used to describe this scene segmentation / packing, and embodiments described below introduce numerous coding tool restrictions, modifications, and signaling that can be used to leverage knowledge about this scene segmentation within the codec. For the sake of understanding the following description, a schematic framework of the encoder and decoder is described with respect to Figures 15a and 15b. Figure 15a shows the video encoder 220. The video encoder 220 is configured to encode inbound video 222 into a data stream 224, where each picture 202 of video 222 is segmented according to the scene segmentation 208 into picture regions 204, each of which maps the respective scene regions 206 of scene 200 onto which. The video encoder 220 supports adjustment of the scene segmentation 208. For example, the video encoder 220 includes an input for receiving information 226 about a scene segment 208, which may be performed by a module connected to the input of the video encoder when providing video 222 based on video data representing a scene 200. The video encoder 220 is configured to signal the scene segment in the data stream 224. For example, the information about the scene segment 208 is illustrated to be present in the data stream 224 in 228 in Figure 15a. In the method described in more detail below, the video encoder 220 is configured to encode the video 222 in the data stream 224, depending on the scene segment 208 or the information 226, respectively.

[0064] A corresponding video decoder is shown in Figure 15b. The video decoder in Figure 15b, i.e., 230, is configured to receive a data stream 224 containing encoded video 222 and to derive scene segmentation information 228 from the data stream 224. As shown by the dashed line 232, the decoder 230 may optionally forward transmit the scene segmentation information, which is output by the video decoder 230 and represents the reconstruction of video 222, to the receiver of the reconstructed video 234. Importantly, the video decoder 230 is configured to decode video 234 from the data stream 224 depending on the scene segmentation information 228. The following description provides a specific example of this dependency.

[0065] More specifically, as previously described, the scene segmentation information 228 may be transmitted in the data stream at several time scales, such as picture level, picture sequence level, or picture group level, or it may be updated intermittently, as in the case of SEI messages, and its range follows until the next update of the scene segmentation 208 via another SEI message. As can be seen from Figures 14a and 14b, for the scene mapping 208, different picture regions 204 of the pictures in the coded video 222 may depict objects in the scene 200 at different scene sampling densities or scene sampling resolutions, for example, as outlined in more detail below, whose environment should be considered in the prediction process. Furthermore, different scene sampling densities for the picture regions, i.e., different numbers of samples per unit area of ​​the scene 200, may result in different coding parameters that are optimal for efficiently coding each picture region 204. Therefore, it may be advantageous to allow setting such parameters on a picture region basis. Furthermore, due to changes in spatial proximity between scene segments 208 and, in particular, picture regions, when comparing picture 202 on one hand and scene 200 on the other, the boundary between adjacent picture regions 204 may represent a boundary where the picture content on both sides of these boundaries does not relate to the directly adjacent portion in scene 200. Therefore, filtering across it, for example, would not make sense. The following explanation illustrates an example of how this should be taken into consideration in the encoding and decoding processes.

[0066] The following explanation concerns, for example, the relationship between coding divisions and scene divisions. To illustrate this, please refer to Figure 16. Figure 16 shows one picture 202 of video 222. To encode video 222 into data stream 224, the video encoder 220 and video decoder 230 divide picture 202 of video 222 into blocks 240. In units of these blocks 240, video 222 is coded into data stream 224. For example, in units of these blocks, the video encoder and video decoder may perform predictive residual coding / decoding. Thus, the division of picture 202 into blocks 240 could be a division into transform blocks. Alternatively, the video encoder and decoder perform video coding / decoding using a switch between intra-predictive mode and inter-predictive mode in units of blocks 240. Thus, a block 240 could be a coding block or coding unit, and the data stream 224 can transmit the predictive mode decision for each block 240. Alternatively, the video encoder and decoder can set prediction parameters for intra-prediction blocks and inter-prediction blocks in units of block 240. That is, block 240 can be a prediction unit or a prediction block. And yet another alternative, block 240 may be the tile to which picture 202 is partitioned, and the coding / decoding of these tiles 240 will be performed without coding interdependencies. The data stream 224 may include coding partitioning information 242 on how picture 202 should be partitioned into blocks 240.

[0067] In the possibilities described below, the video encoder and decoder enable parameterization of coding divisions in such a way that parameterization can be performed individually for the picture region 204 of the picture 202. Coding divisions may be defined, for example, as shown in Figure 17. Pre-dividing of the picture 202 into an array of tree root blocks 244 may pre-divide the picture 202. These tree root blocks 244 may then be subject to hierarchical subtree divisions, as illustrated by the dotted lines in Figure 17. The leaf blocks of this hierarchical multitree subdivision, indicated using reference numeral 246 in Figure 17, may form, for example, block 240. In this case, the size of the tree root block 244, shown in 248 in Figure 17, will define the maximum block size of the resulting block 246. Parameterization may define, for example, this maximum block size. Parameterization may also define a minimum block size 250 for leaf blocks, and / or the maximum number of hierarchical subdivision levels allowed in the hierarchical multitree subdivision. A zero division level corresponds, for example, to no division at all, i.e., leaving the tree root block 244 as is and using it as block 240. A division level of 1 corresponds to a first division of the tree root block 244 into several regions of a first division level. In the example in Figure 17, some leaf blocks 246 are of the first division level and are marked with 1. Regions of the first division level that are further divided into lower regions result in lower regions of a second division level, and there are leaf blocks 246 of such a hierarchical level illustrated exemplarily in Figure 17, marked with 2. Therefore, the maximum number of such hierarchical division levels may be further parameterizations of the coding divisions. The difference between the minimum block size and the maximum block size may also be shown. The encoder may select these parameterizations for each picture region 204 and signal the parameterizations 252 for each picture region 204 in the data stream 224.The video decoder may use parameterization 250 to perform coding partitions based on information 242. For example, based on the maximum block size 248, the decoder may perform a pre-partition to the tree root block 244. Based on the minimum block size, or the representation of the maximum number of blocks at a hierarchical partition level, or the representation of the difference between the minimum and maximum block sizes 248 and 250, respectively, the decoder may infer that some partitions are not feasible in any case. Therefore, in the case of such partitions, the encoder does not signal the corresponding partition flag in the coding partition information 242, and the decoder does not communicate the respective partition flag from the data stream 224. Calling the CTU of the tree root block 244 and assuming the HEVC syntax as the basis, Figure 18 provides an example of partition parameters defined based on the picture region 204. According to Figure 18, different maximum / minimum CTU sizes are defined for each picture region 204, where the CTU sizes defined in HEVC are used as an example. In HEVC, the CTU size is defined collectively for the entire coded video sequence CVS and the entire picture, respectively, as shown in the example in Figure 18. However, the HEVC syntax is extended so that signaling occurs on a region-by-region basis, i.e., for each picture region 204. In detail, Figure 18 shows SPS, indicated by flag 254, that a picture region 204 is defined, and as previously described, they may be of different scene resolutions or scene sampling densities. When a division to a picture region 204 is applied, i.e., when a scene division is applied, the parameterization 250 is signaled multiple times, as shown in 256. The number 256 may be equal to or less than the number of picture regions, and the data stream 224 may contain indices to different parameterization settings for each parameterization region 204 in different pictures.

[0068] Alternatively, the parameters can be implicitly derived from a single set of default parameters using knowledge about different samplings of regions within the picture.

[0069] Next, the mapping of areas in picture 202 may be done in various ways; for example, slices of picture 202 may be mapped to resolution areas, i.e., to one of the number of parameterizations 250, the number of which is indicated by 256. The mapping may be done in units of independently coded tiles as described above; that is, each tile may be associated with one of the parameterizations 250. Alternatively, at the sequence or picture parameter set level, depending on the desired codec flexibility, picture 202 may be subdivided into resolution areas that are coarser or finer than the divisions of picture 202 into slices or tiles.

[0070] Furthermore, the video encoder and video decoder may use scene division 208 or scene division information 228, respectively, as a source for implicitly defining coding block divisions. That is, both the video encoder and decoder may divide the picture 202 into blocks 240 according to the coding divisions, and encode / decode the video in units of these blocks 240. As described, these blocks may be, to name just a few, transformation blocks, prediction blocks, coding blocks, or tiles. Both the video encoder and decoder will perform the coding division according to the scene division 208. This will be illustrated using tile division as an example. That is, according to the embodiment just described, the RWP or scene division information 228 is used when dividing the picture 202 into blocks 250, or other units of a block-based hybrid video codec, for example, using motion-compensated prediction. For example, if the division into blocks 240 signaled via information 228 does not align with the boundaries between picture regions 204 defined in RWP information 228, and the regions 204 adjacent to the RWP boundary have different characteristics such as different spatial sampling densities or different scene resolutions, then the corresponding block or unit structure may be aligned to the RWP boundary without the need for explicit signaling; that is, an implicit division corresponding to the region boundary signaled in the scene division or RWP 228 is derived. See, for example, Figures 19a and 19b. According to Figure 19a, coding division 242 signals the division of picture 202 into blocks 240. Block 240 may be, for example, a tile. Coding division information 242 may simply indicate, for example, the size of block 240, thereby defining the normal subdivision of picture 202 into an array of blocks 240 of that size. However, there is a block 240 that crosses a boundary 250 between two different picture regions 204a and 204b, which may have different scene sampling resolutions, etc. The boundary 250 can be derived from the bitstream 224 for the decoder 230 based on scene segmentation information 228.Therefore, the encoder 220 and decoder 230 implicitly divide these blocks 240 that cross the boundary 250 into two blocks 240a and 240b along the boundary 250 without needing to explicitly signal this division into coding section 242.

[0071] An alternative method for defining coding divisions and coding division information 242 depending on scene division 228 is shown in Figure 19b. Here, a block or coding unit 240 is aligned to the boundary between picture regions 204a and 204b. That is, the coding division information 242 may, for example, simply indicate the size of the block 240, and the video encoder and video decoder subdivide each picture region 204a and 204b individually along the perimeter or boundary of each picture region 240a and 240b, subdividing the picture region into blocks 240 of that size, starting at several positions, such as the upper-left corner of each region. Blocks 240 that extend beyond the boundary of each coding region to its right edge and bottom may be cropped accordingly so that they end at the respective boundary. The result is shown in Figure 19b.

[0072] Similar to the normal subdivision into block 240, implicit block divisions can also be applied to hierarchical multitree subdivisions, as described with respect to Figure 17. For example, suppose that the leaf blocks of hierarchy level 1 in Figure 17, up to the right within the tree root block 244, will cross the boundary between picture regions. In such a case, the division of this region may be implicit between the video encoder and video decoder, and this signaling of each bit in information 242 may be excluded.

[0073] Therefore, as just mentioned, the use of RWP information 228 as a source for inferring some partitions that are not explicitly signaled in coding section 242 may also relate to coding section 242 including hierarchical multitree subdivisions. This can be applied to completely subdividing blocks into leaf regions, but can also be applied to hierarchically subdividing blocks that overlap with adjacent regions. For example, in Figure 19b, block 240 on the right side of region 204a will cross boundary 250. They will be cropped. They will be processed so that the region extends beyond the perimeter of the picture. If one such block 240 is subject to further multitree subdivision into further leaf blocks, for each of the subset of multitree subdivision decisions for multitree subdivision of blocks crossing boundary 250, the encoder and decoder may infer whether it should be performed, and if so, whether the corresponding region of the current subdivision level will be subdivided or not, and as a result the corresponding region subject to each subdivision decision will become a leaf region of the hierarchical multitree subdivision. Without sending a split bit indicator for each split decision as part of information 242, an inference is made based on the scene division of a given picture, i.e., on whether the corresponding region of the current split level would cross boundary 250 if not split. For each other split decision of the multitree repartition, independent of those split decisions inferred based on RWP information 228, each split decision will be signaled in the data stream as part of information 242 via its respective split bit indicator. Figure 19c presents a syntax example for a CU quadtree split by taking the HEVC syntax for CU splits as a base and highlighting the modifications obtained from leveraging RWP tiling, where it is assumed that Tileld(pixel position) is a function that results in different values ​​for pixels located in different RWP regions 204a and 204b, i.e., on different sides of boundary 250, respectively.The syntax achieves a CU, i.e., a CTB split of region 204a that would cross boundary 250 if not cropped, in a way that the leaf blocks are aligned with region boundary 250, while refraining from signaling a split indicator split_cu_flag for splits that can be inferred based on the position of boundary 250. In 251, split_cu_flag is not signaled if the current region that is subject to a split, which would be signaled by split_cu_flag, is a region that would be split anyway if the region were to cross boundary 250. In 251, hierarchical splitting is skipped for regions that are entirely outside region 204a to which the currently hierarchically split block 240 belongs.

[0074] Another situation in which scene segment 208 may affect the coding / decoding process relates to sub-per interpolation filters and motion compensation. According to embodiments of the present application, the idea here is to facilitate sub-per interpolation filters in a block-based hybrid video encoder / decoder to address potentially different spatial sampling characteristics of the RWP region in the picture during motion compensation prediction whenever the current block subject to motion compensation prediction and the area of ​​the reference picture referenced by the motion vector of the current block are located in two differently sampled regions, namely, picture regions 204 related to different scene sampling densities / resolutions.

[0075] In one embodiment, the following applies: a) When downsampling (from reference to present) 2:1, - some of the full sample positions should be treated as sub-sampling positions, or b) When upsampling (from reference to present) 1:2, - sub-per interpolation should be performed and the resulting sample should be treated as a full-per sample, or c) Otherwise, there is no change, and the normal motion compensation prediction is used.

[0076] For example, consider the following examples of pictures in Figures 20a and 20b, which have two regions (left and right). Each of the 4x4 samples depicting the left side of the picture, with a different viewing angle, will depict image content with a relatively smaller viewing angle but higher resolution / fidelity.

[0077] For example, consider the following example in Figure 20a: area 260 of picture region 204 of the current picture 202 on the right, and picture area 262 of picture region 204 of the reference picture. Both areas 260 and 262 depicted are 4x4 coded samples, and the coded samples are indicated by the cross 264. Due to different scene sampling densities / resolutions, areas 260 and 262 are related to different viewing angles. More precisely, both areas 260 and 262 consist of the same number of coded samples, but the left-hand area 262 depicts image content with a ratio of 2:1, resulting in a relatively smaller viewing angle but higher resolution / fidelity than the right-hand area 260.

[0078] All full-Per sample locations or coded sample locations are shown as crosses in the figure. It is clear that the 2x2 subset of samples on the left, 266, i.e., the samples at the locations marked by the rotated dotted crosses, correspond to areas of the version at a lower sampling resolution; that is, these 2x2 samples on the left, indicated by 268, depict the same field of view as the 4x4 samples on the right.

[0079] In this example, in case a), the dashed rectangle 268 on the right indicates the current block, and the motion vector 270 points to the sample position on the left. Some of the coded full pel positions are treated as half pel positions in this case, and only a subset (rotated dashed cross) is considered for reference (again, enclosed by the red dashed rectangle 266).

[0080] Conversely, in case b), as shown in Figure 20b below, when the current block is located on the left side as marked through the red dashed rectangle 272, and the reference block is located on the right side with a relatively low sampling ratio (1:2), the subper sample position (indicated through the rotated cross 276) is treated as the full sample position for reference and a duplicate portion as indicated by 276.

[0081] In Figure 20b, note that area 260 is an area within the reference picture, area 262 is an area within the current picture, and the scene sampling density / resolution where picture area 260 is located is lower than that of one of the picture areas 204 where picture area 262 is located. In other words, existing samples at a higher resolution, i.e., the cross 264 on the left, can be used and considered as subpels for a lower-resolution block 276, for example, if the resolution ratio is 1:2. For example, if the resolution ratio of the blocks is 1:2, then every two samples in the high-resolution block 272 may be skipped, if the motion vector points to an integer sample position considering the lower resolution. Each of these skipped samples in the high-resolution block would correspond to a halfpel sample if the lower resolution were considered.

[0082] In a further embodiment, motion vector predictors may be scaled within an encoder and decoder based on the characteristics of a region in a packed picture 202 containing the current block for which a motion vector is predicted, and predictor blocks that form a source for several motion vector predictors. For example, Figure 21 shows a packed picture 202 having two picture regions 204a and 204b, where the left region 204a is exemplary horizontally downscaled with respect to an unpacked picture, shown on the left side using reference numeral 200 in Figure 21 and subdivided into a scene region 206a corresponding to picture region 204a and a scene region 206b corresponding to picture region 204b. When block 280 is placed within picture region 204b and the dashed block 280 is coded using interpretation, the exemplary motion vector candidate 282, i.e., the motion vector used to code the adjacent block 284 located in picture region 204a, is scaled according to the resampling that picture region 204 underwent before coding, or more precisely, according to the difference in scene sampling resolution between picture regions 204a and 204b, respectively. In a specific example, the horizontal component of the motion vector 282 will increase according to the difference in scene sampling resolution and will then be used as the motion vector candidate 286 for the current block 280. Thus, the candidate 286 used will represent the scaled version of the motion vector 282 for the adjacent block 284.

[0083] Figure 22 below shows how scaling is performed on two blocks 280 and 284 belonging to the high-resolution region 204b. Note that it is assumed that the low-resolution region 204a of picture 202 is downsampled only within the horizontal region by a scaling factor of 1:2.

[0084] The scaling of predictors in the packed picture region is the result of considering the MV predictors in the unpacked region 200. In FIG. 22, how the predictor MV (in the packed picture 290) is mapped to another vector 292 in the unpacked picture 200 is shown. More specifically, in a given example, we focus on the horizontal component MV when the vertical component is not subject to any scaling from RWP208. x The horizontal component MV x is divided into two parts, namely, MV x1 (a part of the motion vector 290 placed in the low-resolution region 204a) and MV x2 (a part of the motion vector 290 placed in the high-resolution region 204b), which (assuming a 1:2 downsampling factor for the low-resolution region) results in the horizontal component of 2×MV x1 +MV x2 in the unpacked region 200. The horizontal component of the predictor 294 in the unpacked region, which is 2×MV x1 +MV x2 , is the predictor used for the current block 280, and its image in the unpacked region 200 is shown at 280'. This predictor 294 needs to be converted back to the packed region, thereby giving the predictor vector 296. However, this motion vector 296 for the current block has two different parts belonging to different resolution regions 206a, 206b, namely, 2×sMV x1 and sMV x2 (where, as explained, 2×MV x1 +MV x2 =2×sMV x1 +sMV x2 in the unpacked region 200). This results in the scaled motion vector 296 in the packed region 202 where the horizontal component is equal to sMV x1 +sMV x2 .

[0085] With respect to the position of the center of the block and the position of the region, the enumerated motion vector components (and parts) can be calculated. For example, in the example considered above (where MV has a positive sign toward the left), - MV x2 =Blockcenter-RegionBoundary - MV x1 =MV x -MV x2 - sMV x2 =Blockcenter-RegionBoundary(Unpacked Region) - 2×MV x1 = 2 × MV x1 +MV x2 -sMV x2 That is the case.

[0086] Further embodiments of the possible dependencies of the encoder and decoder coding / decoding in Figures 15a and 15b relate to filtering parameters. In video codecs of current technology such as HEVC or AVC, filter parameters, such as post-filters, in-loop filters, or deblocking filters, may be derived based on the characteristics of adjacent blocks to be filtered. Filter parameters may relate, for example, to filter strength, filter transfer function, filter kernel size, or deblocking filter range. On the encoder and decoder sides, the characteristics on which such filter parameters are determined may be predictive modes, such as intra-block size or inter-block size. If different resolution content is mixed within picture 202 before encoding, and no special attention is paid in the in-loop deblocking filtering step, this may lead to visual artifacts along the boundary between two differently sampled regions, such as between two differently sampled picture regions 204a and 204b shown in Figure 23. This is due to the fact that between decoding and presentation on the display, the two regions 204a and 204b can be resampled to have equal resolution, i.e., samples per order, and samples modified by the filter within both regions 204a and 204b, i.e., samples within the filter range 300, may occupy different amounts of order and field of view. To prevent visual artifacts in such cases, according to one embodiment, further filter parameters, such as the intensity of such a filter, or the number of samples from the filter range 300, i.e., the filtering boundary considered / modified by the filtering process, are modified relative to each other in accordance with the resolution of adjacent picture regions 204a and 204b. This is shown in Figure 23.

[0087] In a further embodiment, the dependency of the encoding / decoding process in scene segment 208 relates to the derivation of reference samples for filtering processes, such as an in-loop deblocking filtering process.

[0088] When regions within a picture are packed (for example, when different horizontal and vertical resampling are used across the region and the region is rearranged to derive a rectangular picture plane), adjacent samples in the packed picture may not be adjacent in the unpacked picture. Therefore, naive filtering across boundaries with such characteristics is undesirable. Instead, as part of the present invention, RWP information is used to derive samples in the packed picture that have adjacent relationships in the unpacked picture and for which these samples are used in the filtering process.

[0089] For example, see Figure 14b. Here, the boundary between picture regions 204c and 204b is an artificial or scene-discontinuity boundary. That is, within the packed region, i.e., within picture 202, the portions of picture regions 204c and 204b that touch each other at this boundary 310 do not touch each other within the scene 200. Therefore, it becomes possible to truly interrupt or avoid filtering across this boundary 310. However, according to an alternative embodiment, in accordance with the embodiment just outlined, the encoder and decoder fill the filter kernels of each filter that extends beyond the boundary surface 310 from the currently filtered picture region to the adjacent picture region, such as from picture region 204c to picture region 204d, with samples taken from a portion of picture 202 in scene 200 that borders the currently filtered picture region at the boundary surface 310, such as the left-hand edge of picture region 204a in the example just outlined. In the example of Figure 14b, if picture region 204b is currently filtered and the filter kernel extends beyond the boundary surface 310, as can be seen by looking at scene region 200, it can be filled using the right-hand edge portion of picture region 204a.

[0090] According to another embodiment, the video encoder and decoder exhibit scene segment dependency with respect to encoding and decoding, insofar as quantization is involved.

[0091] When the residual signal of a motion-compensated predictive picture is converted in a video codec, the floating-point conversion coefficients typically undergo irreversible quantization on the encoder side. Aside from uniform quantization based on a given step size (quantization parameter i.e., QP), current codecs allow for modification of quantization toward non-uniformity, for example, by incorporating scaling coefficients into the quantization based on the conversion coefficient base frequency. This makes it possible to implement subjectively optimized encoding schemes. Such scaling lists may be known in advance on the decoder side and activated and indexed by bitstream signaling, but it is also possible to transmit custom scaling lists for each picture within the bitstream.

[0092] When regions within a packed picture depict content with variable properties, coding efficiency gains can be achieved by organizing scaling lists to the region properties. For example, these properties may include resolution / sampling, but may also include content properties, etc. Therefore, it is feasible to signal these organized scaling lists on a picture region 204-unit basis. Figure 24a shows one embodiment in which scaling_list_data()[i] is a syntactic structure that carries the scaling list coefficients specifically for the i-th region.

[0093] Each individual area or region 204 within the coded picture 202 can then reference one of the signaled scaling lists, for example, by index to the list scaling_list_data()[i] from the slice header or picture parameter set. Alternatively, num_scaling_lists_areas_minus1 may be equal to the number of regions in the RWP that implicitly signal a scaling list for each individual region.

[0094] In other words, according to the embodiments just outlined, the video encoder and decoder in Figures 15a and 15b allow setting of quantization parameters in sub-picture units, and more specifically, in picture region units 204. The quantization parameters control the spectral shape of the spatial frequency spectrum of the quantization noise of the quantization. See, for example, Figure 24b. Figure 24b shows a transformed block 350 when it can be coded into the data stream 224 by the video encoder 220 to encode one block of the predicted residuals. The transformed block 350 is obtained by the video encoder using several transforms that spectrally decompose the corresponding residual prediction block in x and y. The prediction coefficients of the transformed block 350 are different horizontal frequencies f x and vertical frequency f y This corresponds to, for example, the DC conversion coefficient value being in the upper left corner, and the highest frequency components in x and y being in the lower right corner, both of which are shown in 352 and 354, respectively. The quantization parameters just described, which may be coded in the data stream 224 in a way that can vary between picture regions 204 within a single picture 202, control the spectral shape of the spatial frequency spectrum of the quantization noise, for example, by controlling the quantization step size to be used to non-uniformly quantize and dequantize the conversion coefficients of conversion 350. That is, rather than scaling the quantization step size for quantizing the conversion coefficients of conversion 350 equally for all conversion coefficients of conversion 350, the quantization parameters described herein vary the quantization step size to be applied when quantizing and dequantizing the conversion coefficients of conversion 350 differently. Such scaling may be carried out by conversion coefficient scaling before quantization while maintaining a uniform quantization step size. For example, the quantization parameters determine the slope of the quantization step size to be used for the conversion coefficients of conversion 350, or otherwise the spectral variation.

[0095] In other words, the encoder 15a and decoder 15b can enable quantization and dequantization with spectrally varying coarseness, i.e., coarseness that varies over the conversion coefficient 352 of the conversion 350, and as a result, the quantization noise also varies spectrally. Spatial frequency f x and f y Variations in quantization noise across regions may be achieved by scaling the transformation coefficients 352 before quantization and inversely scaling the quantization transformation coefficients on the decoder side. More precisely, at least one quantization parameter, which may be varied by the encoder for different regions of the picture, can define a scaling coefficient for each transformation coefficient 352 of the transformation 350. These scaling coefficients define a non-uniform spectral shape. For each region, at least one quantization parameter may individually specify a scaling coefficient for each transformation coefficient 352, or the scaling coefficients for different spectral region transformations 350 may be subdivided into subblocks of the transformation 350 to which the transformation 350 is subdivided in rows and columns, or into subblocks of differently sized dimensions. Alternatively, at least one quantization parameter may determine the spectral shape defined by the scaling coefficients according to a parameterized two-dimensional function that defines the scaling coefficient policy with respect to the transformation coefficients 352. The encoder, for example, divides each transformation coefficient 352 by its respective scale factor before quantization, which uses a uniform quantization function, i.e., an equal quantization function for all transformation coefficients 352 of the transformation 350. The decoder receives the quantization parameters for each region, and therefore the resulting scale factor for each transformation coefficient 352, and scales the received quantization transformation coefficients 352 by multiplying each transformation coefficient 352 by the received scale factor, thereby spectrally shaping the quantization noise according to the received quantization parameters, i.e., individually for each region of the current picture.

[0096] Before moving on to another aspect of this application and describing embodiments relating to this aspect, it should be noted that the possibility described above of varying the quantization parameter to vary the spectral shape of the spatial frequency spectrum of quantization noise across pictures can also be used independently of the scene-partition dependency of encoding and decoding as described so far with respect to Figures 15a and 15b. Thus, Figures 15a and 15b may relate to examples of encoders and decoders, and to enabling the setting of the quantization parameter on a sub-picture unit basis, regardless of whether the picture 202 of the encoded video sequence is subject to arbitrary scene partitioning, and regardless of whether the data stream 224 includes a scene partitioning description 228. However, in either case, the data stream 224 will contain information such as 360 in Figure 24a regarding the sub-picture unit variation of the quantization parameter, through which the spectral shape of the quantization noise is varied as outlined above. It should be noted that there are other possibilities in this regard that affect noise colorness more than those described with respect to Figure 24b.

[0097] With respect to Figures 25a and 25b, which show encoders and decoders respectively, the latter environment will be described in more detail. The encoder in Figure 25a may, but may not be, a video encoder. The encoder in Figure 25a, indicated by reference numeral 320, receives a picture 302 which may or may not be a picture of video 322, and encodes the picture 302 into a data stream 324 using transformation-based predictive coding. That is, the encoder 320 may predict blocks of the picture 302 using one or more prediction modes, such as a spatial prediction mode and, in the case of video coding, a temporal prediction mode, and may encode the prediction residuals using transformation coding. For this purpose, the encoder 320 may subdivide the picture 302 into transformation blocks, i.e., block 304, which the encoder 320 then applies transformation 306 to each to give transformation blocks 350. For quantization, encoder 320 descales this transformation 350 using the spectral shape 309 defined by the scale factor described above, and then applies the result to quantization 311 to give quantized transformation coefficients, i.e., quantized transformations 313, for each transformation block 304. The quantization function used in step 311 may be uniform transformation coefficients, i.e., it does not vary across transformation coefficient positions, but the scale factor defining the spectral shape 309 varies across transformation coefficients in the transformation 350 and is defined via at least one quantization parameter described above. However, encoder 320 varies this at least one quantization parameter within picture 302 in subregions of picture 302, such as subregion 304. In the data stream 324, encoder 320 signals the quantization parameter 328 defining the spectral shape 309 for each subregion 304. In other words, information 328 defines a spectral shape 309 for each sub-region 304 such that a different spectral shape is used for a transformation block 304 in a given sub-region 304 compared to a transformation block 304 in a different sub-region of the same picture 302.The encoder 320 may determine the setting of the quantization parameter 328 for each region 304 as part of rate control and / or cost minimization. Alternatively, the encoder 320 may set the quantization parameter 328 according to several other divisions of picture 302, such as according to divisions of picture 302 into regions of different spatial resolutions to which picture 302 is coded, as exemplary references are made to the following figures, particularly Figure 28, or into regions forming different RWP regions, as described with respect to Figures 24a and 24b. In this case, signaling of such other divisions may signal the divisions of picture 302 into regions 304 in parallel, and optionally, the setting of the quantization parameter for each region 304 may also be inferred from other settings for region 304, such as the target spatial resolution to which each part of picture 302, i.e., each region 304, is coded in the data stream 324.

[0098] Figure 25b shows a decoder 330 adapted to the encoder 320 of Figure 25a. It is configured to receive a data stream 324 and reconstruct a picture 302' from it. For this purpose, the decoder uses transform-based decoding, subdivision of the picture 302 into transform blocks 304, subdivision of the picture 302 into regions 304, and uses quantization parameter information 328 in the data stream 324 to use the same quantization parameters for each region during inverse quantization. That is, the decoder 330 takes the quantization transform coefficients of the transforms 313 from the data stream 324, scales them using the spectral shape 309 determined by the quantization parameters for each region 304 where the corresponding blocks 304 of each transform 313 are located, and performs an inverse transform to the spatial region for reconstruction purposes.

[0099] As already explained, spectral variations in quantization coarseness can be controlled in ways other than combining a spectrally uniform quantizer with spectrally variable scaling. The quantization parameters may be embodied by a list of scale coefficients with respect to different parts of the transformation 350, or the spectral shape 309 may be defined in other ways, such as by using a parameterized function. Furthermore, it should be noted that the encoder and decoder do not have to be transformation-based coders; that is, they may encode the picture 302 into the data stream 324 in a way different from the form of quantization transformation coefficients. Quantization may be performed, for example, in the spatial domain instead. Furthermore, subband decomposition may also be used.

[0100] Picture 302 may be used instead.

[0101] Furthermore, as already explained above, the spectral shape 309 does not need to be transmitted anew in the data stream. Rather, it can be set in a default manner for the encoder and decoder using quantization parameters that simply indicate the use of these default quantization parameters. For example, for some regions 304, the use of default quantization parameters may be indicated as active, but for transformation blocks in regions 304 where the quantization parameters are inactive for it, spectrally uniform scaling is used, i.e., the quantization noise is spectrally flat. In further embodiments, several spectrally varying spectral shapes 309 may be pre-set or represent default settings in the encoder and decoder, where the quantization parameters indicate for each region 304 which of the default quantization parameter settings is used to define the spectral shape 309. Furthermore, for some regions 304, a mixture thereof is possible, and the quantization parameters can explicitly define the spectral shape 309 in the data stream 324 without indexing any default settings. In the latter case, the quantization parameters can then define the spectral shape in a manner determined by the encoder, for example, according to some cost minimization and / or in a manner involving rate control.

[0102] With respect to Figures 1a and 1b, the possibility of a video encoder and video decoder enabling temporal variation in the target spatial resolution on which the video is coded is illustrated. According to the embodiments described below, it is possible to enable several sub-picture resolutions on which the video is coded. According to one embodiment, the sub-picture characteristics relate to predictive residual coding. For example, a system like the one shown in Figure 5a can be implemented on a sub-picture basis, similar to the codec system using full picture resampling (reconstructed picture resampling) in variant b) described with respect to Figures 1a and 1b. In such a system, regions on each picture are depicted at resolutions that vary with respect to each other. After reconstruction, the images are resampled to an equal sampling density.

[0103] Similarly, as shown in Figure 5b, the system can be implemented based on variant c) (residual signal resampling) described with respect to Figures 1a and 1b, where resampling is performed selectively on a sub-picture basis. In such a system, the residual signal is resampled on a sub-picture basis, for example, for each CTU, tile, slice, or otherwise defined region (RWP). For the reconstruction of the coded picture, the residual signal is resampled back to full resolution. Thus, DBP22 / 36 is filled with pictures at the reference spatial resolution, but motion compensation and in-loop filtering are also performed at full resolution. In such a system, residual signal resampling is a normative part of the codec, and therefore the pre / post-resampling filter parameters and the size of the sample array should be clearly specified, for example, with respect to the CTU size.

[0104] Figure 26 shows an encoded picture 400 in such a system, where residual resampling is performed on a CTU basis, i.e., in units of CTU 402. Thus, the block structure carrying the residual signal samples and the number of samples obtained are adjusted according to explicit signaling (resample_idc).

[0105] The syntax element resample_idc can be signaled for each individual CTU402, or for larger groups of CTUs, such as slices, tiles, or RWP regions. The syntax table in Figure 27 gives an example of region-specific signaling variations.

[0106] Similarly, precise subpicture filter parameters, such as filter kernels, must be known to both the encoder and decoder to ensure avoidance of drift between the two. Since subpicture characteristics can vary widely across the picture plane and different content, it is prudent to enable signaling of region-specific filter parameters. Signaling can be derived to carry filter parameters, such as kernels, that have associations with picture regions.

[0107] Therefore, one embodiment according to the aspects just outlined may be a video encoder according to Figure 1a and a video decoder according to Figure 1b, in which pictures of a video are encoded / decoded in a manner in which encoding / decoding is performed at a target spatial resolution, which varies in units of picture regions to which the pictures are subdivided. Figure 28 shows, for example, that pictures 16a-16c of video 12 are subdivided into picture regions 500, and dots with respect to picture 16a exemplify the selection of different target spatial resolutions for these regions 500. The variation is signaled in the data stream, such as using the syntax shown in Figure 27. Internally, the decoded picture buffers 22 and 36 of the encoder and decoder, respectively, may buffer the pictures of the video at a reference spatial resolution, such as the original spatial resolution of video 12, or at several other spatial resolutions as described above with respect to Figures 1a and 1b. Thus, the variation in the target spatial resolution in the spatial term may be related to predictive residual coding. Internally, the video encoder and video decoder may operate according to the modified form b or modified form c described above. As already shown above, the embodiments described with respect to Figures 1a and 1b and Figures 26-28 may be combined with the embodiments described above. Therefore, the descriptions of Figures 1a and 1b, as further specified in Figures 2a-14, are also applicable to the embodiments just outlined, insofar as they relate to the encoder and the internal structure of the encoder, etc. Furthermore, the variable quantization parameter function and variable filter function of Figures 15a-24b may also be used in the encoder and decoder according to the embodiments just outlined.

[0108] As explained with respect to Figures 26-28, in some rate-distortion optimizations in encoder 10, it is possible to implement encoder 10 and decoder 30 in Figures 1a and 1b in such a way that the target spatial resolution is determined for each CTU or prediction unit to which each picture is subdivided. The encoder and decoder will perform resampling for each CTU, or each prediction unit, or for several other picture blocks, for those that do not have a reference spatial resolution as described with respect to variant forms b and c in Figures 4b and 5b, respectively. Doing so will result in or lead to higher coding efficiency. For example, when encoding a resolution at a target spatial resolution lower than the reference spatial resolution, such as the original full resolution of the inbound video 12, the residuals of the sampled version are encoded, thereby reducing the number of coefficients. Such a mode may be signaled in the data stream 14 as a new mode or using an additional flag. Signaling can be performed per CTU, i.e., per tree root block, as shown in Figure 26, where block 402 is the tree root block, as illustrated on the right side of Figure 26. Figure 29 presents an example where a flag, i.e., res_subsampled_flag, signals whether resampling is applied, in which case there are only two possible values ​​for the target spatial resolution for different blocks, for example. Alternatively, when the flag in Figure 29 is set, further syntax elements not shown in Figure 29 may be signaled in the data stream, in which case the syntax elements indicate a target spatial resolution different from the reference spatial resolution, and when the flag in Figure 29 is not set, it indicates that the block referred to by this flag is of the reference spatial resolution, i.e., resampling is not required.

[0109] Figure 29 shows that this flag can be transmitted at the conversion block level, but it can also be transmitted at the coding unit level where the prediction mode, i.e., intra / interprediction mode, is varied and signaled. Therefore, according to Figure 30, the flag in Figure 29 will not be repeated for smaller blocks, which would increase the signaling. All residual_coding() syntax within the CTU can, for example, use the same flag.

[0110] Therefore, the flags in Figures 20 and 30 represent the possibility of signaling variations in the target spatial resolution within each picture.

[0111] Figure 31 illustrates that the aforementioned flag res_subsampled_flag may not necessarily be sent to all blocks to which some pictures are segmented. As described above, a picture may be subdivided by the encoder and decoder into blocks of varying sizes, and, as shown in the example in Figure 31, signaling to vary the target spatial resolution compared to some reference spatial resolution may be sent only to blocks of some size within some block size range, for example, to the entire video or to some sequence of those pictures. Figure 31 provides an example, where it is assumed that the subsampling of residuals is more beneficial for blocks of a given size and yields a greater gain for larger blocks, for example. Therefore, syntax is added in data stream 14 to avoid the avoidance of syntax for varying the target spatial resolution for all blocks to which some pictures are segmented. In other words, the example in Figure 31 avoids the signaling overhead spent on varying the target spatial resolution across some pictures, which is undesirable for rate-distortion gain. As shown in the example in Figure 31, the data stream 14 signals to the sequence of pictures that the target spatial resolution is signaled to the pictures in the sequence of pictures in the following way: the pictures are divided into blocks of varying sizes, and all blocks outside a given range of block sizes are encoded / decoded with a reference spatial resolution. For other blocks that simply have block sizes within a given range of block sizes, the data stream 14 signals which target spatial resolution is selected for each block.Therefore, in Figure 31, the SPS includes a flag mixed_resolution_enabled_flag that switches on target spatial resolution variation across the picture spatially, and if set, the block size range is determined by two syntax elements, namely log2_min_res_subsampled_flag_present_block_size_minus3 and log_diff_max_min_res_subsampled_flag_present_coding_block_size. Taking the transmission of target spatial resolution indications on a CU basis, i.e., via res_subsampled_flag as an example, Figure 32 shows that each syntax element indicating the target spatial resolution for some blocks is transmitted only when the block size drops to a predetermined signaled block size range.

[0112] Figure 33 relates to the following issue. When downsampling and upsampling only result in distortion, the res_subsampled_flag described above cannot inherently provide better rate-distortion performance, but it becomes necessary to describe how well the sampled blocks are encoded. Therefore, for those blocks that have a target spatial resolution other than the reference spatial resolution, it becomes possible to add delta_qp, i.e., a variation in the quantization parameters compared to the unresampled blocks. Figure 33 shows an example where this is transmitted in data stream 14 as part of SPS, but it can also be transmitted in PPS. That is, here, the syntax element delta_qp_4_subsampled_blocks indicates which quantization step size must be applied, or more precisely, which offset compared to the quantization step size is used for the unresampled blocks encoded at the reference spatial resolution, in relation to which some pictures are to be resampled, i.e., which are not equal to the target spatial resolution.

[0113] Similarly, a 2:1 subsampling coefficient may be used, or the subsampling coefficient may be flexible and signaled. Two or more may be signaled and fixed for different frames, per CTU size, or per time level.

[0114] While several embodiments are described in the context of the apparatus, it is clear that these embodiments also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, embodiments described in the context of a method step also represent a description of a corresponding block or item, or a feature of the corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0115] The data stream of the present invention can be stored on a digital storage medium or transmitted over a transmission medium such as a wireless transmission medium such as the Internet or a wired transmission medium.

[0116] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementations may be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH® memory, which has electronically readable control signals stored thereon, and which cooperates (or can cooperate) with a programmable computer system on which each method is performed. Therefore, the digital storage medium may be computer-readable.

[0117] The same embodiment of the present invention comprises a data carrier having electronically readable control signals, which can cooperate with a programmable computer system such that one of the methods described herein is performed.

[0118] In general, embodiments of the present invention can be implemented as a computer program product having program code, the program code being operable to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0119] Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0120] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0121] A further embodiment of the method of the present invention is a data carrier (or digital storage medium or computer-readable medium) comprising a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transient.

[0122] A further embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, for example, over the Internet.

[0123] Further embodiments include processing means, for example, a computer or programmable logic device configured or adapted to perform one of the methods described herein.

[0124] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0125] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0126] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware device.

[0127] The apparatus described herein may be implemented using hardware devices, a computer, or a combination of hardware devices and a computer.

[0128] The apparatus described herein, or any component of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0129] The methods described herein may be performed using hardware devices, using a computer, or using a combination of hardware devices and a computer.

[0130] Any method or any component of the apparatus described herein may be performed at least in part by hardware and / or software.

[0131] The embodiments described above are merely examples of the principles of the present invention. Modifications and variations of the configurations and details described herein will be obvious to those skilled in the art. Therefore, it is intended to be limited only to the scope of the immediate patent claims and not to be limited by the specific details presented through the description and explanation of the embodiments herein. [Explanation of symbols]

[0132] 10 Video Encoders 12. Source video, inbound video signal 14 Data Streams Pictures 161, 162, and 163 18 scenes 20 Scene Sections, Regions 22 Decoded picture buffer 30 video decoders 32 Videos Pictures 341, 342, and 343 36 Decoded picture buffer 38 Random access distance 40 Minimum time distance 42 Prediction Loop 44 Resampling 48 Subtractors 50 Predictive signals 52 Source signal 54 Predicted residual signal 56 Transformation, Scaling, and Quantization Modules 58 Predicted residual signal 60 Multiplexer 62 Scale and Inverse Transform Modules, Inverse Quantization and Inverse Transform Modules 64 Adder 66 Filter Control Analysis Module 68 In-loop filters 70 Motion Estimation Module 72 Interpretation Module 74 Interpreter 76 Intra Prediction Module 78 Intrapicture Estimation Module 80 Prediction parameters for the interpretation block 82 Prediction parameters for the intra-prediction block 84 Selector 86 Intra Predictor 88 General-purpose coder control 90 Pictures 92 Initial resampling stage, resampler 94 Inbound Pictures 96, 98, 100, 102 ReSampla 200 unpacked pictures, scenes, and unpacked areas 202 Packed pictures, current picture, packed area 204 Picture Area 206 Scene Section, Scene Area 208 Scene mapping, scene division 220 video encoders 222 Inbound Video, Encoded Video 224 data streams 226 Information about Scene Classification 228 Scene Classification Information 230 Video Decoders 234 Reconstructed Videos 240 blocks 242 Coding Classification Information 244 Tree Root Blocks 246 Leaf Block 248 Maximum block size 250 Minimum block size, parameterization, boundary 270 Motion vectors 272 high-resolution blocks 276 Low-resolution blocks 280 blocks 282 Motion vectors, motion vector candidates 284 adjacent blocks 286 Candidate motion vectors 290 Motion vectors 294 Predictors 296 Predictor vectors, scaled motion vectors 302 Pictures 304 Transformation block, sub-region 306 conversion 309 Spectral Shape 310 Scene discontinuity boundary 311 Quantization 313 Conversion 320 encoders 322 Videos 324 data streams 330 Decoder 350 conversion blocks, conversion 400 coded pictures 500 Picture Area

Claims

1. A method for video decoding, wherein the method is Decoding from a bitstream of encoded video data a flag indicating whether reference picture resampling is possible and whether the reference picture has parameters that may differ from the corresponding parameters of the current picture, wherein the portion of the current picture refers to the portion of the reference picture, and the parameters include at least one of the height of the picture in the luma sample and the width of the picture in the luma sample. Based on the aforementioned flags, it is determined that reference picture resampling is possible, that the parameters of the reference picture may differ from the corresponding parameters of the current picture, and that it is permissible for the spatial resolution to change within the encoded video sequence. To generate the resampled portion of the aforementioned reference picture, the portion of the aforementioned reference picture is resampled, This includes predicting the current picture using the resampled portion of the reference picture, A method in which the distance between a random access frame and the current picture is greater than the distance between the reference picture and the current picture.

2. Determining that the parameter of the reference picture is different from the corresponding parameter of the current picture, In accordance with the determination that the parameters of the reference picture differ from the corresponding parameters of the current picture, a portion of the reference picture is resampled. The method according to claim 1, further comprising:

3. The aforementioned flag is the first flag, The method further includes decoding a second flag from the bitstream indicating whether the spatial resolution changes. The method according to claim 1.

4. To determine whether the size of the reference picture and the size of the current picture are different, Depending on whether the size of the reference picture and the size of the current picture are different, a ratio representing the size of the reference picture and the size of the current picture is derived. Based on the ratio, the portion of the reference picture is resampled, The method according to claim 1, further comprising:

5. Resampling the portion of the aforementioned reference picture is When the spatial resolution of the current picture is greater than the spatial resolution of the reference picture, the portion of the reference picture is upsampled to match the spatial resolution of the current picture. When the spatial resolution of the current picture is smaller than the spatial resolution of the reference picture, the portion of the reference picture is downsampled to match the spatial resolution of the current picture. The method according to claim 1, including the method described in claim 1.

6. Determine that the temporal distance between the reference picture and the current picture is less than the random access distance, so that no random access points are placed between the reference picture and the current picture. The method according to claim 1, further comprising:

7. An electronic device for video decoding, wherein the electronic device is Decoding from a bitstream of encoded video data a flag indicating whether reference picture resampling is possible and whether the reference picture has parameters that may differ from the corresponding parameters of the current picture, wherein the portion of the current picture refers to the portion of the reference picture, and the parameters include at least one of the height of the picture in the luma sample and the width of the picture in the luma sample. Based on the aforementioned flags, it is determined that reference picture resampling is possible, that the parameters of the reference picture may differ from the corresponding parameters of the current picture, and that it is permissible for the spatial resolution to change within the encoded video sequence. To generate the resampled portion of the aforementioned reference picture, the portion of the aforementioned reference picture is resampled, Using the resampled portion of the reference picture, predict the current picture, A processor configured to perform the following: An electronic device in which the distance between a random access frame and the current picture is greater than the distance between the reference picture and the current picture.

8. The aforementioned processor, Determining that the parameter of the reference picture is different from the corresponding parameter of the current picture, In accordance with the determination that the parameters of the reference picture differ from the corresponding parameters of the current picture, a portion of the reference picture is resampled. The electronic device according to claim 7, further configured to perform the following:

9. The aforementioned flag is the first flag, The processor is further configured to decode a second flag indicating whether the spatial resolution changes. The electronic device according to claim 7.

10. The aforementioned processor, To determine whether the size of the reference picture and the size of the current picture are different, Depending on whether the size of the reference picture and the size of the current picture are different, a ratio representing the size of the reference picture and the size of the current picture is derived. Based on the ratio, the portion of the reference picture is resampled, The electronic device according to claim 7, further configured to perform the following:

11. In order to resample the portion of the reference picture, the processor: When the spatial resolution of the current picture is greater than the spatial resolution of the reference picture, the portion of the reference picture is upsampled to match the spatial resolution of the current picture. When the spatial resolution of the current picture is smaller than the spatial resolution of the reference picture, the portion of the reference picture is downsampled to match the spatial resolution of the current picture. The electronic device according to claim 7, further configured to perform the following:

12. The aforementioned processor, Determine that the temporal distance between the reference picture and the current picture is less than the random access distance, so that no random access points are placed between the reference picture and the current picture. The electronic device according to claim 7, further configured to perform the following:

13. When executed, at least one processor of the electronic device, Decoding from a bitstream of encoded video data a flag indicating whether reference picture resampling is possible and whether the reference picture has parameters that may differ from the corresponding parameters of the current picture, wherein the portion of the current picture refers to the portion of the reference picture, and the parameters include at least one of the height of the picture in the luma sample and the width of the picture in the luma sample. Based on the aforementioned flag, it is determined that reference picture resampling is possible, that the parameters of the reference picture may differ from the corresponding parameters of the current picture, and that it is permissible for the spatial resolution to change within the encoded video sequence. To generate the resampled portion of the aforementioned reference picture, the portion of the aforementioned reference picture is resampled, Using the resampled portion of the reference picture, predict the current picture, Store the command to perform the action, A non-temporary computer-readable medium in which the distance between a random access frame and the current picture is greater than the distance between the reference picture and the current picture.

14. A method for video encoding, wherein the method is The ability to resample reference pictures, The reference picture has parameters that may differ from the corresponding parameters of the current picture, wherein the portion of the current picture refers to the portion of the reference picture, and the parameters include at least one of the height of the picture in the luma sample and the width of the picture in the luma sample. It is permissible for spatial resolution to change within the encoded video sequence, To generate a flag having a value indicating, To generate the resampled portion of the aforementioned reference picture, the portion of the aforementioned reference picture is resampled, The process includes encoding the current picture using the resampled portion of the reference picture, A method in which the distance between a random access frame and the current picture is greater than the distance between the reference picture and the current picture.

15. Determine that the temporal distance between the reference picture and the current picture is less than the random access distance, so that no random access points are placed between the reference picture and the current picture. The method according to claim 14, further comprising:

16. The aforementioned flag is the first flag, The method further includes generating a second flag indicating whether each picture included in the sequence parameter set is allowed to be different with respect to its respective spatial resolution. The method according to claim 14.

17. An electronic device for video coding, wherein the electronic device is The ability to resample reference pictures, The reference picture has parameters that may differ from the corresponding parameters of the current picture, wherein the portion of the current picture refers to the portion of the reference picture, and the parameters include at least one of the height of the picture in the luma sample and the width of the picture in the luma sample. It is permissible for spatial resolution to change within the encoded video sequence, To generate a flag having a value indicating, To generate the resampled portion of the aforementioned reference picture, the portion of the aforementioned reference picture is resampled, Encoding the current picture using the resampled portion of the reference picture, A processor configured to perform the following: An electronic device in which the distance between a random access frame and the current picture is greater than the distance between the reference picture and the current picture.

18. The aforementioned processor, Determine that the temporal distance between the reference picture and the current picture is less than the random access distance, so that no random access points are placed between the reference picture and the current picture. The electronic device according to claim 17, further configured to perform the following:

19. The aforementioned flag is the first flag, The processor is further configured to generate a second flag indicating whether each picture included in the sequence parameter set is allowed to be different with respect to its respective spatial resolution. The electronic device according to claim 17.

20. When executed, at least one processor of the electronic device, The ability to resample reference pictures, The reference picture has parameters that may differ from the corresponding parameters of the current picture, wherein the portion of the current picture refers to the portion of the reference picture, and the parameters include at least one of the height of the picture in the luma sample and the width of the picture in the luma sample. It is permissible for spatial resolution to change within the encoded video sequence, To generate a flag having a value indicating, To generate the resampled portion of the aforementioned reference picture, the portion of the aforementioned reference picture is resampled, Encoding the current picture using the resampled portion of the reference picture, Store the command to perform the action, A non-temporary computer-readable medium in which the distance between a random access frame and the current picture is greater than the distance between the reference picture and the current picture.

Images