Sub-picture dependent video coding concepts

Abstract

Video coding concepts are described that relate to encoding, decoding, extraction and mixing of video data streams that are coded with pictures subdivided into independently coded sub-pictures. The concepts relate to: extraction of sub-picture specific video data streams with subdivided picture layers and non-subdivided picture layers; handling of scaling windows for inter-layer prediction and handling of inter-layer prediction tools for such video data streams; decoder capability requirements for such data streams, determination of layer specific constraints for such data streams; and mixing of sub-pictures coded with different types of independent coding.

Description

[0001] describe

[0002] Embodiments of this disclosure relate to an encoder for encoding video content into a video data stream. Further embodiments relate to an apparatus for extracting a sub-picture-specific video data stream from a multi-layer video data stream. Further embodiments relate to a decoder for decoding a video data stream. Further embodiments relate to methods for encoding video content into a video data stream, methods for decoding a video data stream, and methods for extracting a sub-picture-specific video data stream from a multi-layer video data stream. Further embodiments relate to video data streams.

[0003] In video encoding, images of a video sequence can be encoded into a video data stream using multiple sub-pictures, each of which includes a portion of the original image. In other words, the video data stream can include multiple sub-pictures associated with equal presentation times. By selecting one or more of these sub-pictures for decoding or presentation, the video content to be presented can be selected on the decoder side. For example, such video data streams can be utilized in viewport-dependent streaming scenarios. As in cases where one or more sub-pictures are to be presented instead of the entire image, it may be advantageous to decode only a portion of the video data stream rather than the entire video data stream. This allows for a better rate-distortion relationship in the presented video content at a given decoding cost. For this purpose, a portion of the video data stream can be extracted before decoding.

[0004] Therefore, it is desirable to have the following concept: to encode and process video data streams containing multiple sub-pictures in a way that provides a good trade-off between accurate extraction of sub-picture-specific video data streams and low signaling overhead.

[0005] Embodiments of the first aspect of the invention rely on the idea of ​​distinguishing, in the extraction, definition, or description of a sub-picture-specific video data stream of a multi-layer video data stream, layers into which video is encoded using a non-sub-picture partitioning approach and layers into which video is encoded using a subdivision approach of two or more sub-pictures. Distinguishing between layers encoded using a non-sub-picture partitioning approach and layers encoded using a sub-picture partitioning approach allows the extraction or description of a sub-picture-specific video data stream, which includes pictures extending beyond the sub-pictures represented by the sub-picture-specific video data stream. Thus, for example, embodiments are permitted for the extraction or description of a sub-picture-specific video data stream where the sub-pictures of a layer encoded using a sub-picture partitioning approach depend on the pictures of a layer encoded using a non-sub-picture partitioning approach. Furthermore, in applications such as viewport-dependent streaming, encoding pictures available in a sub-picture-specific video data stream using a non-partitioning approach allows pictures of at least non-partitioning quality to be immediately available even if the viewport changes.

[0006] An embodiment of the first aspect of the invention provides an apparatus for extracting a sub-picture-specific video data stream from a multi-layer video data stream. The multi-layer video data stream (which comprises multiple layers) consists of or includes bitstream portions such as NAL units, each bitstream portion belonging to one layer within a layer. Note that wherever "consists of" appears herein, it should not be construed as requiring the subject to exclusively include the object of that composition; rather, in addition to bitstream portions, the data stream may also include other portions as described herein, the arrangement of which follows certain rules such as the set of bitstream portions belonging to a timestamp within an access unit. According to the first aspect, extracting a sub-picture-specific video data stream from a multi-layer video data stream, for example, using a device for extracting sub-picture-specific video data streams, includes, for each layer in a set of layers (e.g., an output layer set), checking whether the corresponding layer encodes video in a manner that subdivides a picture of the video into two or more sub-pictures, said two or more sub-pictures being encoded into the corresponding layer independently, such that for each picture of the video, the sub-picture of the corresponding picture is encoded into a distinct bitstream portion of the corresponding layer; or whether the corresponding layer encodes video in a non-sub-picture-subdivision manner. If the corresponding layer encodes video in a non-sub-picture-subdivision manner, the extraction includes taking over the bitstream portion belonging to the corresponding layer from the multi-layer video data stream into the sub-picture-specific video data stream, such that the sub-picture-specific video data stream completely encodes the video of the corresponding layer. If the corresponding layer encodes the video in a manner that subdivides the video images into two or more sub-images, then extracting the sub-layer-specific video data stream includes: for each bitstream portion belonging to the corresponding layer, reading information from the bitstream portion that reveals which of the two or more sub-images the corresponding bitstream portion encodes; and if the corresponding bitstream portion encodes sub-images belonging to a predetermined set of one or more sub-images, then taking over the corresponding bitstream portion of the corresponding layer from the multi-layer video data stream to the sub-image-specific video data stream.

[0007] Further embodiments are provided by multi-layer video data streams and encoders for encoding multi-layer video data streams, the multi-layer video data streams including decoder capability requirement information regarding the following: sub-picture-specific video data streams extractable from the multi-layer video data streams, as described by the device for extracting sub-picture-specific video data streams. The decoder capability requirement information is signaled in the video data stream to inform the decoder capability requirement information, taking into account more efficient utilization of decoder capabilities and / or selecting the optimal possible video data stream for decoding based on the capabilities of the decoder to be used for decoding.

[0008] According to an embodiment of a second aspect of the invention, a multi-layer video data stream includes a first layer, wherein a bitstream portion of the first layer encodes the first video in such a manner that a picture of the first video is subdivided into two or more sub-pictures, the two or more sub-pictures being encoded into the bitstream portion of the first layer independently, such that for each picture of the first video, the corresponding sub-picture of the first video is encoded into a distinct bitstream portion of the first layer. The multi-layer video data stream further includes a second layer, wherein a bitstream portion of the second layer has a second video. The bitstream portion of the first layer encodes the first video using vector-based predictions from a reference picture. Furthermore, the bitstream portion of the first layer encodes the first video in such a manner that a picture including the second video is used as a reference picture; and that a vector used in the vector-based predictions (encoded by the bitstream portion of the first layer and composed of the bitstream portion of the first layer) is scaled and offset relative to the picture of the first video and the reference picture, respectively, according to the size and position of a scaling window in the multi-layer video data stream. The concept according to the second aspect includes: in extracting a sub-picture-specific video data stream from a multi-layer video data stream, taking over the bitstream portion belonging to the second layer from the multi-layer video data stream and transferring it to the sub-picture-specific video data stream, such that the sub-picture-specific video data stream fully encodes the video of the corresponding layer therein. The extraction of the sub-picture-specific video data stream further includes: taking over each bitstream belonging to the first layer and encoding one or more sub-pictures belonging to a predetermined set of sub-pictures from the multi-layer video data stream and transferring it to the sub-picture-specific video data stream. Furthermore, the extraction of the sub-picture-specific video data stream includes: signaling a scaling window to fit the first and / or second layers in the sub-picture-specific video data stream, such that the spatial region of the scaling window for the second video's pictures spatially corresponds to the spatial region of the scaling window for the predetermined set of one or more sub-pictures.

[0009] According to the embodiments of the second aspect, it can be ensured that after the extraction of the sub-image-specific video data stream (during which time the size of the first layer's images may change due to sub-image extraction), scaling of the vectors used for vector-based prediction is performed using a scaling window, such that the vectors are scaled to a position and size as expected by the scaling window provided in the multi-layer video data stream. Therefore, the embodiments of the second aspect also take into account the combination of the use of vector-based inter-layer prediction with sub-image extraction in cases where the relative image size between an image and its inter-layer reference image changes due to sub-image extraction, for example, where the reference layer's image is completely forwarded to the sub-image-specific video data stream (e.g., as described with respect to the first aspect).

[0010] According to an embodiment of a third aspect of the invention, in the encoding and / or decoding of a multi-layer video data stream, an inter-layer prediction tool is used to encode a first version of the video from / to a first layer of the multi-layer video data stream. The inter-layer prediction tool is used to predict from a second layer of the multi-layer video data stream, wherein the first version is encoded into the first layer using a first configuration setting for independent encoding of each sub-picture of the first version of the video. Furthermore, a second version of the video is encoded from / to the second layer of the multi-layer video data stream using a second configuration setting for independent encoding of each sub-picture. Depending on whether the first configuration setting and the second configuration setting have a predetermined relationship, a predetermined subset of one or more inter-layer prediction tools is deactivated in the inter-layer prediction from the second layer.

[0011] Therefore, the embodiments according to the third aspect allow for precise control over the use of the inter-layer prediction tool, so that inter-layer prediction can also be used in cases where images of multiple layers of bitstreams are subdivided into sub-images, particularly in scenarios where images of different layers of multiple layers of video data streams are subdivided into different numbers of sub-images, for example, in cases where images of one layer are subdivided into sub-images while images of another layer are not subdivided into sub-images.

[0012] An embodiment of the fourth aspect of the invention provides encoding or processing of a multi-layer video bitstream, wherein images are encoded in one or more first layers in a manner subdivided into independently coded sub-images, and in one or more second layers in a non-subdivided manner. The multi-layer video bitstream encodes a plurality of reference decoder capability requirements sufficient for decoding a set of layers, each layer set comprising at least one first layer of the first layers and at least one second layer of the second layers. For each of the reference decoder capability requirements, the multi-layer video data stream includes information regarding: a first score attributable to the corresponding reference decoder capability requirement of the at least one first layer and a second score attributable to the corresponding reference decoder capability requirement of the at least one second layer.

[0013] Information about the first and second scores is taken into account for a precise determination of the decoder capability requirements associated with each of the several video data streams, which can be extracted from multi-layer video data streams (such as sub-picture specific video data streams).

[0014] An embodiment of the fifth aspect of the invention provides for the selective application of constraints (e.g., layer-specific constraints) to those layers of a set of layers indicated in a multi-layer video data stream, wherein video pictures are encoded therein in a manner that subdivides them into independently coded subpictures. During the extraction of subpicture-specific video data streams, omitting the application of constraints to layers (where the picture size of the layer is preserved, e.g., layers whose pictures are not subdivided) avoids imposing disproportionately tight constraints on non-subdivided layers, when constraints might tighten due to reduced picture size.

[0015] An embodiment of the sixth aspect of the invention provides an indication in a video data stream that indicates whether a picture encoded into the video data stream is encoded by one or more independently encoded sub-pictures, the indication distinguishing different types of encoding independence between the one or more independently encoded sub-pictures and their surroundings. The embodiment of the sixth aspect considers mixing portions of the bitstream (such as different types of NAL units) into a single picture (i.e., an access unit) of the constituent bitstream. Therefore, the embodiment of the sixth aspect considers greater flexibility in the mixing of video bitstreams.

[0016] The embodiments and advantageous implementations of this disclosure are described in more detail below with reference to the accompanying drawings, in which:

[0017] Figure 1 Examples of encoders, extractors, and decoders according to embodiments are shown.

[0018] Figure 2 Examples of encoders, video data streams, and extractors according to embodiments are shown.

[0019] Figure 3 Examples of video data streams and examples of erroneous extraction of sub-image-specific video data streams are shown.

[0020] Figure 4 An example of a sub-image-specific video data stream is shown.

[0021] Figure 5 Examples of an encoder, extractor, and multi-layer video data stream according to a second aspect of the present invention are shown.

[0022] Figure 6 The adaptability of the scaling window according to the embodiment is shown.

[0023] Figure 7 This shows another example of extracting a sub-image-specific video data stream.

[0024] Figure 8 This shows an example of the output image area fraction of a sub-image-specific video data stream.

[0025] Figure 9 Examples of scores for the subdivision and non-subdivision layers required by the reference decoder capability are shown.

[0026] Figure 10 Examples of encoders, video data streams, devices for mixing and / or extracting video data streams, and decoders are shown.

[0027] Figure 11 An example of a bitstream with hybrid decoder-side boundary processing and encoder-side boundary processing is shown.

[0028] Figure 12 An example of a single-layer bitstream with hybrid decoder-side and encoder-side boundary processing is shown.

[0029] Figure 13 An example of a bitstream portion that depends on the type of independence is shown.

[0030] In the following description, embodiments are discussed in detail; however, it should be recognized that these embodiments provide many applicable concepts that can be embodied in a wide variety of video coding concepts. The specific embodiments discussed are merely illustrative of specific methods for implementing and using the concepts, and are not intended to limit the scope of the embodiments. In the following description, numerous details are set forth to provide a more detailed explanation of embodiments of this disclosure. However, it will be understood by those skilled in the art that other embodiments can be practiced without these specific details. In other instances, well-known structures and apparatuses are shown in block diagram form rather than in detail so as not to obscure the examples described herein. Furthermore, unless otherwise specifically indicated, features of the different embodiments described herein can be combined with each other.

[0031] In the following description of the embodiments, the same or similar elements or elements having the same functionality are equipped with the same reference numerals or identified by the same names, and repeated descriptions of elements equipped with the same reference numerals or identified by the same names are generally omitted. Therefore, the descriptions provided for elements having the same or similar reference numerals or identified by the same names are interchangeable or can be applied to each other in different embodiments.

[0032] The following description of the accompanying drawings is based on... Figure 1 The presentation of the encoder, extractor, and decoder begins. Figure 1 The encoder and extractor provide examples of frameworks into which embodiments of the present invention can be built. Thereafter, a description of embodiments of the concepts of the present invention and how such concepts can be built are presented. Figure 1 The description is in the encoder and extractor. However, regarding the subsequent... Figure 2The embodiments described in the following figures can also be used to form something other than according to... Figure 1 The described framework operates through an encoder and extractor. Further note that the encoder, extractor, and decoder can be implemented separately from each other; however, for illustrative purposes, they are presented here... Figure 1 They were described together in the text.

[0033] 0. According to Figure 1 The encoder 40, extractor 10, decoder 50, and video data stream 14

[0034] Figure 1 An example of encoder 40, extractor 10, and decoder 50 is shown. Encoder 40 encodes video 24 into video data stream 14. Video data stream 14 (also referred to herein as a bitstream) may be stored or transmitted, for example, on a data carrier. Video 24 may include a series of pictures, each of which may be associated with a presentation time in presentation time sequence. Video 24 may include a plurality of pictures 26, which are in... Figure 1 The images in 26 are represented by images 260 and 261. Each image in 26 can be compared with a layer (such as...). Figure 1 It is associated with the first layer L1 or the second layer L0 in the matrix. For example, in Figure 1In this example, image 260 is associated with the second layer L0, while image 261 is associated with the first layer L1. Image 260 of layer L0 can form video sequence 240. Image 261 of layer L1 can form video sequence 241. In this example, video sequences 240 and 241 of video 24 may include images representing the same content and associated with equal presentation times but with different resolutions. However, note that video sequences 240 and 241 of video 24 associated with layers L0 and L1 may have different frame rates. Therefore, for example, video sequence 240 does not necessarily include image 260 for every presentation time, while video sequence 241 includes image 261 for the given presentation time. Encoder 40 may encode image 261 of video sequence 241 into video data stream 14 depending on (i.e., referencing) image 260 of video sequence 240 (e.g., referred to as a reference image, for example for inter-layer prediction), where image 260 is temporally co-located with image 261, i.e., associated with the same presentation time. In other words, the image of layer L0 can be a reference image for the image of layer L1. Therefore, decoding image 261 from video data stream 14 may require encoding image 260 into video data stream 14. In these examples, layer L0 may be referred to as the base layer, and the image associated with layer L0 may have a first resolution. Image 261 associated with the first layer L1 (which may be referred to as the enhancement layer) may have a second resolution, which may be higher than the first resolution. For example, encoder 40 may encode into video data stream 14 in video 24 so that layer L0, which represents video 24 at the first resolution, can be decoded from video data stream 14 independently of layer L1, thereby producing video at the first resolution. Where image 261 of layer L1 depends on image 260 of layer L0, video sequences 240, 241 may optionally be decoded together, which may result in video having a resolution higher than the first resolution, such as a second resolution. Therefore, there may be multiple options for decoding video data stream 14, and these individual options are accompanied by different data rates of the data streams to be decoded and result in videos with different resolutions.

[0035] It should be pointed out that, in Figure 1 The number of layers shown is exemplary, and video data stream 14 may have more than two layers. It should also be noted that video data stream 14 does not necessarily include multiple layers, but in some examples, it includes only a single layer, such as in the examples of the embodiments described in Section 6.

[0036] Extractor 10 may receive video data stream 14 and may extract a not necessarily appropriate subset of the bitstream portion of video data stream 14 to provide extracted video data stream 12. In other words, extracted video data stream 12 may correspond to video data stream 14, or may include apportionment of video data stream 14. It should be noted that extractor 10 may optionally modify the content of the bitstream portion of video data stream 14 when forwarding the bitstream portion in extracted video data stream 12. For example, extractor 10 may modify descriptive data including information about how to decode the encoded video data of extracted video data stream 12. Extractor 10 may select the bitstream portion of video data stream 14 to be forwarded to extracted video data stream 12 based on the Output Layer Set (OLS) (which is to be extracted from video data stream 14). For example, extractor 10 may receive an OLS indication indicating the OLS to be extracted or presented. Video data stream 14 may include an OLS indication indicating the set of OLS that can be extracted from video data stream 14. OLS can indicate which layer or all layers of the extracted video data stream 14 should be forwarded, in whole or in part, to the extracted video data stream 12.

[0037] In other words, the OLS may, for example, indicate a (not necessarily appropriate) subset of the layers of the multi-layer video data stream 14. The OLS may be indicated within the multi-layer data stream itself (such as in OLS indication 18), and may be included in the Video Parameter Set (VPS) of the bitstream 14. In fact, more than one such OLS may be indicated in the data stream 14, where the OLS used for extraction is determined by external means (e.g., via API 20). Note that while OLS indication 18 primarily indicates one or more layers of video to be output or rendered, it may also indicate non-output reference layers that belong to one or more output layers (because one or more output layers depend on that layer directly or indirectly (via another reference layer)) and that are not output / rendered. OLS indication 18 may indicate one or more OLS.

[0038] Optionally, the OLS indication 18 may indicate video parameters for the OLS in, for example, a Video Parameter Set (VPS). For instance, the video parameters may indicate reference decoder capability requirements (also referred to as decoder capability requirements, (reference) rank information, or (reference) rank indication) that pose a requirement for the decoder to be able to decode the bitstream described by the OLS. Note that the video parameters may indicate one or more reference decoder capability requirements for an OLS, because the bitstream described by the OLS can also be scalable after extraction by extractor 10 by selecting / extracting one or more layers and / or one or more temporal sublayers of the OLS. For example, a mixer or merger (e.g., Figure 10The device 100 can use the extracted video data stream 12 to form a bitstream, or the decoder can select a sub-bitstream of the extracted video data stream 12 for decoding.

[0039] The extracted video data stream 12 is forwarded to the decoder 50, which decodes the extracted video data stream 12 to obtain the decoded video 24'. The decoded video 24' differs from video 24 in that, relative to video 24, the decoded video 24' may not necessarily include all the content of video 24, and / or may have a different resolution, and / or may have distortion, such as based on quantization loss.

[0040] A layer's image 26 in the video data stream 14 may include one or more sub-images 28, or may be subdivided into multiple sub-images 28. Note that the reference numeral 26 will be used below to refer to an image of a layer; for example, image 26 may refer to image 261 of layer L1 or image 260 of layer L0. The encoder 40 may encode the sub-images 28 of image 26 independently of each other; that is, it may decode one sub-image of image 26 without needing another sub-image 28 of image 26. For example, the extractor 10 may not necessarily forward all sub-images 28 of a layer, but may forward a subset of sub-images 28 of each image in the layer's images 26, or only one sub-image 28. Therefore, the data rate of the extracted video data stream 12 (which may be referred to in this case as sub-image-specific video data stream 12) may be lower than the data rate of the video data stream 14, so that decoding the extracted video data stream 12 may require fewer decoder resources. Figure 1 In the example, extractor 10 forwards image 240 of layer L0 and sub-image 28 of image 261 of layer L1. In this scenario, the decoded video 24' includes decoded video sequence 24'0 representing decoded images of video sequence 240. Furthermore, according to this example, the decoded video sequence 24' includes decoded video sequence 24'1 representing decoded image 26'1, which includes sub-image 28 of image 261 of video sequence 241.

[0041] Note that the encoder not only indicates one or more meaningful / extractable layers in the OLS indication, but also, according to embodiments, provides the data stream with information such as available buffer memory for DPB and / or CPB, processing kernel, desired decoding latency, etc. (the decoder can use this information to determine whether a particular OLS indicated is decodeable by the decoder); this information may be included in the decoder capability requirement information.

[0042] After a very general description of the concepts of multi-layer video data streams, sub-pictures, bitstream scalability, and reference pictures, several embodiments of the extraction process and / or encoding method for implementing the extracted video data stream 12 and associated indications in video data stream 14 are described below. Note that the features described regarding the extraction process also represent a description of the corresponding video data stream (from which the extracted video data stream is to be extracted) and a description of the corresponding encoding process of the video data stream. For example, the features specifying that the extractor 10 derives information from video data stream 14 should also be understood as features of video data stream 14 that make the information deriveable from the video data stream, and should also be understood as features of encoder 40 in correspondingly encoding video data stream 14.

[0043] 1. Based on the encoder 40, extractor 10, and video data stream 14 of the first aspect.

[0044] This section references Figure 1 The embodiments according to the first aspect are described, wherein the details described in Section 0 may optionally be applied to the embodiments according to the first aspect.

[0045] According to the embodiment of the first aspect, Figure 1 The extractor 10 (which may also be referred to as a device 10 for extracting sub-picture-specific video data streams) is configured to extract sub-picture-specific video data streams 12 from a multi-layer video data stream 14. That is, according to the first aspect, the video data stream 14 includes multiple layers and includes bitstream portions 16, each bitstream portion 16 belonging to one layer (e.g., layers L0, L1) of the multi-layer video data stream 14. For example, each bitstream portion 16 may include an indication such as a layer ID indicating the layer to which the corresponding bitstream portion 16 belongs. According to an embodiment of the first aspect, the extractor 10 is configured to, for each layer in the layer set, check whether the corresponding layer encodes the video 24 in such a manner that the images of the video 24 (e.g., images 261, 262, i.e., images encoded into the corresponding layer) are subdivided into two or more sub-images 28, the two or more sub-images 28 being encoded into the corresponding layer in a mutually independent manner, such that for each image 261, 262 of the video 24, the sub-image 28 of the corresponding image is encoded into a different bitstream portion 16 of the corresponding layer; or whether the corresponding layer encodes the video 24 in a manner that is not subdivided into sub-images.

[0046] For example, a tier set is a tier set indicated by OLS, such as, for example, regarding Figure 1 As described, the extractor 10 is instructed by external means such as API 20 to extract which OLS, or the extractor infers which OLS to extract, for example, in the absence of corresponding instructions.

[0047] For example, encoding sub-pictures 28 independently of each other can mean that each sub-picture 28 is encoded independently of any other sub-pictures (of the picture to which it belongs), and that each bitstream portion 16 encodes only one sub-picture (no more than one sub-picture, but may be only a portion of a sub-picture). In other words, independently encoded sub-pictures 28 may not require any bitstream portion other than the bitstream portion that encodes the picture to which the sub-picture 28 belongs in the bitstream portion 16. Encoding video 24 using a non-sub-picture subdivision method can mean, for example, that the number of sub-parts encoded into the corresponding layer is 1, or that the number of sub-parts encoded into the corresponding layer is 1. In other words, encoding using a non-sub-picture subdivision method can mean encoding in a manner where each picture is encoded as a sub-picture.

[0048] If the corresponding layer uses a non-sub-image subdivision method to encode the video into it (e.g., Figure 1 If the layer L0), then the extractor 10 can extract the bitstream portion 16 (belonging to the corresponding layer). Figure 1 The dotted bitstream portion 16 in video data streams 14 and 12 is taken over from the multi-layer video data stream 14 and transferred to the sub-picture-specific video data stream 12, so that the sub-picture-specific video data stream 12 causes the corresponding layer (e.g., Figure 1 The video 24 of the video sequence 240 is fully encoded therein. Encoding the video of the corresponding layer fully therein may mean, for example, that all bitstream portions 16 of the bitstream portion 16 of the corresponding layer (e.g., layer L0 of multi-layer video data stream 14) are included in the sub-picture-specific video data stream 12. Alternatively or additionally, encoding the video 24 of the corresponding layer fully into the sub-picture-specific video data stream may mean that the bitstream portion 16 taken over in the sub-picture-specific video data stream 12 is independent of the selection of which of two or more sub-pictures of another layer is forwarded in the sub-picture-specific video data stream 12, i.e., which belongs to a predetermined set of one or more sub-pictures mentioned below.

[0049] If the extractor finds that the corresponding layer (i.e., the currently inspected layer) encodes video 24 (e.g., video sequence 241 of layer L1) in a manner that subdivides the video image 26 of the layer (e.g., image 261 of layer L1) into two or more sub-images 28, then the extractor 10 can encode video 24 (e.g., video sequence 241 of layer L1) belonging to the corresponding layer (such as...) in a manner that subdivides the video image 26 of the layer into two or more sub-images 28. Figure 1Each bitstream portion 16 of layer L1) in the layer reads information from that bitstream portion 16, which reveals which of the two or more sub-pictures is encoded into that bitstream portion 16. If the corresponding bitstream portion 16 of the corresponding layer encodes a sub-picture belonging to a predetermined set of one or more sub-pictures, the extractor can take over the bitstream portion 16 of that layer from the multi-layer video data stream 14 to the specific data stream 12 of the sub-picture. For example, in Figure 1 In this context, the predetermined set of sub-images includes only the sub-images 28 shown by crosshairs. For example, extractor 10 may discard, leave, or remove (i.e., not forward) sub-images belonging to the corresponding layer but encoding sub-images that do not belong to the predetermined set into each bitstream portion therein.

[0050] Figure 2 More details are shown regarding such Figure 1 The scenario shown, according to an embodiment of the first aspect of the invention, involves signaling information in video data stream 14 and sub-picture-specific video data stream 12. According to... Figure 2 The exemplary illustration shows that the predetermined set of sub-images includes only one sub-image 28, namely, the sub-image 28 of the crosshairs of image 261 in layer L1. For example... Figure 2 As shown, based on the discovery that video (i.e., video sequence 240) is encoded into layer L0 using a non-subpicture segmentation method, extractor 10 forwards or takes over all bitstream portions 16 (dotted portions) associated with layer L0. Bitstream portions associated with layer L1 are encoded into video data stream 14 using subpicture segmentation. Figure 2 In the example, the bitstream portion of image 261 in layer L1 encoded into sub-image 28 is shown with cross-hatching, and the bitstream portion of image 261 encoded into other sub-images is shown with simple hatching. Figure 2 As shown, the bitstream portion belonging to images with equal presentation times can be a portion of a Common Access Unit (AU). Furthermore, note that each image 26 or sub-image 28 can be encoded into one or more bitstream portions 16. For example, images 26 or sub-images 28 can be further subdivided into slices, and each slice in said slice can be encoded into one or more bitstream portions. Figure 2 In the illustrative example, the predetermined set of sub-pictures 28 (i.e., the set of one or more sub-pictures to be forwarded in the sub-picture-specific video data stream 12 for decoding) contains only one sub-picture, namely, the cross-shaded sub-picture 28. Therefore, in the sub-picture-specific video data stream 12 of the bitstream portion associated with picture 261 of layer L1, only the cross-shaded bitstream portion of sub-picture 28 is forwarded.

[0051] In other words, device 10 may be sensitive to layers encoded in units of two or more independently encoded sub-images 28, or insensitive to when the bitstream portion 16 of a layer is taken over into sub-stream 12.

[0052] For example, a predetermined set of one or more sub-pictures can be provided to the extractor 10 via an external means such as API 20. That is, the extractor 10 can receive information about which sub-pictures of the layers 26 of the multi-layer video data stream 14 should be forwarded to the decoder 50.

[0053] For example, a subpicture identifier (e.g., SH_subpic_ID) can provide information revealing which bitstream segment a layer, where subpicture 28, among two or more subpictures, belongs. This information (i.e., the subpicture identifier) ​​can be provided, for example, in the header of the corresponding bitstream segment (e.g., a slice header). In other words, the bitstream segment 16 to which video data for a picture 26 or subpicture 28 is encoded may have an associated subpicture identifier in the descriptive data of the bitstream segment, which, for example, indicates the subpicture to which the bitstream segment belongs by means of an index.

[0054] In the example, video data stream 14 may include an association table 30, such as subpicIDVal, which is used to associate subpicture identifiers in the bitstream portion with the spatial location of the subpicture in the picture of video 24.

[0055] According to an embodiment, extractor 10 can perform a check to determine whether a layer is encoded into video using a non-subpics subdivision approach or a subpics subdivision approach by evaluating a syntax element that signals a specific layer in the video data stream 14. For example, the corresponding syntax element can be signaled in the sequence parameter set associated with the layer. For example, the syntax element can reveal the number of subpictures or sub-parts encoded into the corresponding layer. The syntax element can be included in picture or subpicture configuration data 22, which in this example can be signaled in the video data stream 14 (e.g., a sequence parameter set (SPS)). For example, the corresponding syntax element could be the `sps_num_subpics_minus1` syntax element.

[0056] like Figure 2As shown, video stream 14 may optionally include an OLS indicator 18, which indicates a set of layers, such as OLSs, for example, a set of layers to be at least partially forwarded in sub-picture-specific video stream 12. As previously described, OLS indicator 18 may, for example, include a set of OLSs that include the set of layers to be forwarded in sub-picture-specific video stream 12. For example, API 20 indicates to extractor 10, for example, which OLS in OLS indicator 18 should be forwarded in sub-picture-specific video stream 12 by indicating an index pointing to an OLS in OLS indicator 18.

[0057] Video data stream 14 may optionally further include decoder capability requirement information 60, which may include, for example, information about picture size, buffer size (such as the buffer size of CPB and / or DPB), and similar information, such as HRD, DPD, and TPL information. This information may be given in video data stream 14 in the form of a list of its various versions, where the version applies to a certain OLS referenced by an index. That is, for a certain retrievable OLS (e.g., the OLS indicated by OLS indicator 18), an index pointing to the corresponding HRD, DPD, and / or TPL information may be signaled.

[0058] The decoder capability requirement information 60 may be information about a sub-picture-specific video data stream 12 that can be extracted from the multi-layer video data stream 14. For example, the decoder capability requirement information 60, or a portion thereof, may be determined by the encoder 40 by actually performing an extraction process, such as the extractor 10 performing an extraction process to extract the perspective extractable sub-picture-specific video data stream 12, wherein the encoder 40 may perform the extraction process at least to some extent in order to determine certain parameters of the decoder capability requirement information 60.

[0059] Note that extractor 10 can modify (adapt) some data in data stream 12 when forming data stream 14. Figure 1 In this context, this possibility is indicated by using an apostrophe (apostrophe) for the reference symbol 16 of the bitstream portion of the video data stream 12. This also applies to... Figure 2 The components of flow 12 shown may also optionally be Figure 1This is a portion of the video data stream 12. These components may also exist in embodiments of video data streams 14 and 12 according to other aspects described herein. For example, picture configuration data 22 in stream 14 indicates the picture size of the complete picture 26 encoded into stream 14, while picture configuration data 22' in stream 12 indicates the picture size of picture 26' encoded into stream 14. With respect to layer L1, picture 26' (i.e., picture 261' from a sub-picture 28) is now encoded into stream 12 in a non-sub-picture subdivision; bitstream blocks 16 (such as VCL NAL units) to which the actual picture content is encoded can be taken over as is without any modification, at least regarding their arithmetic encoding portions. As can be seen, even OLS indication 18' can be taken over from stream 14 into stream 12. It can even be preserved as it is in stream 14; the association table 30 may also have been modified accordingly; modified versions of any data item in data items 18, 20 and 22 may be hidden / nested in stream 12, i.e., may have been provided therein by encoder 40, and thus may be simply used by the extractor to replace the corresponding rejected version as it appears in stream 12 when stream 14 is formed, or may be interpreted by extractor 10 in operation based on the overall information in stream 14 and put into stream 12; decoder 50 may no longer be in the position used to see how image 261 of layer L1 for sub-image subdivisions looked in stream 14 when receiving or being fed stream 12.

[0060] Note that, depending on the inspection, extractor 10 may further determine whether certain layer-specific parameter sets (such as PPS and SPS) are modified or regenerated from scratch accordingly, so that some of these parameters are adapted to the sub-picture-specific data stream 12. Such parameters include the picture size and sub-picture configuration, trimming window offset, and rank indicator in the picture or sub-picture configuration data 22 mentioned above. Thus, such modifications, such as modifications to the picture or sub-picture configuration data 22 in the data stream 12 regarding picture size and sub-picture configuration, do not occur for layers that are not sub-picture subdivisions, i.e., "if the corresponding layer encodes the video into it in a manner that is not sub-picture subdivisions"; however, such modifications do occur "if the corresponding layer encodes the video (24) into it in a manner that subdivides the picture (26) of the video into two or more sub-pictures (28)".

[0061] For example, when image 260 is forwarded in a sub-image-specific video data stream 12, these images can be used as a reference layer for sub-image 28 of layer L1. For example, as regarding Figure 1As described, the multi-layer video data stream 14 can take into account, for example, the scalability of the data stream's data rate. To this end, a base layer (e.g., layer L0) can encode video at a first resolution, and an enhancement layer (e.g., layer L1) can be a layer that encodes information about the video at a second resolution higher than the first resolution. For example, decoding an image 261 encoded into the enhancement layer might require information about images temporally juxtaposed from the base layer. In other words, the base layer can be a reference layer for the enhancement layer. Therefore, in this scenario, if both the base layer and the enhancement layer are part of a layer set, it can be ensured that a reference image for a sub-image of the enhancement layer (which is part of the base layer) is available in the sub-image-specific video data stream. Furthermore, in the example where a lower-resolution video is encoded into the base layer using a non-sub-image segmentation approach, and therefore, according to the presented concept, a portion of the base layer's bitstream is taken over into the sub-image-specific video data stream. Therefore, it can be ensured that at least a low-resolution video of the entire video content is available in the video data stream specific to the sub-image, so that if the sub-image to be rendered and thus decoded changes, the at least low-resolution image of the base layer is immediately available to the decoder, taking into account the rapid changes of the sub-image.

[0062] In other words, when using scalable encoding with subpicks, such as in viewport-dependent 360-degree video streaming scenarios, a common setup is to have a low-resolution base layer depicting the entire 360-degree video scene and an enhancement layer containing the scene at a higher fidelity or spatial resolution, but where the enhancement layer image is subdivided into several independently encoded subpicks. In such a setup, it is possible to extract a single subpick from the enhancement layer and the non-subpick base layer, such as a portion of the image corresponding to the client's viewing direction, and decode it together with the full 360-degree video base layer.

[0063] In bitstream configurations such as those described above (with subpictures having non-subpicture reference layers), existing extraction procedures, such as those in the VVC specification, will not produce a consistent bitstream. For example, after creating a temporary outBitstream from a verbatim copy of inBitstream (e.g., to derive OLS indicator 18' from OLS indicator 18), when extracting a subpicture with index subpicIdx, the following steps are performed for each layer i in the extracted OLS during the subpicture subbitstream extraction process in the current VVC draft specification:

[0064] - Remove all VCL NAL units from outBitstream where nuh_layer_id equals nuh_layer_id of layer i and sh_subpic_id is not equal to SubpicIdVal[subpicIdx]. That is, only take over the VCL NAL units where nuh_layer_id equals nuh_layer_id of layer i and sh_subpic_id equals SubpicIdVal[subpicIdx] into the extracted bitstream.

[0065] Note that all VCL NAL units with sub-image IDs different from the sub-images to be extracted are removed.

[0066] Here, SubpicIdVal is an AU-specific mapping from the order of the subpick index Idx in the bitstream and associated signaling structure where the subpick appears, and the identifier ID value is carried as the syntax element sh_subpic_id in the slice header of the slice belonging to the subpick, and is facilitated by the extractor, for example, to identify the subpick during extraction.

[0067] Figure 3 An example of a multi-layer video data stream 14 is shown, which can be as follows: Figure 1 and Figure 2 An example of the described video data stream 14. According to... Figure 3 The sequence of four access units of the video is shown. Each image in image 261 of layer L1 is subdivided into two sub-images, the first sub-image being indexed with index 0 and the second sub-image being indexed with index 1, while image 260 of layer L0 is encoded in a non-subdivided manner.

[0068] As is customary, it should be as follows Figure 3 When the original bitstream depicted at the top is extracted by a subpic (e.g., subpicIdx1), an inconsistent bitstream 12* will be created because L1 will lack its interlayer reference picture (ILRP) from L0, as in Figure 3 See the bottom. Note that it's possible the sub-image ID changes during the bitstream process, so extraction is index-based. Clearly, such a step would result in the removal of all NAL units (e.g., bitstream portions) from non-sub-image layers because they have sub-image index 0.

[0069] Therefore, according to the embodiments of the first aspect, it is possible to extract the sub-image-specific video data stream in scenarios where the sub-image-specific video data stream includes non-sub-image layers. In other words, the embodiments are capable of extracting the sub-image-specific video data stream, wherein the sub-image may have references to non-sub-image layers.

[0070] In other words, by contrast, embodiments according to the first aspect of the invention allow NAL units to be extracted (i.e., removed) from each layer in the OLS depending on its sub-image configuration. For example, in one embodiment, the above removal step is conditional on the presence of sub-images (e.g., by the number of sub-images in layer SPS), as described below:

[0071] - When sps_num_subpics_minus1 is greater than 0, remove from outBitstream all VCL NAL units where nuh_layer_id is equal to nuh_layer_id of the i-th layer and sh_subpic_id is not equal to SubpicIdVal[subpicIdx].

[0072] Pay special attention to the section "when sps_num_subpics_minus1 is greater than 0" to restrict the removal of VCL NAL units to layers or pictures with at least two subpics.

[0073] In another embodiment, based on the presence of sub-images, additional steps in the extraction process regarding non-VCLNAL units of each layer (such as SPS / PPS adjustments regarding image size, trimming window offset, grade IDC, and sub-image configuration) are omitted, as described below.

[0074]

[0075] Similarly, note the part "when sps_num_subpics_minus1 is greater than 0", which restricts layer-specific tasks to layers or pictures with at least two subpics.

[0076] Figure 4 An exemplary result is shown when the above embodiments are used in the extraction process to extract a sub-image-specific video data stream 12, which results in a consistent bitstream.

[0077] 2. Based on the encoder 40, extractor 10, and video data stream 14 in the second aspect.

[0078] This section describes embodiments of the encoder 40, extractor 10, and video data stream 14 according to a second aspect of the present invention. The encoder 40, extractor 10, and video data stream 14 according to the second aspect can be configured as follows... Figure 1 The encoder 40, extractor 10, and video data stream 14 are described, and may optionally be configured as described in Section 1 regarding... Figure 2 The first aspect described.

[0079] Figure 5An embodiment of encoder 40, extractor 10, decoder 50, and video data stream 14 is shown, wherein the video data stream 14 includes at least a first layer L1 and a second layer L0, for example, as per [reference to...]. Figure 1 and Figure 2 As described. In Figure 5 The sub-image 28, video sequence 24, and image 26 of the scene exemplarily shown in the example can be arranged according to... Figure 1 and Figure 2 The description of the scene shown.

[0080] According to an embodiment of the second aspect, the multi-layer video data stream 14 comprises or includes bitstream portions 16 (e.g., NAL units), each bitstream portion 16 belonging to a layer in the multi-layer video data stream 14, for example, as indicated by a layer ID associated with the corresponding bitstream portion. The multi-layer video data stream 14 includes a first layer (e.g., Figure 5 In layer L1 (on the left-hand side), its bitstream portion 16 encodes the first video 241 in such a way that it subdivides the image 261 of the first video 241 into two or more sub-images 28 (e.g., half of the image 261 of layer L1 with simple shading and half with cross shading). The sub-images 28 are encoded into the bitstream portion 16 of layer L1 in a mutually independent manner, such that for each image 261 of the first video 241, the corresponding sub-image 28 of the first video is encoded into a distinct bitstream portion 16 of layer L1. For example, as already discussed... Figure 2 As explained, the sub-image 28 of the simple shading line is encoded into the simple shading line bitstream portion of the video data stream 14, while the sub-image 28 of the cross-shading line is encoded into the cross-shading line bitstream portion of the video data stream 14. The multi-layer video data stream 14 further includes a second layer (e.g., Figure 5Layer L0 (on the left-hand side) has a bitstream portion into which the second video 240 is encoded. For example, the second video 240 involves video content that at least covers and is temporally aligned with the video content of the first video 241, making inter-layer prediction of the first video based on the second video feasible. For example, the video content of picture 260 in layer L0 may cover the video content of picture 261 temporally aligned with layer L1. In the example, but not necessarily, the second video 240 is encoded into the second layer L0 of the video data stream 14 using a non-subpicture subdivision approach with respect to spatial picture regions of the second video 240 that spatially correspond to the first video 241. In other words, the number of sub-parts encoded into the second layer L0 can be 1. Even further, each picture in the second layer L0 can be encoded as a subpicture, or, alternatively, if subdivided into sub-parts, it is possible that one sub-part completely covers the footprint of the first video. The first-layer bitstream portion (i.e., the bitstream portion containing the cross-shading and simple shading lines) encodes the first video 241 therein using vector-based predictions from the reference image. Furthermore, the first-layer bitstream portion encodes the first video 241 therein in such a way that it uses an image including (e.g., at least) the second video 240 as a reference image; and that it uses a vector 70 (used by the first-layer bitstream portion for encoding in the vector-based user prediction, and which is either composed of or signaled by the first-layer bitstream portion) to be scaled and offset relative to the first video image 261 and the reference image 260 according to the size and position of the scaling window 72 in the multi-layer video data stream 14.

[0081] For example, the images in the second video 240 are at least included in a reference image, which has been used to encode the bitstream portion of the first layer using vector-based prediction. For example, Figure 5 Image 260 can be a reference image for image 261. In other words, image 260 of the second video 240 can serve as an inter-layer reference image for encoding image 261 of the first video 241. For example, vector-based prediction can be included in a motion vector-based prediction framework, where the term "motion" generally refers to the fact that the image content in the first video 241 predicted from a reference image moves from its corresponding image content in the reference image, thereby being predicted spatially by a vector. That is, it is shifted from there, where the reference image can be, for example, a previously encoded image of the first video 241 or an image of the second video 240 that is simultaneously or temporally aligned. For example, in Figure 5 The image 26 depicted above each other in the illustrations of the first video 241 and the second video 240 can be such an image.

[0082] The scaling of vector 70 based on the size and position offset of scaling window 72 can be performed, for example, based on the positional offset between scaling windows (e.g., scaling window 720 of layer L0 and scaling window 721 of layer L1). Furthermore, for example, the scaling of vector 70 based on the size and position of scaling window 72 can be performed based on the size ratio between scaling windows (e.g., scaling windows 720 and 721).

[0083] For example, the size and position of the scaling window 72 can be signaled in the multi-layer video data stream 14 on a per-image basis, using a scaling window indicator 72. For example, the size and position of the scaling window 72 can be signaled by: the size of the scaling window in x and y, and an indication of the position of one corner, or an indication of the positions of two opposite corners of the scaling window. Note that it may also be feasible to signal certain positioning of the scaling window, rather than using the coordinates or coordinates and size of the scaling window, such as by using a corresponding flag to indicate that the scaling window coincides with the image boundary. In the latter case, in the example, the positioning of the scaling window is indicated by coordinates, etc., only when the flag indicates that the positioning of the scaling window does not coincide with the image boundary.

[0084] According to the second aspect, the extractor 10 is configured to take over the bitstream portion belonging to the second layer L0 from the multi-layer video data stream 14 to the sub-picture-specific video data stream 12, such that the sub-picture-specific video data stream 12 fully encodes the video 240 of the corresponding layer. For example, the extractor 10 may forward all bitstream portions 16 of the bitstream portion 16 of layer L0, or may forward bitstream portions associated with layer L0, independent of which sub-pictures of two or more sub-pictures of the first layer L1 are forwarded to the sub-picture-specific video data stream 12, i.e., independent of which sub-pictures of the first layer L1 belong to a predetermined set of one or more sub-pictures as described below. In other words, according to the second aspect, the device 10 may take over the bitstream portion of the second layer L0, for example, as per the description of the second aspect. Figure 1 and Figure 2 This describes the case of layer L0, which is encoded using a non-sub-image segmentation method.

[0085] According to the second aspect, the extractor 10 is further configured to take over from the multi-layer video data stream 14 to the sub-picture-specific video data stream 12 each bitstream portion that belongs to the first layer L1 and encodes one or more subpictures belonging to a predetermined set of subpictures 28. For example, the extractor 10 may discard or leave (i.e., not forward) subpictures 28 belonging to the corresponding layer (e.g., layer L1) but not belonging to the predetermined set of subpictures 28. In other words, according to the second aspect, the extractor 10 can selectively forward bitstream portions of the first layer L1 that belong to the predetermined set of subpictures, as per the second aspect. Figure 1 and Figure 2 As described in the first aspect. (Regarding...) Figure 1 and Figure 2 As described, a predetermined set of one or more sub-images can be indicated by information provided to extractor 10 by the API.

[0086] Extractor 10 is further configured to modify the scaling window signaling 74 for the first layer L1 and / or the second layer L0 in the sub-picture-specific video data stream 12, such that the spatial region of the scaling window 720' for the picture of the second video spatially corresponds to the spatial region of the scaling window 721' for a predetermined set of one or more sub-pictures.

[0087] like Figure 5 As shown, after extracting the video data stream 12 specific to a sub-image, the sub-image defined by a predetermined set of one or more sub-images is due to the sub-images of the predetermined set of one or more sub-images (e.g., Figure 5 The sub-image 28) is extracted and customized to become or constitute the image 261' of the first layer L1 of the data stream 12. The extractor 10 can modify the scaling window signaling so that the scaling of the vector 70 of the second layer L0 of the sub-image-specific video data stream 12 after scaling using scaling windows 720', 721' is associated with, or mapped to, a vector 70 having size and position in the juxtaposed image 261' of the first layer L1 of the sub-image-specific video data stream 12, which corresponds to the size and position of the vector 70 associated with or mapped to the image of the second layer L0 using scaling windows 720, 721 in the multi-layer video data stream 14. In other words, extractor 10 can modify scaling window signaling 74 such that scaling windows 720', 721' provide a determination of a vector 70 at a given position within image 261' of the first layer L1 of data stream 12. This determination results in a vector whose size and position correspond to the vector 70 at the corresponding position in image 261 of layer L1 of multi-layer video data stream 14 (the corresponding position corresponds to the position within image 261' relative to the image boundary, not in terms of content). However, since the size of image 261' may have changed relative to image 261, when defining the scaling window relative to the image boundary, it may be necessary to modify or move the scaling window regarding its position relative to the boundary of image 261'.

[0088] According to an embodiment, the extractor 10 can modify the scaling window signaling 74 for the first layer L1 such that the scaling window 721' of the first layer L1 corresponds in position and size to the scaling window 720' of the second layer. This example is in... Figure 5 The illustration is provided using zoomed windows 720' and 721' drawn with continuous lines.

[0089] Alternatively, the extractor 10 can modify the scaling window signaling 74 by defining a scaling window 720' of the second layer L0 to coincide with the periphery of a predetermined set of one or more sub-images, as described in... Figure 5 The text is a jumbled mess of characters and doesn't form coherent sentences. It appears to be a collection of fragments from various sources, possibly related to a map or a forum. Figure 6 The scaling windows 720' and 721' shown in the diagram indicate this.

[0090] According to another embodiment, the extractor 10 can modify the scaling window signaling for both the first layer and the second layer, that is, modify the scaling windows 720' and 721' so that the outer peripheries of the scaling windows 720' and 721' coincide.

[0091] For example, extractor 10 can modify the scaling window signaling 74 by using sub-image configuration data 22 in data stream 14. In other words, extractor 10 can modify the sub-image configuration data 22 in data stream 14 to provide sub-image configuration data 22' in data stream 12. For example, sub-image configuration data 22 may include an indication of the scaling window position and size. Extractor 10 can modify the scaling window position and size in configuration data 22 to obtain sub-image configuration data 22'.

[0092] Note, as about Figure 2 The provided annotations regarding the hidden inclusion, modification, or regeneration of the bitstream portion by extractor 10 also apply. Figure 5 (Refer to the apostrophe for the bitstream portion of data stream 12), where information 74 is another example of information that can be hidden by the encoder or modified or regenerated by the extractor during operation.

[0093] Figure 6 The example above illustrates how the scaling window 720' of the second layer L0 is modified by extractor 10. In other words, Figure 6 An example of adjusting the scaling window is shown when extracting the right sub-images of layers L1 and L0.

[0094] The following is an exemplary specification for an implementation used to modify the scaling window.

[0095]

[0096]

[0097] According to Figure 5 and Figure 6The embodiments of the second aspect described take into account the combination of extraction of sub-picture-specific video data streams (such as data stream 12) and inter-layer prediction, because the vector 17 used in the vector-based prediction can be correctly mapped in the decoded sub-picture-specific video data stream 12.

[0098] In other words, while the raw bitstream contains control information that guides inter-layer prediction to apply the correct upscale filter and MV offset in the corresponding PPS via a so-called scaling window, it is crucial to adjust these high-level parameters in the sub-bitstream resulting from extracting an EL sub-picture plus the complete BL to allow for correct decoding. Therefore, as part of this chapter, the scaling window offset parameters are adjusted to compensate for the dropped sub-pictures in the enhancement layer in terms of MV offset and scaling factor derivation. Figure 6 The example shown illustrates a layer with two sub-images in the enhancement layer and how the scaling windows of the base layer L0 and the enhancement layer L1 are adjusted to result in correct inter-layer predictions on the decoder side, just as initially performed on the encoder side. Specifically, the scaling window in the layer (L1) containing the sub-images is rewritten to cover only the fully encoded image region corresponding to the extracted sub-images after extraction, as shown in... Figure 6 As shown on the right-hand side, the scaling window of the layer (L0) without sub-images to be rewritten is to cover only the image region corresponding to the extracted sub-images of the higher layer, that is, to cover the image region or samples needed to predict the remaining L1 encoding. Similarly, as in the example described above, the scaling window 721' of the first layer L1 is modified to extract one or more sub-images defined by a predetermined set of one or more sub-images from the first layer L1 by the extractor 10.

[0099] 3. Based on the encoder 40 and decoder 50 of the third aspect.

[0100] This section references Figure 1 The embodiments according to the third aspect are described, wherein the details described in Section 0 may optionally be applied to the embodiments according to the third aspect. Furthermore, as... Figure 2 and Figure 5 The details of the data streams 12 and 14 shown may also be included in the embodiments of the third aspect, in which case the corresponding descriptions of Sections 1 and 2 apply accordingly.

[0101] As previously described, encoder 40 may apply interlayer prediction in the encoding of image 26 of video 24. For example, encoder 14 may use interlayer prediction to encode image 261 of layer L1. For example, encoder 40 may use image 260, which is temporally juxtaposed with image 261, to perform interlayer prediction in the encoding of image 261. For example, encoder 40 may use temporal motion vector prediction (TMVP) to predict motion vectors for encoding image 261. Other examples of methods that can be used for interlayer prediction are prediction refinement using optical flow (PROF) or motion vector wrapping, which will be described more precisely below. As previously described, the image used to predict the image to be encoded may be referred to as the reference image of the image to be encoded.

[0102] In cases where TMVP would require resampling of the reference image motion vector (MV) storage, such as for single-layer resolution variations or multi-layer spatial scalability (as it may be provided by multi-layer video sequences24), the use of TMVP is disabled using the gate variable RprConstraintActive[currentPic][refPic], as described below.

[0103] The requirement for bitstream consistency is that the image referenced by sh_collocated_ref_idx should be identical for all slices of the encoded image, and RprConstraintActive[sh_collocated_from_10_flag? 0:1][sh_collocated_ref_idx] should be equal to 0.

[0104] Note - The above constraints require that the juxtaposed images have the same spatial resolution and the same scaling window offset as the current image.

[0105] Figure 7 An example is shown of extracting sub-picture-specific video data streams from multi-layer bitstreams, where picture 261 of the first layer L1 is subdivided into sub-pictures 28, while the juxtaposed picture 260 of the second layer L0 is not subdivided, i.e., it is encoded into a single sub-picture. As previously stated, the embodiments in this section are about Figure 1 Described, thus for Figure 1 The descriptions of the elements and corresponding reference symbols are applicable. Furthermore, regarding... Figure 2 The description of the corresponding elements can also be applied to embodiments according to the third aspect.

[0106] When, as explained above (e.g., Sections 1 and 2), layers with sub-images are combined with layers without sub-images in OLS, and as in Figure 7 The left-hand side above, as shown by the original bitstream, will also allow the encoder to use TMVP when no spatial scalability is used.

[0107] However, when as in Figure 7 As shown on the right-hand side, problems arise when extracting sub-images and adjusting the scaling window during the extraction process to allow inter-layer prediction of sample values ​​to function correctly. After extraction, the conditions used to derive RprConstraintActive[currentPic][refPic] change, and TMVP is not allowed, as can be seen in the following sections of the current VVC draft specification, particularly the section defining "RprConstraintActive[i][j]":

[0108]

[0109]

[0110] The section defining RprConstraintActive[i][j] defines the condition corresponding to the comment described earlier in this section, which states that "the above constraints (regarding the reference image) require that the juxtaposed images have the same spatial resolution and the same scaling window offset as the current image." Any syntax-based predictions such as TMVP are prohibited when the current and reference images have different sizes, for example, when there are variations in single-layer resolution in a CVS (coded video sequence) or in inter-layer prediction in a spatially scalable bitstream, because such predictions would require special precautions in the prediction mechanism involved. For example, for syntax prediction purposes, syntax such as MV candidates is stored in an additional regular storage grid (e.g., 16×16 samples), and the (CTU) block boundaries of the current image are aligned with the boundaries of this regular storage without a change in resolution. Therefore, finding the juxtaposed MV storage location is relatively straightforward, but the boundaries may no longer align when the sampling ratio is not equal to 1 (i.e., a change in resolution between the reference and current images) or when arbitrary positional offsets are used (through different scaling window positions between the reference and current images). Therefore, deriving the correct MV storage location for prediction would be an additional implementation burden, and in the case of VVC, design trade-offs would be needed to avoid integrating special precautions for this situation, since it is assumed that it rarely occurs in the main use case of single-layer resolution variation, and multi-layer applications are not the primary target of the codec.

[0111] Also note that currently, motion vector prediction does not use a scaling window. A scaling window is only used for sample prediction based on motion vectors used for blocks; that is, the scaling window defines how the motion vectors should be offset. However, for motion vector prediction, ignoring the scaling window and mapping the top-left block of one image to the top-left block of another image (juxtaposed block), in the example shown in the figure, this would result in a completely different juxtaposed block when performing extraction, compared to before extraction.

[0112] Another tool similarly affected to TMVP is Prediction Refinement Utilizing Optical Flow (PROF), which is part of the affine motion prediction model in VVC and was introduced to more effectively represent certain types of motion (e.g., zooming in or out, rotation), and applies CU-based affine motion compensation predictions if this mode is selected. The affine motion model for CU is described by either a four-parameter model with two control points at the top left and top right corners or a six-parameter model with three control points at the top left, top right, and bottom left corners. To achieve finer-grained motion compensation, Prediction Refinement Utilizing Optical Flow (PROF) is used to refine each luminance prediction sub-block to target the effect of sample-by-sample motion compensation. Predicted samples in the luminance sub-blocks are refined by adding gradient-derived differences and sample-based motion vector differences. PROF does not work for chrominance samples.

[0113] However, during affine motion prediction, deriving the PROF-related flag cbProfFlagLX based on RprConstraintActive[i][j] and extracting the difference in the derivation before and after the sub-image will lead to encoder / decoder mismatch.

[0114] Another affected tool is motion vector wrapping, which is used for certain types of content (360-degree video in isometric cylindrical projection formats). For example, when an object leaves the encoded image plane at the vertical boundary of an image, it typically re-enters the image plane at the opposite vertical boundary, much like the nature of projection from the underlying layer. This fact is facilitated in motion vector wrapping by using sample padding with the opposite boundary when the motion vector points outside the image.

[0115] However, during sample prediction, a flag (refWraparoundEnabledFlag) for controlling motion vector wrapping is derived based on RprConstraintActive[i][j]. Therefore, when there are differences in the derivation before and after extracting sub-images, it will lead to encoder / decoder mismatch, similar to the case of TMVP or PROF.

[0116] The third aspect provides a concept for controlling the use of inter-layer prediction tools such as TMVP, PROF, and MV surround to avoid the aforementioned problems that may occur when extracting sub-picture-specific video data streams. Embodiments of the third aspect can avoid these problems by controlling the use of these inter-layer prediction tools during the encoding of the original bitstream (i.e., multi-layer video data stream 14) to avoid different behaviors before and after sub-picture extraction.

[0117] According to an embodiment of the third aspect, the encoder 40 for encoding video 24 into a multi-layer video data stream 14 supports sub-picture independent encoding. Furthermore, the encoder 40 is configured to encode a first version 241 of video 24 into the first layer L1 of the multi-layer video data stream 14 using a set of inter-layer prediction tools for predicting the second layer L0 of the multi-layer video data stream 14 and a first configuration setting for sub-picture independent encoding. The encoder 40 according to the third aspect is further configured to encode a second version 240 of video 24 into the second layer L0 of the multi-layer video data stream 14 using a second configuration setting for sub-picture independent encoding. The encoder 40 is configured to check whether the first configuration setting and the second configuration setting have a predetermined relationship. If the first configuration setting and the second configuration setting do not have a predetermined relationship, the encoder 40 disables or avoids using a predetermined subset of one or more inter-layer prediction tools.

[0118] For example, the first and second configuration settings for the first layer L1 and the second layer L0 may consider whether (or not) the corresponding layers are encoded using a sub-image subdivision approach. In other words, the first and second configuration settings may include information about the number of sub-images (e.g., distinguishing between one and more than one, where one sub-image implies encoding the image in a non-subdivision manner). For example, the first and second configuration settings for the first layer L1 and the second layer L0 may include information about one or more of the following: the boundaries of the sub-images, sub-image identifiers, and sub-image boundary processing.

[0119] For example, a pre-defined relationship can be established such that a first configuration setting has a pre-defined relationship with a second configuration setting if one or more of the following conditions are met:

[0120] The first and second videos are encoded into the first and second layers respectively by subdividing them into more than one sub-image.

[0121] - The first video and the second video are encoded into the first layer and the second layer respectively by subdividing them into multiple sub-images that are equal in size within the first video and the second video.

[0122] The first video and the second video are encoded into the first and second layers respectively by subdividing them into multiple equal sub-images within the first and second videos, where the boundaries of the sub-images are spatially consistent.

[0123] - The first video and the second video are encoded into the first layer and the second layer respectively by subdividing them into multiple sub-images. The multiple sub-images are consistent in the sub-image IDs that signal the sub-images in the multi-layer video data stream.

[0124] - The first video and the second video are encoded into the first layer and the second layer respectively by subdividing them into multiple sub-images, and the multiple sub-images are treated in the same way in terms of boundary processing.

[0125] For example, a predetermined subset of one or more inter-layer predictions includes one or more of the following: inter-layer motion vector prediction (e.g., TMVP), optical flow refinement based on vector inter-layer prediction, and motion vector wrapping in vector inter-layer prediction.

[0126] For example, encoder 40 can signal the first configuration setting and the second configuration setting in a multi-layer video data stream.

[0127] In the example, if the first configuration setting has a predetermined relationship with the second configuration setting, the encoder 40 may activate or utilize a predetermined subset or subset of one or more interlayer prediction tools.

[0128] According to an embodiment of the third aspect, a decoder 50 for decoding a multi-layer video data stream is configured to use a set of inter-layer prediction tools for prediction from the second layer L0 of the multi-layer video data stream and to decode a first version 241 of video 24 from the first layer L1 of the multi-layer video data stream using a first configuration setting for independent encoding for each sub-picture. Note that the multi-layer video data stream can be a sub-picture-specific video data stream 12 or a multi-layer video data stream 14. In other words, although in Figure 1 In this context, decoder 50 is shown as decoding a sub-picture-specific video data stream 12, but decoder 50 can also be able to decode multi-layer video data streams 14, as it can be forwarded by extractor 10, for example, when selecting another OLS, as in Figure 1 As shown in the scenario. Since the multi-layer video data stream decoded by decoder 50 can correspond to the sub-picture-specific video data stream 12, the first version 241 of video 24 may differ from the version mentioned with respect to encoder 40, because the multi-layer video data stream may have undergone an extraction process, as shown regarding... Figure 1-7As described. Therefore, in the example, the first version 241 of the video referred to with respect to decoder 50 may differ from the version referred to with respect to encoder 40 by omitting one or more sub-pictures. The same applies to the first configuration settings, which refer to decoder 50 and involve picture regions encoded into the extracted / remaining / normally removed portions of the data stream to be decoded by decoder 50. For example, if the multi-layer video data stream entering the decoder only involves one of the first two sub-pictures containing the picture 28, the first configuration data will only indicate non-sub-picture segmentation.

[0129] According to the third aspect, decoder 50 can be configured to decode a second version 240' of video 24' from the second layer L0 of a multi-layer video data stream (i.e., the video data stream provided to decoder 50) using a second configuration setting that encodes each sub-picture independently. Decoder 50 is configured to check whether the first configuration setting and the second configuration setting have a predetermined relationship. If the first configuration setting and the second configuration setting have a predetermined relationship, decoder 50 can disable a predetermined subset of one or more inter-layer prediction tools, as described with respect to encoder 40.

[0130] According to an embodiment, decoder 50 can be configured to decode each sub-image independently, including vector trimming at the sub-image boundaries and / or boundary padding at the sub-image boundaries.

[0131] For example, the variable RprConsconstraintsActive described above can represent a comparison between the first configuration setting and the second configuration setting. For example, encoder 40 and decoder 50 can derive this variable depending on whether the first configuration setting and the second configuration setting have a predetermined relationship, that is, set the variable accordingly, and use the variable to determine whether to use a predetermined subset of the inter-layer prediction tool.

[0132] In one embodiment, the derivation of RprConstraintActive[i][j] is adjusted as follows:

[0133]

[0134]

[0135] Alternatively, in one embodiment, the requirement for bitstream consistency is that, in predictions between the current image and a reference image with different sub-image configurations (enabled and (vs.) disabled), the bitstream does not activate TMVP, PROF, or MvWrapAround, and the bitstream carries an indication of the constraint.

[0136] While the meaning of the above derivation does not affect the decoder in the example for TMVP, for PROF and MvWrapAround, the decoder can execute the above modified derivation and behave differently when executing / checking the corresponding tools.

[0137] Alternatively, such checks may be performed depending on the following: the number or identifiers of sub-images in the current and reference images, the properties of boundary treatment for such sub-images, or other characteristics.

[0138] 4. Based on the encoder 40 and extractor 10 in the fourth aspect.

[0139] This section references Figure 1 The embodiments according to the fourth aspect are described, wherein the details described in Section 0 may optionally be applied to the embodiments according to the fourth aspect. Furthermore, as... Figure 2 and Figure 5 The details of the data streams 12 and 14 shown may also optionally be included in the embodiments of the third aspect, in which case the corresponding descriptions of Sections 1 and 2 apply accordingly. Preferably, the features described with respect to the first, second, and third aspects may also be combined with the fourth aspect.

[0140] When extracting sub-pictures from a multi-layer video data stream (such as video data stream 14), as in the examples of the embodiments in Sections 1 through 3 above, a signal or derived score (e.g., indicated by ref_level_fraction_minus1) may be sent. This score is later used to determine certain level constraints (e.g., bitrate, CPB size, number of tiles, etc.) of the extracted sub-bitstream. For example, these level constraints may impose decoder capability requirements, or be related to decoder capability requirements, which the decoder must meet in order to be able to decode the video sequence or video data stream described by the OLS as referred to by the capability requirements.

[0141] By convention, when it does not exist, the score is inferred to be equal to the size of the sub-image relative to the size of the entire image in its layer, as described below:

[0142] When it does not exist, the value of ref_level_fraction_minus1[i][j] is inferred to be equal to Ceil(256*SubpicSizeY[j]÷PicSizeInSamplesY*MaxLumaPs(general_level_idc)÷MaxLumaPs(ref_level_idc[i])-1.

[0143] However, several problems exist with this mechanism. First, the above inference does not take into account the fact that a bitstream can manage multiple layers. For example, the ratio SubpicSizeY[j]÷PicSizeInSamplesY is insufficient because some class constraints have bitstream ranges (e.g., bit rate and CPB size), while other class constraints have layer ranges (e.g., number of tiles).

[0144] Therefore, as part of this aspect, in one embodiment, it is inferred in a way that takes into account all layers in the OLS, i.e., it is specific to the derived bitstream.

[0145] Therefore, as part of this aspect, in one embodiment, the inference of the score for each sub-image is modified to also incorporate the rank influence of the reference layer without sub-images, as described below.

[0146] When it does not exist, the value of ref_level_fraction_minus1[i][j] is inferred to be equal to max(255, Ceil(256*SumOfSubpicSizeY[j]÷SumOfPicSizeInSamplesY*MaxLumaPs(general_level_idc)÷MaxLumaPs(ref_level_idc[i])-1)). The maximum value function can be excluded, thus resulting in Ceil(256*SumOfSubpicSizeY[j]÷SumOfPicSizeInSamplesY*MaxLumaPs(general_level_idc)÷MaxLumaPs(ref_level_idc[i])-1).

[0147] Where SumOfSubpicSizeY and SumOfPicSizeInSamplesY are the sums of all samples of the corresponding subpictures in all layers of the OLS.

[0148] Figure 8 It shows, for example, in Figure 1 , Figure 2 , Figure 5 The example shown is an extraction of a sub-image-specific video data stream 14 from a scene. Figure 8 It indicates the relative change in output image area between the original data stream (e.g., video 24 of data stream 14 before extraction) and the video sequence 24' of sub-image-specific data stream 12 after extraction.

[0149] The second issue takes into account the fact that sub-images can be combined with reference layers that do not contain corresponding sub-images, such as... Figure 8 As shown, L0 does not have sub-images.

[0150] To cover this situation or provide a more accurate level limit, in another embodiment, an inference of, for example, ref_level_fraction_minus1 is made, as described below:

[0151] Where SumOfSubpicSizeY and SumOfPicSizeInSamplesY are the sums of all samples in all layers of the OLS corresponding to the subpicks and all layers of the OLS that do not contain subpicks.

[0152] In another embodiment, the inference of ref_level_fraction_minus1[i][j] remains unchanged relative to the prior art level (it preserves the layer-specific score for layers containing sub-images). Instead, alternatively for the above changes, an OLS-specific score variable is derived from the layer-specific score, as described below:

[0153]

[0154]

[0155] Where k is the OLS index, i is the reference level index, and j is the sub-image index.

[0156] The above embodiments are expected to have an equal rate distribution across all layers. To allow the encoder to also freely determine the rate distribution among sub-picture layers and non-sub-picture layers, in another embodiment of the invention, the fraction of layers without sub-pictures within the OLS of the bitstream is explicitly signaled. That is, for a given reference level, the total OLS fraction of the bitstream corresponding to the non-sub-picture layers is signaled.

[0157]

[0158] non_subpic_layers_fraction[i] specifies the fraction of the Bitstream / OLS-specific rank limit associated with the layer in the Bitstream / OLS that makes sps_num_subpics_minus1 equal to 0. non_subpic_layers_fraction[i] is equal to 0 when vps_max_layers_minus1 is equal to 0; or non_subpic_layers_fraction[i] should be equal to 0 when there are no layers in the Bitstream / OLS that make sps_num_subpics_minus1 equal to 0.

[0159] Set the variable OlsRefLevelFraction[i][j] of the j-th sub-image of the i-th ref_level_idc to non_subpic_layers_fraction[i]+(255-non_subpic_layers_fraction[i])÷255*ref_level_fraction_minus1[i][j]+1.

[0160] OlsRefLevelFraction[i][j] is used specifically for OLS, as discussed in Section 5.

[0161] According to an embodiment of the fourth aspect, the multi-layer video data stream 14 includes a set of layers, such as an OLS (Optical Layer System) like the selection of an OLS for decoding. For example, the OLS may include layers L0 and L1, such as... Figure 1 As shown in the diagram. Note that the multi-layer video data stream encoded by encoder 40 may include not only this layer set (i.e., this one layer set), but may include additional OLS, and may include, in addition to, the following: Figure 1 The layers shown are additional layers besides layers L0 and L1. The encoder 40, according to the fourth aspect, is configured to encode unreduced versions of the layer set (e.g., layers L0 and L1) into a multi-layer video data stream 14, wherein video versions (e.g., video sequences 240 and 241) are encoded into the unreduced version of the layer set in units of one or more independently encoded sub-pictures per layer. For example, the phrase "in units of one independently encoded sub-picture" should indicate layers encoded without subdivision into sub-pictures, i.e., pictures (such as picture 260 of video version 240) form a sub-picture. For example, one or more such unsubdivided layers may exist in the layer set, and one or more sub-picture-divided layers may exist in the layer set.

[0162] The encoder 40, according to the fourth aspect, is further configured to encode reference decoder capability requirements associated with the unreduced version of the decoded layer set into the layer video data stream 14. Furthermore, the encoder 14, according to the fourth aspect, is configured to, for each layer of the layer set, encode information about the image size of the corresponding image in the unreduced version encoded into the layer size (such as in...) Figure 2The information 22 described in the figure is encoded into the multi-layer video data stream 14, and for each sub-picture in one or more independently encoded sub-pictures of the corresponding layer, information about the sub-picture size is encoded into the multi-layer video data stream 14. The encoder 40 of the fourth aspect is configured to determine the decoder capability requirements associated with the corresponding reduced version of the decoding layer set for each of one or more reduced versions of the layer set for which the sub-picture-related portion of at least one layer has been removed (e.g., sub-picture-specific video data stream 12 or another version of video data stream 12 that does not necessarily include the entire multi-layer video data stream 14), such as OlsRefLevelFraction[i][j] (note that OlsRefLevelFraction[i][j] is specific to the reduced version associated with the j-th sub-picture).

[0163] Encoder 40 determines the decoder capability requirements for the corresponding reduced version by scaling the reference decoder capability requirements by a factor determined using the quotient of the sum of the sub-picture sizes of the independently encoded sub-pictures contained in the corresponding reduced version of the layer set divided by the sum of the picture sizes. In other words, a reduced version of the layer set refers to a version of the layer set for which the bitstream portions relating to other sub-pictures (such as sub-pictures that are not extracted, i.e., those that are not part of a predetermined set of sub-layers to be forwarded in the sub-picture-specific video data stream 14) have been removed.

[0164] For example, encoder 40 can provide decoder capability requirements in multi-layer video data stream 14, as shown in, for example... Figure 2 The decoder capability requirement shown is 60.

[0165] Decoder capability requirement information can be used as an indication of whether the extracted bitstream can be decoded by the decoder receiving the bitstream. For example, decoder capability requirement information can be used by the decoder receiving the bitstream to set up or initialize the decoder.

[0166] By determining specific decoder capability requirements for each reused version, decoder capability requirements can be provided more precisely, so that information indicating unnecessarily high decoder capability requirements can be avoided, and, for example, the decoder can select the highest quality OLS based on its capabilities.

[0167] Accordingly, according to an embodiment of the extractor 10 of the fourth aspect, the extractor 10 for extracting a sub-picture-specific video data stream 12 from a multi-layer video data stream 14 comprising a layer set is configured to derive reference decoder capability requirements from the multi-layer video data stream 14 related to the unreduced version of the decoding layer set, into which the unreduced version of the layer set is encoded, and to encode video versions 240, 241 into the unreduced version of the layer set in units of one or more independently encoded sub-pictures per layer. According to this embodiment, the extractor 10 is configured to, for each layer of the layer set, derive information from the multi-layer video data stream regarding the image size in the corresponding layer of the unreduced version of the layer set (such as in...). Figure 2 The information shown in 22) is used to derive information about the size of the sub-picture from the multi-layer video data stream for each sub-picture in one or more independently encoded sub-pictures of the corresponding layer. Furthermore, according to this embodiment, the extractor 10 is configured to, for a predetermined reduced version of the layer set for which at least one layer's sub-picture-related portion has been removed (e.g., predetermined by means of OLS indication 20 provided by the API), determine the decoder capability requirements associated with the predetermined reduced version of the decoding layer set by scaling the reference decoder capability requirements by a factor determined using the quotient of the sum of the picture sizes of the independently encoded sub-pictures contained in the predetermined reduced version of the layer set divided by the sum of the picture sizes.

[0168] In the following text, please refer to Figure 1 and Figure 2 The alternative embodiments of the fourth aspect will be described.

[0169] According to these alternative embodiments, the multi-layer video data stream 14 includes multiple layers, such as layer L0 and layer L1. Furthermore, the encoder 40 is configured to handle one or more first layers (such as...) Figure 1 The encoder 14 encodes video images 26 (such as images 260 and 261) into layers of a multi-layer video data stream by subdividing them into independently coded sub-images (layer L1) and by using a non-subdivided approach with respect to one or more second layers (such as layer L0). In other words, the encoder 14 can encode images of video 24 into a first type of layer (such as first layer L1) by subdividing them into independently coded sub-images, and can encode images into one or more second type of layers (such as second layer L0) by using a non-subdivided approach. Note that the multi-layer video data stream 14 may include one or more first type layers and one or more second type layers. The encoder 40 according to these alternative embodiments is configured to encode the layer set of layers (such as in...) Figure 2The indication 18 shown is encoded into the multi-layer video data stream 14 such that at least one first layer is included in the layer set, and at least one second layer is included in the layer set. For example, the indication of the layer set may be an OLS indication 18. The multi-layer video data stream 14 may include indications of multiple layer sets, as shown regarding Figure 1 As described.

[0170] The encoder 40 according to these alternative embodiments is configured to, for each of a plurality of reference decoder capability requirements sufficient for a set of layers to decode layers, assign a first score (e.g., ...) to the corresponding reference decoder capability requirement attributed to the at least one first layer. Figure 9 The first score 134) and the second score 132 (e.g., attributable to the corresponding reference decoder capability requirements of the at least one second layer) Figure 9 The information of the second fraction 134 is encoded into the multilayer video data stream 14. For example, the first fraction 134 can be indicated by the syntax element ref_level_fraction_minus1[i][j] introduced above, and the second fraction 132 can be indicated by the syntax element non_subpic_layer_fraction[i][j] introduced above.

[0171] For example, a layer set whose reference decoder capability requirement is sufficient for decoding layers may mean that each reference decoder capability requirement is defined at least by a minimum CPB size. These reference decoder capability requirements may differ from one another with respect to their indicated minimum CPB size. Note that for some of these reference decoder capability requirements, the layer set may collectively consume only a small fraction, for example, only a small fraction of the fully available CPB size, and therefore, like regarding... Figure 10 Mixers such as Mixer 100 can use this information (i.e., reference decoder capability requirements) to determine how many OLSs of different streams can be mixed together, while still satisfying the grade / reference decoder capability requirements.

[0172] For example, decoder capability requirements (also known as decoder capability information (DCI)) can indicate the requirements for the decoder to decode the set of layers associated with the decoder capability requirements. For example, a set of layers can have several reference decoder capability requirements associated with it. For example, each reference decoder capability requirement in the reference decoder capability requirements associated with the set of layers can refer to different versions of video 24 encoded into that set of layers, such as different versions of video including different sub-pictures encoded into different sub-picture-specific video data streams.

[0173] The fourth aspect of the embodiments includes a device for processing multi-layer video data stream 14, which can process video data streams provided by alternative embodiments of the encoder 40 just described. For example, the device for processing multi-layer video data stream 14 is a mixer (such as mixer 100 described below), an extractor (such as...) Figure 1 , Figure 2 , Figure 5 The device is referred to as device 10, 100, or a combination thereof, including an extractor 10, a decoder (such as decoder 50), or a combination thereof. Therefore, the device for processing the data stream is referred to as device 10, 100. For example, decoder 50 and extractor 10 may be combined, or one of them may include the other. The device for processing a multi-layer video data stream according to these embodiments is configured to indicate a layer set for decoding layers from the multi-layer video data stream 14, the layer set including at least one first layer and one second layer, as previously described. The device is configured to, for each of a plurality of reference decoder capability requirements, decode information from the multi-layer video data stream 14 regarding a first score attributable to a corresponding reference decoder capability requirement of the at least one first layer and a second score attributable to a corresponding reference decoder capability requirement of the at least one second layer.

[0174] Figure 9 Examples of first-layer image 261 and second-layer image 260, as well as examples of the first and second scores 134 and 132, are shown respectively, because according to the alternative embodiment of the fourth aspect described above, they can be signaled in the video data stream 14. Figure 9 In the example, the first layer image 261 is subdivided into two sub-images 28. Figure 9 a shows the use of in Figure 9 The determination of the first score 1341 and the second score 1321 of the first reference decoder capability requirements associated with the first reference level indicated in the document. Figure 9 In the illustrative example, the first reference level is compared with the first CPB 1301 having an exemplary value of 1000 (in Figure 9 In this context, it can be associated with an illustrative example representing the decoder's capability requirements. Figure 9 In this context, for each sub-image or image, it indicates its relative share in the total image size of the video stream associated with the corresponding reference decoder capability requirements. For example, according to... Figure 9 a. Image 260 accounts for 20% of the total image size in the video stream. In other words, Figure 9 a shows the determination of reference decoder capability requirements for a video data stream including images 260 and 261 (both with sub-images contained therein). Image 260, as a second layer, is used to determine a second fraction 1321 relative to the share of the total CPB indicated for the corresponding decoder capability requirements. (The last sentence appears to be incomplete and possibly contains errors.) Figure 9 The decoder capability requirement of the video data stream shown in a requires that the associated first fraction 1341 be the remainder of the CPB, that is, the share distributed on the sub-pictures of the first layer picture 261 that is not used by the second layer picture 260.

[0175] Figure 9 b shows the relationship with Figure 9 The video data stream shown in a is similar to, but includes only, one sub-image from sub-image 28 of image 261 in the first layer. Therefore, in Figure 9 In example b, the corresponding CPB levels are listed in size ratio. Figure 9 The fraction of a sub-picture in sub-picture 28 is smaller because the video data stream actually only includes one sub-picture from the first layer of sub-pictures. Picture 260 has a higher share relative to the total picture area signaled in the video data stream, i.e., 33% instead of 20%. (See also: Regarding...) Figure 9 As described in a, taking into account the following Figure 9 In case b, image 261 only includes one of the two sub-images 28, and accordingly, the determination for... Figure 9 Examples b are the second score 1322 and the first score 1342 associated with the reference decoder capability requirements.

[0176] According to an embodiment, encoder 40 can encode information about the first and second scores by writing a first score into a multi-layer video data stream on one hand and dividing a second score by (a minus the first score) into the multi-layer video data stream on the other hand. Devices 10 and 100 can decode information about the first and second scores accordingly by reading the first score from the multi-layer video data stream and further reading the second score. Alternatively, devices 10 and 100 can decode information about the first and second scores by reading the first score and another score from the data stream; and deriving the second score by multiplying the other score by the difference between a first score and a second score.

[0177] According to some examples, encoder 40 can specify one or more reference decoder capability requirements among several reference decoder capability requirements (e.g., ...). Figure 1 The decoder capability requirement 60 shown is encoded into a multi-layer video data stream. In other words, encoder 40 can encode information from one or more, but not necessarily all, of a plurality of reference decoder capability requirements into a multi-layer video data stream. For example, information about other reference decoder capability requirements can be derived through default rules. For example, in Figure 9 In the example, based on information about the relative sizes of the sub-images subdivided from image 261 in the first layer, regarding Figure 9 a and Figure 9 Information about one decoder capability requirement in the decoder capability requirements of b can be derived from the other decoder capability requirement.

[0178] For example, encoder 40 can be configured to encode information for at least one of several reference decoder capability requirements, such that the first fraction 134 and the second fraction 132 relate to reduced versions of multi-layer video data streams, for example, Figure 9 The first score is 1342, and the second score is 1322. The encoder can encode information such that, for each of one or more independently coded sub-pictures present in the reduced version, the first score is deriveable from the multi-layer video data stream. For example, refer to... Figure 9 a, knowing the ratio between the two sub-images of image 261 (e.g., in Figure 9 In the case that (a is 50%) (this percentage can be derived from parameters in the video encoded by encoder 40 into data stream 14, such as, for example, the width and / or height of a sub-picture measured in a sample or tile), devices 10, 100 can derive a first score for the right sub-picture of picture 261 from the first score of the left sub-picture of picture 261.

[0179] Additionally or alternatively, encoder 14 may be configured to encode information for at least one additional reference decoder capability requirement among several reference decoder capability requirements, such that the first score and the second score (e.g., Figure 9 The first score 1341 and the second score 1321 of a relate to a non-reduced version of the multi-layer video data stream, and for each sub-picture in the independently encoded sub-picture of the at least one first layer, the first score 134 is derived from the multi-layer video data stream 14.

[0180] like Figure 9 As shown, devices 10 and 100 can add the first fraction 134 and the second fraction 132 to obtain the actual decoder capability requirements, such as CPB, for the corresponding layer set. As illustrated, the actual decoder capability requirements can be specific to the video data stream specific to the sub-picture.

[0181] As mentioned, devices 10 and 100 can be mixers, meaning they can mix multi-layer video data stream 14 with other video data streams. To this end, devices 10 and 100 can use information about the first and second fractions to form another data stream based on data stream 14 and other input video data streams, for example, as about... Figure 10 As described.

[0182] 5. Based on the encoder 40 and extractor 10 in the fifth aspect.

[0183] This section refers to Figure 1The embodiments according to the fifth aspect are described, wherein the details described in Section 0 may optionally be applied to the embodiments according to the fifth aspect. Furthermore, as... Figure 2 and Figure 5 The details of the data streams 12 and 14 shown may also optionally be included in the embodiments of the third aspect, in which case the corresponding descriptions of Sections 1 and 2 apply accordingly. Optionally, the features described with respect to the first, second, third, and especially the fourth aspects may also be combined with the fifth aspect. In particular, examples of the embodiments described in Section 5 may relate to the embodiments described in Section 4, particularly regarding decoder capability requirements and the first and second scores.

[0184] The fifth aspect of the embodiments may involve the selective application of constraints on layers of the OLS.

[0185] As mentioned earlier, the syntax element `ref_level_fraction_minus1[i][j]` (whether explicitly signaled or inferred) is used to impose certain layer-specific constraints, such as the number of tiles in the prior art. However, for (reference) layers without sub-images, these constraints may not be necessary to be so stringent, or may even be difficult to meet from the encoder's perspective, since the extraction process does not encounter the corresponding layer, yet tightens the constraints that the layer must satisfy.

[0186] Therefore, in another embodiment, part of this aspect involves selectively imposing the following constraints only on layers in the OLS that contain sub-images (i.e., layers that will undergo image size reduction during the extraction process). Thus, the encoder can utilize fewer restrictions on layers that are not sub-image subdivisions.

[0187] According to an embodiment of the fifth aspect, the encoder 10 may encode image 261 into one or more first layers (e.g., first layer L1) by subdividing it into independently coded sub-images 28, and encode image 260 into one or more second layers (e.g., layer L0) in a non-subdivided manner, for example, as Figure 1 or Figure 2 As shown. The encoder may selectively check layer-specific constraints for the first layer of a layer set (i.e., the layer in which an image is encoded using sub-image subdivision (i.e., the layer in which an image is encoded by sub-image subdivision) or for the corresponding layer set of one or more OLSs indicated in OLS Instruction 18. In other words, checks may be performed for layers in which images are encoded using sub-image subdivision, but checks may be omitted for layers in which images are encoded without subdivision. In other words, checks may be performed only for layers in which images are encoded using sub-image subdivision.

[0188] The constraints for selectively checking the first layer relate to a predetermined set of parameters for the corresponding first-layer sub-pictures. In other words, for each sub-picture in the corresponding first-layer sub-pictures, the encoder can check constraints for each of one or more reference decoder capability requirements, said constraints being specific to said reference decoder capability requirements. For example, one or more reference decoder capability requirements for a corresponding sub-picture (for which they are checked) can refer to the sequence of sub-pictures generated by extracting the corresponding sub-pictures from the corresponding first layer, or it can refer to the contribution of the sub-picture sequence to the video data or streaming video sequence generated by extracting the corresponding sub-pictures from the corresponding first layer. As mentioned in Section 0, the bitstream described by the OLS can also be extended after extraction by extractor 10 by the selection / extraction of one or more layers and / or one or more temporal sub-layers of the OLS. This also applies to the case where extractor 10 extracts a sub-picture-specific video data stream, such that one or more reference decoder capability requirements signaled by the encoder can be applied to each sub-picture in the video data stream to a degree defined by a score of the corresponding reference decoder capability requirement (the score being sub-picture-specific, for example, the first score described in Section 4). In other words, one or more reference decoder capability requirements can be associated with a corresponding sub-picture, each of which is applicable to possible sub-bitstreams (extractable from the sub-picture-specific video data stream), and each of which imposes reference decoder capability requirement-specific constraints.

[0189] For example, the parameters on which constraints are imposed may include one or more of the following parameters:

[0190] - The number of samples (e.g., LUMA samples), such as MaxLumaPs

[0191] - For example, image size based on the number of samples, such as image width and / or image height.

[0192] - The number of tiles in a row and / or the number of tiles in a column, such as MaxTileRows and / or MaxTileCols

[0193] - Total number of tiles

[0194] For example, encoder 10 may use a score attributable to the corresponding reference decoder capability requirement of the corresponding sub-image (e.g., the first score 134 as described in Section 4) to determine a parameter or attribute for a single sub-image, such as RefLevelFraction[i][j] in the following example, where i indicates the corresponding reference decoder capability requirement and i indicates the corresponding sub-image.

[0195] The following are example specifications highlighting changes related to the current VVC draft specification:

[0196] The requirement for bitstream consistency is that each layer in the bitstream generated by extracting the j-th subpicture (where j is in the range of 0 to sps_num_subpics_minus1 (inclusive), and sps_num_subpics_minus1 is greater than 0) and conforming to the profile where general_tier_flag is equal to 0 and the tier is equal to ref_level_idc[i] (where i is in the range of 0 to num_ref_level_minus1 (inclusive)) should comply with the following constraints for each bitstream consistency test as specified in Appendix C of the VVC specification:

[0197] -Ceil(256*SubpicSizeY[j]÷RefLevelFraction[i][j]) should be less than or equal to MaxLumaPs, where MaxLumaPs is specified for the level ref_level_idc[i] in Table A.1 (of VVC).

[0198] The value of -Ceil(256*(sps_subpic_width_minus1[j]+1)*CtbSizeY÷RefLevelFraction[i][j]) should be less than or equal to Sqrt(MaxLumaPs*8).

[0199] The value of -Ceil(256*(sps_subpic_height_minus1[j]+1)*CtbSizeY÷RefLevelFraction[i][j]) should be less than or equal to Sqrt(MaxLumaPs*8).

[0200] The value of SubpicWidthInTiles[j] should be less than or equal to MaxTileCols, and the value of SubpicHeightInTiles[j] should be less than or equal to MaxTileRows, where MaxTileCols and MaxTileRows are specified for the level ref_level_idc[i] in Table A.1 of (VVC).

[0201] The value of -SubpicWidthInTiles[j]*SubpicHeightInTiles[j] should be less than or equal to MaxTileCols*MaxTileRows*RefLevelFraction[i][j], where MaxTileCols and MaxTileRows are specified for the level ref_level_idc[i] in Table A.1 of (VVC).

[0202] In another embodiment, for an OLS layer where sps_num_subpics_minus1 equals 0, RefLevelFraction[i][j] is derived to be equal to 255 (e.g., with a fraction of 100%).

[0203] In another embodiment, the limits associated with MaxLumaPs, the maximum and minimum aspect ratios, slices or subpicks, or the number of tile columns or rows are layer-specific and use a layer-specific ref_level_fraction (i.e., 255 for layers without subpicks), while the layer-specific limits associated with CPB size, bit rate, and MinCR, as described below, are enforced based on an OLS-specific OLSRefLevelFraction equal to OLS_fraction[k] (e.g., as derived above in Section 4).

[0204] The following are the derived variables SubpicCpbSizeVcl[i][j] and SubpicCpbSizeNal[i][j]:

[0205] SubpicCpbSizeVcl[i][j]=Floor(CpbVclFactor*MaxCPB*OLSRefLevelFraction[i][j]÷256)(D.5)

[0206] SubpicCpbSizeNal[i][j]=Floor(CpbNalFactor*MaxCPB*OLSRefLevelFraction[i][j]÷256)(D.6)

[0207] As specified in Clause A.4.2, MaxCPB is derived from ref_level_idc[i].

[0208] The following are the derived variables SubpicBitRateVcl[i][j] and SubpicBitRateNal[i][j]:

[0209] SubpicBitRateVcl[i][j]=Floor(CpbVclFactor*ValBR*OLSRefLevelFraction[i][j]÷256)(D.7)

[0210] SubpicBitRateNal[i][j]=Floor(CpbNalFactor*ValBR*OLSRefLevelFraction[i][j]÷256)(D.8)

[0211] Note 1 - When extracting sub-pictures, the CpbSize of the resulting bitstream (indicated or inferred in SPS) is greater than or equal to SubpicCpbSizeVcl[i][j] and SubpicCpbSizeNal[i][j], and its bitrate (indicated or inferred in SPS) is greater than or equal to SubpicBitRateVcl[i][j] and SubpicBitRateNal[i][j].

[0212] For the value of SubpicSizeInSamplesY for AU 0, the sum of the NumBytesInNalUnit variables for AU 0 corresponding to the j-th subpic should be less than or equal to FormatCapabilityFactor*(Max(SubpicSizeY[j],fR*MaxLumaSr*OLSRefLevelFraction[i][j]÷256)+MaxLumaSr*(AuCpbRemovalTime[0]-AuNominalRemovalTime[0])*OLSRefLevelFraction[i][j])÷(256*MinCr), where MaxLumaSr and FormatCapabilityFactor are the values ​​applicable to AU 0 specified in Tables A.2 and A.3 at the level ref_level_idc[i], respectively, and MinCr is derived as indicated in A.4.2 of the (VVC draft).

[0213] The sum of the NumbytesInNalUnit variables corresponding to the AU n (where n is greater than 0) of the j-th sub-image should be less than or equal to FormatCapabilityFactor*MaxLumaSr*(AuCpbRemovalTime[n]-AuCpbRemovalTime[n-1])*OLSRefLevelFraction[i][j]÷(256*MinCr), where MaxLumaSr and FormatCapabilityFactor are the values ​​applicable to AU n specified in Tables A.2 and A.3 at the level ref_level_idc[i], and MinCr is derived as indicated in A.4.2 of the (VVC draft).

[0214] The following describes an embodiment of the encoder 40, extractor 10, and multi-layer video data stream 14 according to the fifth aspect.

[0215] According to the fifth aspect, the multi-layer video data stream comprises multiple layers. The multi-layer video data stream 14, for example by means of encoder 40, employs a method of subdividing one or more first layers (e.g., layer L1) into independently coded subpictures, and a method of unsubdividing one or more second layers (e.g., layer L1), to encode video pictures 26 into the layers of the multi-layer video data stream. Furthermore, the multi-layer video data stream 14 includes an indication of a layer set, such as OLS as described in previous sections, wherein, according to the fifth aspect, the layer set includes at least one first layer, i.e., a layer in which pictures are encoded using a method of subdividing into independently coded subpictures. According to the fifth aspect, the multi-layer video data stream encodes reference decoder capability requirement information therein, for example, as described in Section 4, the reference decoder capability requirement information relates to a non-reduced version of the decoding layer set and includes a first decoder capability requirement parameter for each layer of the layer set and a second decoder capability requirement parameter for the layer set. As described in Section 4, a non-reduced version may refer to a version of the video data stream encoded into the layer set (which includes all video data encoded into the layer set), i.e., all subpictures of at least one first layer of the layer set. For example, the first decoder capability requirement parameter could be the number of tiles into which the image of a layer is subdivided. For example, a tile is a segment of an image that is independently encoded. For example, the second decoder capability requirement parameter could refer to the required or necessary image buffer size, such as ACPB and / or the decoded image buffer (DPB) size and / or the minimum compression ratio (minCR).

[0216] According to the fifth aspect, encoder 40 is configured to, for each of one or more reduced versions of a layer set (for which at least one sub-picture-related portion of the first layer has been removed), determine additional reference decoder capability requirement information related to the corresponding unreduced or reduced version of the decoding layer set. A reduced version of the layer set may refer to a video data stream that does not contain all the video data of the layer set (e.g., sub-picture-related video data stream 12), that is, when extracting the corresponding reduced version of the layer set, the sub-picture-related portion (i.e., as per the description of the sub-picture-related portion) is omitted from it. Figure 1 , Figure 2 , Figure 5 The video data stream described is a portion unrelated to a predetermined set of sub-pictures. Encoder 40 is configured to determine additional reference decoder capability requirement information by scaling a first decoder capability requirement parameter with a first score to obtain a third decoder capability requirement parameter for each layer of the layer set. For example, the first score may be determined based on the ratio of the picture size between the pictures in each layer within the layer set and the pictures in a scaled-down version of that layer set. Encoder 40 may optionally encode the first score into the multi-layer video data stream 14. Therefore, extractor 10 may optionally derive the first score from the multi-layer video data stream 14, or alternatively determine the first score as performed by encoder 40.

[0217] Encoder 40 is further configured to determine additional reference decoder capability requirement information by scaling the second decoder capability requirement parameters using a second score to obtain a corresponding reduced version of the fourth decoder capability requirement parameters for the layer set. The second score may, for example, be different from the first score.

[0218] The second score can be determined based on the ratio encoding up to section 4, such as score 134 of score 132. Encoder 40 may optionally encode the second score into the multi-layer video data stream 14. Accordingly, extractor 10 may optionally derive the second score from the multi-layer video data stream 14, or determine the second score in a similar manner to that performed by encoder 40.

[0219] Encoder 10 forms a sub-picture-specific video data stream 12 based on multi-layer video data stream 14 by removing at least one sub-picture-related portion of the first layer of multi-layer video data stream 14; and providing a third decoder capability requirement parameter and a fourth decoder capability requirement parameter to the sub-picture-specific video data stream 12. Alternatively, encoder 10 forms the sub-picture-specific video data stream 12 based on multi-layer video data stream 12 by removing at least one sub-picture-related portion of the first layer; and providing a first score and a second score to the sub-picture-specific video data stream 12 for determining additional reference decoder capability requirement information related to a predetermined unreduced version of the layer set decoded by the decoder.

[0220] 6. According to Figure 10 Encoder 40 and decoder 110

[0221] Figure 10 A first encoder 401 and a second encoder 402 are shown, wherein each encoder may optionally correspond to, as shown in the figure below. Figure 1 The encoder 40 is the same as described in Sections 1-5. Encoder 401 is configured to encode video data stream 121, while encoder 402 is configured to encode video data stream 122. Video data streams 121 and 122 may optionally, but not necessarily, include multiple layers. In other words, according to... Figure 10 The video data streams 121 and 122 can optionally be multi-layer video data streams (such as video data stream 14), but in this example, they may include only a single layer. In addition, according to... Figure 10 The video data streams 121 and 122 may optionally correspond to the video data stream 14 as described in the previous chapters.

[0222] Encoder 401 is configured to encode image 261 into video data stream 121. Encoder 402 is configured to encode image 262 into video data stream 122. Images 261 and 262 may optionally be encoded as one or more independently encoded sub-images 28 into the respective video streams 121 and 122. Encoders 401 and 402 are configured to encode an indication 80 into the respective video streams 121 and 122, the indication 80 indicating whether images 261 and 262 are encoded as one or more independently encoded sub-images into the respective video streams. The indication 80 distinguishes different types of encoding independence between one or more independently encoded sub-images 28 and the surrounding area of ​​one or more independently encoded sub-images. For example, the surrounding area of ​​one or more independently encoded sub-images may refer to the image region surrounding the respective sub-image.

[0223] For example, the indicator 80 can be an m-array syntax element (where m > 1), or it can contain more than one syntax element to indicate whether any such independently encoded sub-pictures exist, and if so, additional syntax elements reveal the type of decoding independence.

[0224] Figure 10A device 100 is further illustrated for providing a video data stream 20 based on one or more input video data streams 12 (such as video data stream 121 and video data stream 122). For example, device 100 may mix video data streams 121, 122 to determine video data stream 20. Thus, device 100 may be referred to as mixer 100. Optionally, device 100 may be further configured to extract video data stream 20 from one or more input video data streams (such as video data stream 121 and / or video data stream 122). For example, mixer 100 may extract part or all of the video data from each of one or more input video data streams in its input video data streams and provide video data in video data stream 20, wherein the video data is extracted from different input video data streams in its input video data streams. For example, as Figure 10 As shown, device 100 can extract one or more sub-pictures from each of the video data streams 121, 122, and then combine them with one or more sub-pictures from the sub-pictures associated with the same presentation time to obtain video data stream 20.

[0225] Figure 10 Decoder 110 is further shown, which may optionally correspond to decoder 50 as described with respect to the previous chapters. Decoder 110 is configured to decode from video data stream 20 in indication 80.

[0226] In the example, regardless of which type of coding independence is indicated by Indication 80, encoders 401, 402 and decoder 110 may, in response to (e.g., only if the following conditions are met, or not if the following conditions are not met) Indication 80 instructs the encoding of pictures into the video data stream as one or more independently coded subpictures, perform one or more of the following actions:

[0227] - Read sub-picture identifiers from each bitstream packet of the video data stream. These sub-picture identifiers indicate which of the one or more independently encoded sub-pictures will be encoded into the corresponding bitstream packet, and / or

[0228] - Prevent filters within a loop from crossing sub-image boundaries, and / or

[0229] - Depending on which of one or more independently coded sub-pictures is encoded into a predetermined bitstream block, the picture location is derived from the block address of the picture's block, which is encoded into a predetermined bitstream portion of the video data stream (and, for example, otherwise such picture location is derived from the block address by inevitably referencing a predetermined picture location such as its top-left corner), and / or

[0230] -Considering the different NAL unit types that may appear in a single access unit of a video data stream. For example, allowing... Figure 10 The mixer 100 employs a method such that in the mixed data stream 90, one sub-picture 28 has one NAL unit type, while another sub-picture 28 within the same picture is encoded using a different NAL unit type; the NAL unit types are encoded into bitstream blocks 16 and remain unmodified when they become blocks 16' of stream 90; the decoder 110 indeed sees no conflicts because the pictures are encoded with complete coding independence, regardless of the type used for the sub-pictures 28 in the decoded (mixed) picture 26', and / or

[0231] - On the encoder side, for each sub-image in one or more independently encoded sub-images, derive decoder capability requirement information.

[0232] For example, the encoding independence of different types between sub-images of an image includes:

[0233] - An encoder-constrained encoding type, wherein the decoder is configured such that, if an encoder-constrained encoding type is indicated by an indication of independently encoded sub-images, independence processing is not performed at the boundaries of independently encoded sub-images, and

[0234] - Decoder-aware encoding type, wherein the decoder is configured to perform independence processing at the boundaries of independently encoded sub-images by vector trimming of vector-based predictions and / or boundary padding at the boundaries of independently encoded sub-images if the decoder-aware encoding type is indicated by an indication of the independently encoded sub-images.

[0235] In other words, the embodiment of the sixth aspect may take into account the mixing of sub-images or MCTS.

[0236] The current VVC draft specification includes many measures for implementing the functionality of dividing images into sub-images to achieve extraction and merging, such as:

[0237] ● Include the sub-image identifier in the slice header, or

[0238] ● The block addressing scheme depends on the sub-image identifier, or

[0239] ● For consistency information of a single sub-image, or

[0240] ● Mix NAL unit types (e.g., IRAP and TRAIL) in the image.

[0241] However, all these measures depend on the boundary handling properties of the sub-images, namely, applying the same method to samples and extrapolated grammatical prediction samples located outside the boundaries of the sub-images as to the boundaries of the image (i.e., special boundary padding on the decoder side). This allows for independent encoding of sub-images, followed by extraction and merging. This property is also used to implement all the above measures for the extraction and merging functionality. Figure 11 This illustrates a bitstream in which only the sub-images processed by this decoder side are used in the first two access units of the bitstream. In other words, Figure 11 An example is shown where the bitstreams of decoder side-boundary processing and encoder side-boundary processing are mixed in their independently encoded sub-images.

[0242] However, there are alternative methods to independently encode such rectangular regions, namely, motion constraints followed during encoding, or in other words, encoder side boundary processing. The encoder can easily avoid references that would lead to reconstruction errors after extraction or merging by not pointing outside the perimeter of each sub-image. Based on the signaling of the current technology level, the encoder would have to instruct from... Figure 11 The second IRAP forward in the L1 model does not encode the right sub-image independently because no decoder-side boundary processing occurs, and therefore, some of the above measures for extracting and merging functionality can be used effectively. Most importantly, consistency information is tied to decoder-side boundary processing. However, the mere indications for other means of independent encoding should enable the encoder and allow the network device or decoder to facilitate said means.

[0243] It is important to note that mixing different types of such bitstreams on independent encodings between adjacent sub-images does not necessarily require more than a single layer, such as... Figure 12 As shown. Figure 12 An example of a single layer that mixes the independence of sub-images is shown.

[0244] Therefore, part of this aspect is to allow the decoder to recognize that, despite the lack of boundary padding, all the aforementioned measures (addressing scheme, consistency information) for independent region encoding can still be used by indicating in the bitstream the means of encoding regions independently through constrained encoding of certain subpictures. In one embodiment of this aspect, the flag indicating the decoder-side padding process (sps_subpic_treated_as_pic_flag[i] equals 1) is changed to sps_subpic_treated_as_pic_mode[i], where the new state (equals 2) indicates that the i-th subpicture is encoded independently by means of encoder-side boundary processing constraints.

[0245] The following example is shown:

[0246]

[0247] A value of 42 for `sps_subpic_treated_as_pic_mode[i]` indicates that, during decoding excluding loop filtering operations, the i-th sub-picture of each encoded picture in the CLVS is treated as a picture. A value of 0 or 1 for `sps_subpic_treated_as_pic_mode[i]` indicates that, during decoding excluding loop filtering operations, the i-th sub-picture of each encoded picture in the CLVS is not treated as a picture. If it does not exist, the value of `sps_subpic_treated_as_pic_mode[i]` is assumed to be 42. A value of 3 for `sps_subpic_treated_as_pic_mode[i]` is reserved for future use by ITU-T|ISO / IEC.

[0248] Furthermore, the following constraint applies: different types of NAL units cannot be mixed unless the sub-image is constrained on the encoder side or the boundary is processed on the decoder side.

[0249] When any two subpics in an image have different NAL unit types, the value of sps_subpic_treated_as_pic_mode[] should not be equal to 0 for all subpics in an image that contain at least one P or B slice.

[0250] Figure 13 Examples of allowed and prohibited mixing of NAL units are shown, depending on whether the sub-image is processed independently of encoder-side constraints or decoder-side boundary processing.

[0251] 7. Other embodiments

[0252] In the preceding sections 0 through 6, while some aspects have been described as features within the context of the device, it is clear that such descriptions can also be considered descriptions of the corresponding features of the method. Although some aspects have been described as features within the context of the method, it is evident that such descriptions can also be considered descriptions of the corresponding features concerning the functionality of the device.

[0253] Some or all of the method steps can be performed by (or using) hardware devices such as microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps can be performed by such devices.

[0254] The encoded image signal of this invention can be stored on a digital storage medium or transmitted over a transmission medium such as the Internet, such as a wireless or wired transmission medium.

[0255] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software, or at least partially in hardware or at least partially in software. This implementation can be performed using a digital storage medium (e.g., floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, or flash memory) on which electronically readable control signals are stored, which cooperate (or are capable of cooperating with) a programmable computer system to cause the corresponding methods to be executed. Therefore, the digital storage medium may be computer-readable.

[0256] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0257] Generally, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product is run on a computer. The program code may, for example, be stored on a machine-readable medium.

[0258] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0259] In other words, therefore, embodiments of the method of the present invention are computer programs having program code for performing one of the methods described herein when the computer program is run on a computer.

[0260] Therefore, another embodiment of the method of the present invention includes a data carrier (or digital storage medium, or computer-readable medium) on which a computer program is recorded, said computer program being used to perform one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0261] Therefore, another embodiment of the method of the present invention represents a data stream or signal sequence of a computer program used to perform one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0262] Other embodiments include processing components, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0263] Another embodiment includes a computer on which a computer program is installed, the computer program being used to perform one of the methods described herein.

[0264] Further embodiments of the invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program to a receiver for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a memory device, or a similar device. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0265] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0266] The device described in this article may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0267] The methods described in this article can be performed using hardware devices, computers, or a combination of hardware devices and computers.

[0268] As can be seen from the foregoing detailed description, various features are combined in the examples for the purpose of streamlining this disclosure. This approach of the disclosure should not be construed as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as reflected in the following claims, the subject matter may lie in fewer features than all the features of a single disclosed example. Therefore, the following claims are thus incorporated into this detailed description, wherein each claim may independently represent a separate example. While each claim may independently represent a separate example, it should be noted that although dependent claims may refer in the claims to a specific combination with one or more other claims, other examples may also include combinations of dependent claims with the subject matter of each other dependent claim, or combinations of each feature with other dependent or independent claims. Such combinations are proposed herein unless stated not to be expected. Furthermore, it is desirable to include features of the claims in any other independent claims, even if the claim is not directly dependent on that independent claim.

[0269] The embodiments described above are merely illustrative of the principles of this disclosure. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended that one be limited only by the scope of the pending patent claims and not by the specific details presented as part of the description and interpretation of the embodiments herein.

Claims

1. An encoder (40) for encoding video (24) into a multi-layer video data stream (14), said encoder comprising: processor; as well as A memory storing instructions that, when executed by the processor, cause the processor to: Using one or more inter-layer prediction tools from a set of inter-layer prediction tools for prediction from the second layer (L0) of the multi-layer video data stream (14), and using a first configuration setting for independent encoding of each sub-picture, a first version (241) of the video is encoded into the first layer (L1) of the multi-layer video data stream (14), and Using the second configuration settings for the independent encoding of each sub-image, the second version (240) of the video is encoded into the second layer (L0) of the multi-layer video data stream (14). Depending on whether the first configuration settings and the second configuration settings have a predetermined relationship, a variable representing the comparison between the first configuration settings and the second configuration settings is set, and When it is determined based on the variables that the first configuration setting and the second configuration setting do not have the predetermined relationship, at least one of the set of inter-layer prediction tools is disabled or avoided.

2. The encoder as claimed in claim 1, wherein, When the instruction for checking whether the first configuration setting and the second configuration setting have the predetermined relationship is executed by the processor, the processor identifies the predetermined relationship when the first configuration setting and the second configuration setting indicate at least one of the following: The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images equal in both versions. The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple equal sub-images within the first and second versions of the video, wherein the boundaries of the sub-images are spatially consistent. The first and second versions of the video are respectively encoded into the first and second layers by subdividing them into multiple sub-images. The multiple sub-images are consistent in the sub-image IDs that signal the sub-images in the multi-layer video data stream (14), or The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images, and the multiple sub-images have consistent boundary processing.

3. The encoder as claimed in claim 1, wherein, At least one inter-layer prediction tool includes at least one of the following: Inter-layer motion vector prediction Vector-based inter-layer prediction optical flow refinement, or Motion vector wrapping in vector-based inter-layer prediction.

4. The encoder of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: The first configuration setting and the second configuration setting are signaled in the multi-layer video data stream (14).

5. The encoder of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: When it is determined that the first configuration setting and the second configuration setting have the predetermined relationship, any of the inter-layer prediction tools from the set of inter-layer prediction tools are activated or utilized to encode the first version of the video (241).

6. A decoder (50) for decoding multi-layer video data streams (12, 14), the decoder comprising: processor; as well as A memory storing instructions that, when executed by the processor, cause the processor to: Obtain the multi-layer video data stream (12, 14) including the first layer (L1) and the second layer (L0). A first version (241) of video (24) is decoded from the first layer (L1) of the multi-layer video data stream using one or more inter-layer prediction tools from a set of inter-layer prediction tools for prediction from the second layer (L0) of the multi-layer video data stream and using a first configuration setting for independent encoding of each sub-picture. The second version (240) of the video (24) is decoded from the second layer (L0) of the multi-layer video data stream using a second configuration setting for independent encoding of each sub-image. Based on a variable representing the comparison of the first and second configuration settings, which depends on whether the first and second configuration settings have a predetermined relationship, it is checked whether the first and second configuration settings have a predetermined relationship. If it is determined based on the variables that the first configuration setting and the second configuration setting do not have the predetermined relationship, then at least one of the inter-layer prediction tools is deactivated.

7. The decoder of claim 6, wherein independent decoding of each sub-image comprises at least one of the following: Perform vector trimming based on vector prediction at the boundaries of sub-images, or Fill the boundaries at the edges of the sub-images.

8. The decoder of claim 6, wherein the first configuration setting and the second configuration setting have the predetermined relationship when the first configuration setting and the second configuration setting indicate at least one of the following: The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images equal in both versions. The first and second versions of the video are respectively encoded into the first and second layers by subdividing them into multiple sub-images. These multiple sub-images are identical in the sub-image IDs that signal the sub-images in the multi-layer video data stream. The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images, and the multiple sub-images have consistent boundary processing.

9. The decoder as claimed in claim 6, wherein, The at least one inter-layer prediction tool includes at least one of the following: Inter-layer motion vector prediction Vector-based inter-layer prediction optical flow refinement, or Motion vector wrapping in vector-based inter-layer prediction.

10. A method for encoding video (24) into a multi-layer video data stream (14), the method comprising: Using one or more inter-layer prediction tools from a set of inter-layer prediction tools for prediction from the second layer (L0) of the multi-layer video data stream (14), and using a first configuration setting for independent encoding of each sub-picture, a first version (241) of the video is encoded into the first layer (L1) of the multi-layer video data stream (14), and Using the second configuration settings for the independent encoding of each sub-image, the second version (240) of the video is encoded into the second layer (L0) of the multi-layer video data stream (14). Depending on whether the first configuration settings and the second configuration settings have a predetermined relationship, a variable representing the comparison between the first configuration settings and the second configuration settings is set, and When it is determined based on the variables that the first configuration setting and the second configuration setting do not have the predetermined relationship, at least one of the set of inter-layer prediction tools is disabled or avoided.

11. The method of claim 10, wherein, Checking whether the first configuration setting and the second configuration setting have the predetermined relationship includes identifying the predetermined relationship when the first configuration setting and the second configuration setting indicate at least one of the following: The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images equal in both versions. The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple equal sub-images within the first and second versions of the video, wherein the boundaries of the sub-images are spatially consistent. The first and second versions of the video are respectively encoded into the first and second layers by subdividing them into multiple sub-images. The multiple sub-images are consistent in the sub-image IDs that signal the sub-images in the multi-layer video data stream (14), or The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images, and the multiple sub-images have consistent boundary processing.

12. The encoder of claim 10, wherein, The at least one inter-layer prediction tool includes at least one of the following: Inter-layer motion vector prediction Vector-based inter-layer prediction optical flow refinement, or Motion vector wrapping in vector-based inter-layer prediction.

13. The method of claim 10, further comprising: The first configuration setting and the second configuration setting are signaled in the multi-layer video data stream (14).

14. The method of claim 10, further comprising, when determining that the first configuration setting and the second configuration setting have the predetermined relationship, activating or utilizing any of the inter-layer prediction tools from the set of inter-layer prediction tools to encode the first version of the video (241).

15. The method of claim 10, further comprising: Generate flags indicating that the first and second versions of the video include different numbers of sub-images; The flag is included in the multi-layer video data stream; as well as Based on the flag, check whether the first configuration setting and the second configuration setting have a predetermined relationship.

16. A non-transitory computer-readable medium comprising a program that, when run by a processor, causes the processor to perform the method of claim 10.

17. A method for decoding multi-layer video data streams (12, 14), the method comprising: Obtain the multi-layer video data stream (12, 14) including the first layer (L1) and the second layer (L0). A first version (241) of video (24) is decoded from the first layer (L1) of the multi-layer video data stream using one or more inter-layer prediction tools from a set of inter-layer prediction tools for prediction from the second layer (L0) of the multi-layer video data stream and using a first configuration setting for independent encoding of each sub-picture. The second version (240) of the video (24) is decoded from the second layer (L0) of the multi-layer video data stream using a second configuration setting for independent encoding of each sub-image. Based on a variable representing the comparison of the first and second configuration settings, which depends on whether the first and second configuration settings have a predetermined relationship, it is checked whether the first and second configuration settings have a predetermined relationship. If it is determined based on the variables that the first configuration setting and the second configuration setting do not have the predetermined relationship, then at least one of the inter-layer prediction tools is deactivated.

18. The method of claim 17, wherein independent decoding of each sub-image comprises at least one of the following: Perform vector trimming based on vector prediction at the boundaries of sub-images, or Fill the boundaries at the edges of the sub-images.

19. The method of claim 17, wherein the first configuration setting and the second configuration setting have the predetermined relationship when the first configuration setting and the second configuration setting indicate at least one of the following: The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images equal in both versions. The first and second versions of the video are respectively encoded into the first and second layers by subdividing them into multiple sub-images. These multiple sub-images are identical in the sub-image IDs that signal the sub-images in the multi-layer video data stream. The first and second versions of the video are encoded into the first and second layers respectively by subdividing them into multiple sub-images, and the multiple sub-images have consistent boundary processing.

20. The method of claim 17, wherein, The at least one inter-layer prediction tool includes at least one of the following: Inter-layer motion vector prediction Vector-based inter-layer prediction optical flow refinement, or Motion vector wrapping in vector-based inter-layer prediction.

21. A non-transitory computer-readable medium comprising a program that, when run by a processor, causes the processor to perform the method of claim 17.

Images