Concept for picture / video data stream allowing effective reduction or efficient random access

Abstract

To provide a picture / video data stream allowing effective reduction or efficient random access.SOLUTION: A reduced video data stream is reducibly rendered so that pictures of the reduced video data stream are restricted only in a predetermined subarea 22 of the pictures of the original video data stream 10, and so that transcoding, such as re-quantization, is avoided and a conformance of the reduced video data stream relative to the codec underlying the original video data stream is maintained. This is achieved by providing the video data stream with information 50 comprising an indication of the predetermined subarea and replacement indices for redirecting to refer to indices 48 formed by a payload portion 18 and / or replacement parameters for adjusting a first set of coding parameter settings 20a to bring about a second set of coding parameter settings 20b.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to video / image coding and in particular to concepts that allow for an efficient reduction of such data streams, that allow for easier handling of such data streams and / or that allow for more efficient random access to video data streams. [Background technology]

[0002] There are many video codecs that allow scalability of video data streams without transcoding, i.e. without the need to perform decoding and encoding consecutively. Examples of this kind of scalable video data streams are data streams that are scalable, for example, with respect to temporal resolution, spatial resolution or signal-to-noise ratio, by simply omitting some of the enhancement layers of each scalable video data stream. However, to date, there is no video codec that allows scalability with low computational complexity regarding scene delimitation. In HEVC, concepts exist or have been proposed for restricting HEVC data streams to image sub-regions, but these are still computationally complex.

[0003] Furthermore, depending on the application, the image content to be encoded into the data stream may be in a form that cannot be effectively encoded within the rectangular image area that is usually provided. For example, panoramic image content may be projected onto a two-dimensional plane, forming an image area such that the projection target, i.e., the footprint of the panoramic scene onto the image area, may be non-rectangular or even non-convex. In that case, more efficient encoding of the image / video data would be advantageous.

[0004] Furthermore, random access points are provided to existing video data streams in such a way that they cause significant bitrate peaks. In order to reduce the negative effects caused by these bitrate peaks, a reduction in the time granularity of the occurrence of these random access points can be considered. However, this increases the average time duration for randomly accessing such a video data stream, and therefore it would be advantageous to have a concept that solves this problem at hand in a more efficient way. Summary of the Invention [Problem to be solved by the invention]

[0005] It is therefore an object of the present invention to solve the above-mentioned problems. According to the present application, this is achieved by the subject matter of the independent claims. [Means for solving the problem]

[0006] According to a first aspect of the present application, a video data stream is reducibly rendered such that the constraints of the images of the reduced video data stream are merely limited to a predefined sub-region of the images of the original video data stream, and such that transcoding such as requantization is avoided and the compatibility of the reduced video data stream to the underlying codec of the original video data stream is maintained. This is achieved by providing a video data stream with information consisting of a representation of the predefined sub-region, a substitution index for redirecting an index configured by a payload portion for reference and / or a substitution parameter for adjusting a first set of encoding parameter settings to produce a second set of encoding parameter settings. The payload portion of the original video data stream has been coded with an image of the parameterized video therein using a first set of encoding parameter settings indexed by an index included in the payload portion. Additionally or alternatively, a similar measure is feasible for auxiliary enhancement information. It is therefore possible to reduce the video data stream to a reduced video data stream by performing redirection and / or adjustment, such that the second set of coding parameter settings is indexed by the payload portion index and thus becomes the valid coding parameter setting set, remove parts of the payload portion that refer to regions of the image outside the predefined sub-region, and modify location indications such as slice addresses of the payload portions to indicate positions measured from the periphery of the predefined sub-region instead of the periphery of the image. Alternatively, an already reduced data stream not including parts of the payload portion that refer outside the predefined sub-region may be modified by fly adjustment of parameters and / or auxiliary enhancement information.

[0007] According to a further aspect of the present invention, the transmission of image content is made more efficient in that the image content does not have to be formed or ordered in a predefined manner, e.g. the way the typically rectangular image regions supported by the underlying codec are written. Rather, a data stream having images encoded therein is provided to include, for a set of at least one predefined sub-region of the image, displacement information indicating the displacement within the region of the target image with respect to an undistorted or one-to-one corresponding or congruent copy to the region of the target image of the set. Providing such displacement information is useful, e.g. when conveying within an image a projection of a panoramic scene, e.g. when the projection is non-rectangular. This displacement information is also useful when, due to the reduction of the data stream, the image content conveyed within the smaller image of the reduced video data stream loses its suitability, e.g. in the case of an interesting panoramic view section to be transmitted within the reduced video data stream crossing a transition boundary such as a prediction.

[0008] According to a further aspect of the present invention, the negative impact of bit rate peaks in the video data stream caused by random access points is reduced by providing a video data stream with two sets of random access points: encoding a first set of one or more images into the video data stream with suspending temporal prediction at least within a first image sub-region so as to form a set of one or more first sets of random access points, and encoding a second set of one or more images into the video data stream with suspending temporal prediction within a second image sub-region different from the first image sub-region so as to form a set of one or more second random access points. In this way, it is possible for a decoder which randomly accesses the video data stream or wants to resume decoding to select one of the first and second random access points which are distributed in time and which enable at least random access with respect to the second image sub-region in the case of the second random access point and at least with respect to the first image sub-region in the case of the first random access point.

[0009] The above concepts can be advantageously used together. Further advantageous embodiments are the subject matter of the dependent claims. Preferred embodiments of the present application are described below with reference to the drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 shows a schematic diagram of a video data stream according to an embodiment of the present application for a first aspect according to which a video data stream can be reduced into a reduced video data stream relating to a sub-region of an image of the reducible video data stream. [Diagram 2] FIG. 2 shows a schematic diagram illustrating the interdependencies between the payload portion and the parameter set portion of the reducible video data stream of FIG. 1 according to an embodiment to illustrate the parameterization with which images are encoded in the reduced video data stream. [Diagram 3]FIG. 3 shows a schematic diagram for explaining a possible content of information provided by an embodiment of the video data stream of FIG. 1 enabling reduction. [Figure 4] FIG. 4 shows a schematic diagram illustrating a network device for receiving a reducible video data stream and deriving therefrom a reduced video data stream. [Diagram 5] FIG. 5 shows a schematic diagram illustrating an operation mode when reducing a video data stream according to an embodiment using parameter set redirection. [Figure 6] FIG. 6 shows a schematic diagram illustrating a video decoder 82 that receives a reduced video data stream and reconstructs images of the reduced video data stream that instead merely show sub-regions of images of the original video data stream. [Figure 7] FIG. 7 shows a schematic diagram of an alternative operation mode when reducing a reducible video data stream at this point using parameter set adjustment using substitutions in the information provided by the reducible video data stream. [Figure 8] FIG. 8 shows an example of a syntax for information that a reducible video data stream can provide. [Figure 9] FIG. 9 shows an alternative example of a syntax for the information that a reducible video data stream can provide. [Figure 10] FIG. 10 shows a further example of the syntax of information that a reducible video data stream can provide. [Figure 11] FIG. 11 shows a further example of syntax, now for information, to replace the SEI message. [Figure 12] FIG. 12 illustrates an example of a syntax table that may be used to form information relating to a multi-layer video data stream. [Figure 13]FIG. 13 shows a schematic diagram illustrating the relationship between tiles in a sub-region of an image in one reducible video data stream and corresponding tiles in an image in the other reduced video data stream, in accordance with one embodiment, to illustrate the possibility of spatial rearrangement of these tiles. [Figure 14] FIG. 14 shows an example of an image obtained by linearly predicting a panoramic scene. [Figure 15] FIG. 15 shows an example of an image with image content corresponding to a 3D projection of a panoramic scene. [Figure 16] FIG. 16 shows an image efficiently filled using the three-dimensional projection content of FIG. 15 by rearranging it. [Figure 17] FIG. 17 shows an example of a syntax table for displacement information using a data stream that can be provided by an embodiment according to the second aspect of the present invention. [Figure 18] FIG. 18 shows a schematic diagram illustrating the structure of a video data stream according to an embodiment related to the second aspect of the present invention. [Figure 19] FIG. 19 shows a schematic diagram illustrating possible contents for replacing information according to an embodiment. [Figure 20] FIG. 20 shows a schematic diagram illustrating an encoder configured to be able to form displacement information and simultaneously a reducible data stream. [Figure 21] FIG. 21 shows a schematic diagram illustrating a decoder configured to receive a data stream including displacement information to illustrate a possibility how the displacement information can be used advantageously. [Figure 22] FIG. 22 shows a schematic diagram illustrating a video data stream including sub-region specific random access images according to an embodiment related to a further aspect of the present application. [Figure 23A] FIG. 23A shows a schematic diagram illustrating a possible arrangement of sub-regions used according to different alternatives. [Figure 23B] FIG. 23B shows a schematic diagram illustrating a possible arrangement of the sub-regions used according to different alternatives. [Figure 23C]FIG. 23C shows a schematic diagram illustrating a possible arrangement of sub-regions used according to different alternatives. [Figure 23D] FIG. 23D shows a schematic diagram illustrating a possible arrangement of sub-regions used according to different alternatives. [Figure 23E] FIG. 23E shows a schematic diagram illustrating a possible arrangement of sub-regions used according to different alternatives. [Figure 24] FIG. 24 shows a schematic diagram illustrating a video decoder configured to receive a video data stream having sub-region specific random access images interspersed therein according to an embodiment. [Diagram 25] Figure 25 shows a schematic diagram illustrating the situation of Figure 24, but illustrates an alternative mode of operation of the video decoder in that, when the video decoder randomly accesses the video data stream, the video decoder waits until the image region of the input video data stream is completely covered by the sub-region of the sub-region specific random access image before outputting or presenting the video. [Figure 26] FIG. 26 shows a schematic diagram illustrating a network device receiving a video data stream including sub-region specific random access images, the sub-regions forming the sub-regions simultaneously with respect to which the video data stream can be reduced. [Figure 27] FIG. 27 shows a schematic diagram illustrating a network device receiving a data stream that is provided with displacement information and that can be reduced, to illustrate how the network device 231 can potentially provide sub-region specific displacement information to a reduced video data stream. [Figure 28] FIG. 28 shows an example of an isolated region of a sub-region of interest of an image, illustratively a cylindrical panorama. [Figure 29] FIG. 29 shows a syntax table of the TMCTS SEI message of HEVC. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present description relates to the above-identified aspects of the present application. To provide background for the first aspect of the present application, which relates to sub-region extraction / reduction of a video data stream, an example of an application in which such a need may arise and the challenges of satisfying this need are described, and the overcoming of the same are motivated as follows, taking HEVC as an example.

[0012] A spatial subset, i.e., a set of tiles, can be signaled in HEVC using a Temporal Motion Constrained Tile Set (TMCTS) SEI message. A tile set defined in such a message has the following characteristics: "The inter prediction process is constrained such that there are no sample values ​​outside of each identified tile set, and no sample values ​​at fractional sample positions derived using one or more sample values ​​outside the identified tile set are used for inter prediction of any sample within the identified tile set." In other words, samples of a TMCTS can be decoded independently of samples not associated with the same TMCTS of the same layer. A TMCTS encompasses one or more rectangular unions of one or more tiles as shown in Figure A using rectangle 900. In the figure, the region of interest 900 seen by the user encompasses two discontinuous image patches.

[0013] The detailed syntax of the TMCTS SEI message is shown in Figure B for reference.

[0014] There are many applications where it is beneficial to create an independently decodable rectangular spatial subset of a video bitstream, i.e. a Region of Interest (RoI), without the processing overhead of video transcoding. These applications include, but are not limited to:

[0015] Panoramic Video Streaming: Only the unique spatial region of the wide-angle video, e.g., 360° viewing angle, is presented to the end user via a head-mounted display. Aspect Ratio Adjustment Streaming: The aspect ratio of the encoded video is adjusted live at the server according to the client's display characteristics. · Decoding complexity adjustment: Low cost / low tech devices that cannot decode a given encoded video bitstream due to level limitations may accommodate a spatial subset of the video.

[0016] In considering the state of the art described thus far for the above list of exemplary applications, a number of issues arise.

[0017] · There is no means to make HRD parameters, i.e. buffering / timing information, of a spatial subset of the video bitstream available to the system layer. · No conformance points exist in the video to easily convert a spatial subset of a given video bitstream into a conforming video bitstream. · There is no way for an encoder to convey a guarantee that a tile set with a given identifier can be easily converted into a conforming video bitstream.

[0018] Given solutions to the problems listed above, all of the above example applications can be realized in a standards-compliant manner. Defining this capability within the video coding layer is expected to be a key fit point for the application and system layers.

[0019] The HEVC specification already includes a process for the extraction of sub-bitstreams that may reduce the temporal resolution or amount of layers of the encoded video bitstream, i.e. the spatial resolution, signal fidelity or number of views.

[0020] The present invention provides solutions to the identified problems, in particular: 1. Means for extracting a spatial subset, i.e. a single TMCTS-based video bitstream, from a coded video sequence via the definition of a TMCTS-based sub-image extraction process 2. Means for conveying and identifying the correct parameter set values ​​and (optionally) SEI information for the extracted sub-image video sequence. 3. Means for the encoder to convey specific sub-region extraction guarantees that allow for bitstream constraints on the video bitstream and the TMCTS

[0021] The embodiments described below overcome the problems outlined by providing the video data stream with information that is not required for the reconstruction of the video image from the payload portion of the video data stream, the information including an index of a given sub-region and a replacement index and / or replacement parameter, the importance and function of which are explained in more detail below. The following description is not limited to HEVC or HEVC improvements only. Rather, the embodiments described next can be implemented in any video codec technology to provide such video coding technology with additional adaptation points to provide reduced sub-region specific video data streams. Details are then presented of how the embodiments described next can be specifically implemented to form an extension of HEVC.

[0022] 1 shows a video data stream 10 according to one embodiment of the present application, i.e. the video data stream can be reduced in a conformation-preserving manner into a reduced video data stream, whose images simply represent certain sub-regions of images 12 of a video 14 that are encoded into the video data stream 10 without the need for transcoding, or more precisely, for time-consuming and computationally complex operations such as requantization, spatial to spectral conversion and vice versa and / or re-performing motion estimation.

[0023] Video data stream 10 in Figure 1 is shown to include a parameter setting portion 16 that indicates encoding parameter settings 80, and a payload portion 18 in which images 12 of video 14 are encoded. In Figure 1, portions 16 and 18 are illustratively distinguished from one another by using hatching for payload portion 18 and indicating that parameter setting portion 16 is not hatched. Furthermore, portions 16 and 18 are illustrated as being interleaved with one another within data stream 10, although this is not necessarily the case.

[0024] The payload portion 18 encodes the images 12 of the video 14 in a special way. In particular, FIG. 1 shows an exemplary predefined sub-region 22 for which the video data stream 10 should have the capability to be reducible to a reduced video data stream. The payload portion 18 has an image 12 encoded therein such that, as far as the predefined sub-region 22 is concerned, any coding dependency is restricted not to cross the boundaries of the sub-region 22. That is, an image 12 is encoded in the payload portion 18 such that the coding in a sub-region 22 is not dependent on the spatial neighborhood of such region 22 in this image. If an image 12 is encoded in the payload portion 18 using temporal prediction, the temporal prediction may be restricted within the sub-region 22 that is coded such that any part within the sub-region 22 of a first image of the video 14 is dependent on regions of references (other images) of the video 14 outside the sub-region 22. That is, a corresponding encoder generating the video data stream 14 restricts the set of available motion vectors for encoding the sub-region 22 in such a way that they do not point to parts of the reference image, so that the formation of the motion compensated prediction signal requires or involves samples outside the reference image sub-region 22. It is noted that as far as spatial dependencies are concerned, the spatial dependency restrictions may concern sample-by-sample spatial prediction, for example by continuing the arithmetic coding spatially across the boundaries of the sub-region 22, spatial prediction of the coding parameters and spatial prediction regarding the coding dependencies.

[0025] Thus, the payload portion 18 encodes the contoured image 12 according to restricting the coding dependencies so as not to reach parts outside the predetermined sub-region 22, and thus may include a syntactically ordered sequence 24 of syntax elements including, for example, motion vectors, image reference indices, partition information, coding modes, residual sample values ​​indicative of transform coefficients or quantized prediction residuals, or one or any combination of these. Most importantly, however, the payload portion 18 has the image 12 of the video 14 encoded therein in a parameterized manner using a first set 20a of coding parameter settings 20. For example, the coding parameter settings in the set 20a may specify, for example, an image size of the image 12, e.g., the vertical height and horizontal width of the image 12. To explain how the image size "parameterizes" the coding of the image 12 into the payload portion 18, reference is made briefly to FIG. 2. FIG. 2 illustrates an image size coding parameter 26 as an example of one of the coding parameter settings of the set 20a. Obviously, the image size 26 indicates the size of the image area that has to be "encoded" by the payload portion 18, for example by filling with a predetermined sample value such as zero, which may correspond to the color black, by indicating that each sub-block of an image 12 remains unencoded. Thus, the image size 26 influences 28 the amount or size 30 of the syntax description 24 of the payload portion 18. Furthermore, the image size 26 influences 28 the position indications 32 in the syntax description 24 of the payload portion 18, for example with respect to the value range of the position indications 32 and the order in which the position indications 32 appear in the syntax description 24. For example, the position indications 32 may include slice addresses in the payload portion 18. The slices 34 are portions of the data stream 10 in units in which the data stream 10 can be transmitted to a decoder, as shown in FIG. 1. Each image 12 may be encoded in the data stream 10 in units of such slices 34, and is subdivided into slices 34 according to the decoding order in which the images 12 are encoded in the data stream 10.Each slice 34 corresponds to and is encoded within a corresponding region 36 of image 12, but the region 36 is either inside or outside of sub-region 22, i.e. does not cross a sub-region boundary. In such a case, each slice 34 may be given a slice address that indicates the location of the corresponding region 36 within the image region of image 12, i.e. relative to the periphery of image 12. To give a specific example, the slice address may be measured relative to the top left corner of image 12. Obviously, such slice address should not exceed a value that exceeds the value of the slice address in an image of image size 26.

[0026] Similar to the image size 26, the set of encoding parameter settings 20a may also define a tile structure 38 into which the image 12 may be subdivided. Using dashed-dotted lines 40, FIG. 1 illustrates an example of a subdivision of the image 12 into tiles 42, such that the tiles are arranged in a tile array of columns and rows. In any case where the image 12 is encoded into the payload portion 18 using a tile subdivision into tiles 42, this may be due to, for example, 1) spatial interdependencies across tile boundaries are not permitted and therefore not used, and 2) the decoding order in which the image 12 is encoded into the data stream 10 traverses the image 12 in a raster-scan tile order, i.e., each tile is traversed before visiting the next tile in the tile order. Thus, the tile structure 38 influences the decoding order 44 in which the image 12 is encoded into the payload portion 18 and accordingly influences the syntax description 24. In a similar manner to the image size 26, the tile structure 38 also has an influence 28 on the position indicators 32 within the payload portion 18, i.e., the order in which different instantiations of the position indicators 32 are allowed to occur within the syntax description 24.

[0027] The encoding parameter settings of the set 20a may also include buffer timing 46. The buffer timing 46 may indicate, for example, the encoded image buffer removal time. Certain portions of the data stream 10, such as individual slices 34 or portions of the data stream 10 that refer to one image 12, should be removed from the encoded image buffer of the decoder, and these temporal values ​​affect or relate to the size 28 of the corresponding portions in the data stream 10, and the buffer timing 46 also affects 28 the amount / size 30 of the payload portions 18.

[0028] That is, as illustrated in the description of FIG. 2, the encoding of the image 12 into the payload portion 18 is "parameterized" or "described" using the set 20a of encoding parameter settings 20, in the sense that any differences between the set 20a of encoding parameter settings 20, on the one hand, and the payload portion 18 and its syntax description 24, on the other hand, are identified as inconsistent with the conformance requirements required to comply with the data stream identified as conforming, and are therefore identified as conforming.

[0029] The first set of encoding parameter settings 20a is included in the payload portion 18 and is referenced or indexed by an index 48 interspersed or contained within the syntax description 24. For example, the index 48 may be included in a slice header of the slice 34.

[0030] The indexed set of encoding parameter settings 20a is cancelled simultaneously or together with the payload portion 18, parts of the payload portion 18 that do not belong to the sub-region 22, and the resulting reduced data stream is modified to maintain conformance. This approach is not followed by the embodiment of Fig. 1. Although such a correlation modification of both the encoding parameter settings in the indexed set 20a on the one hand and the payload portion 18 on the other hand does not require a full decoding and encoding detour, the computational overhead to perform this correlation modification would nevertheless require a significant amount of analysis steps and the like.

[0031] Thus, the embodiment of FIG. 1 follows another approach in which the video data stream 10 includes, i.e. is provided with, information 50 that is not required to reconstruct the image 12 of the video from the payload portion 18. The information may include an indication of the predefined sub-region and a replacement index and / or replacement parameters. For example, the information 50 may indicate the predefined sub-region 22 with respect to its position within the image 12. The information 50 may indicate the position of the sub-region 22, for example, in tiles. Thus, the information 50 may identify a set of tiles 42 within each image to form the sub-region 22. The set of tiles 42 within each image 12 may be fixed between images 12. That is, the tiles formed within each image 12, the sub-regions 22 may be aligned with each other and the tile boundaries of these tiles forming the sub-regions 22 may spatially occupy between different images 12. It should be mentioned that the set of tiles is not limited to form a contiguous rectangular tile sub-array of the image 12. However, there may be an overlay-free and gap-free abutment of tiles in each of these images 12 forming the sub-region 22, this gapless and overlay-free abutment or juxtaposition forming a rectangular region. However, it will be appreciated that the display 50 is not limited to showing the sub-regions 22 in tiles. It should be recalled that in any case, the use of tile subdivisions of the images 12 is merely optional. The display 50 may, for example, show the sub-regions 22 in units of samples or by some other means. In yet another embodiment, the location of the sub-regions 22 may even form default information known to the participating network devices and decoders supposed to handle the video data stream 10, the information 50 simply indicating the reduction possibilities related to or present in the sub-regions 22. As already mentioned above and as shown in FIG. 3, the information 50 includes, besides the indication 52 of the given sub-region, a replacement index 54 and / or a replacement parameter 56.The replacement index and / or replacement parameter is for modifying the indexed set of encoding parameter settings, i.e. the set of encoding parameter settings indexed by an index within the payload portion 18, so that the indexed set of encoding parameter settings fits the payload portion of the reduced video data stream in which the payload portion 18 is modified on the one hand by the removal of those parts that relate to parts of the image 12 outside the sub-region 22 and modifying the position indication 32 to relate to the periphery of the sub-region 22 rather than to the periphery of the image 12.

[0032] To clarify the latter situation, reference is made to Fig. 4 which shows a network device 60 which, according to Fig. 1, is arranged to receive and process the video data stream 10 and derive therefrom a reduced video data stream 62. The term "reduced" in the "reduced video data stream" 62 denotes two things: firstly, the fact that the reduced video data stream 62 corresponds to a lower bit rate compared to the video data stream 10, and secondly, the images in which the reduced video data stream 62 is encoded are smaller than the images 12 of the video data stream 10, in that the smaller images of the reduced video data stream 62 simply represent sub-regions 22 of the images 12.

[0033] As will be described in more detail below, to perform that task, the method apparatus 60 includes a reader 64 configured to read information 50 from the data stream 10, and a reducer 66 that performs a reduction or expansion process based on the information 50, as will be described in more detail below.

[0034] Fig. 5 illustrates the functionality of the network device 60 in the exemplary case of using a replacement index 54 in the information 50. In particular, as shown in Fig. 5, the network device 60 uses the information 50 to remove 68 from the payload portion 18 references to portions 70 of the data stream 10 that are not related to the sub-region 22, i.e., regions of the image 12 outside the sub-region 22. The removal 68 can be performed, for example, on a slice basis, with a reducer 66 indicating slices 34 in the payload portion 18 that are not related to the sub-region 22 based on a position indication or slice address in the slice header of the slice 34 on the one hand and an indication 52 in the information 50 on the other hand.

[0035] In the example of Fig. 5, where the information 50 has a replacement index 54, the parameter setting portion 16 of the video data stream 10 has, besides the indexed set of coding parameter settings 20a, a non-indexed set of coding parameter settings 20b that is not referenced or indexed by an index 48 in the payload portion 18. When performing the reduction, the reducer 66 replaces the index 48 in the data stream 10 with a replacement index 54 according to a replacement shown in Fig. 5 with a curve 72, exemplarily shown in Fig. 5. By replacing the index 48 with the replacement index 54, a redirection 72 occurs according to which the index included in the payload portion of the reduced video data stream 62 references or indexes the second set of coding parameter settings 20b, so that the first set of coding parameter settings 20a is no longer indexed. The redirection 72 can therefore include the reducer 66 removing 74 the no longer indexed set of coding parameter settings 20a from the parameter setting portion 16.

[0036] The reducer 66 also modifies the position indications 32 within the payload portion 18 as measured relative to the perimeter of the predetermined sub-region 22. This modification is illustrated in Figure 5 with the change in merely one illustratively represented position indication 32 from data stream 10 to reduced video data stream 62 being shown diagrammatically by showing the position indications 32 in reduced video data stream 62 in a cross-hatched manner whilst showing the position indications 32 in data stream 10 without hatching by curved arrows 78.

[0037] Thus, to summarise the description of Fig. 5, the network device 60 is able to obtain the reduced video data stream 62 in a manner with relatively low complexity. The laborious task of correctly adapting the set of encoding parameter settings 20b to correctly parameterise or adapt the amount / size 30, position indication 32 and decoding order 44 of the payload portions 18 of the reduced video data stream 62 can be performed elsewhere, such as in the encoder 80, representatively shown with the dashed box in Fig. 1. An alternative is to change the order between the evaluation of the information 50 and the reduction by the reducer 66, as will be further explained below.

[0038] Figure 6 shows a situation in which the reduced video data stream 62 is provided to a video decoder 82 to show that the reduced video data stream 62 is encoded into a video 84 of a smaller image 86, i.e. an image 86 of a smaller size compared to image 12 and merely showing a sub-region 22 thereof. The reconstruction of the video 84 thus occurs by the video decoder 82 decoding the reduced video data stream 62. As explained with reference to Figure 5, the reduced video data stream 62 comprises a reduced payload portion 18 that is encoded into the smaller image 86 as parameterized by a second set of encoding parameter settings 20b or as correspondingly described.

[0039] The video encoder 80 may, for example, encode the image 12 into the video data stream 10, according to the encoding constraints described above with respect to Fig. 1 in relation to the sub-region 22. The encoder 80 may, for example, perform this encoding using an optimization of a suitable rate-distortion optimization function. As a result of this encoding, the payload portion 18 indexes the set 20a. Furthermore, the encoder 80 generates the set 20b. For this purpose, the encoder 80 may, for example, adapt the image size 26 and the tile structure 38 from their values ​​in the set 20a to correspond to the size of the sub-region 22 and the occupied tile set. Beyond that, the encoder 80 performs a reduction process substantially as described above with respect to Fig. 5 itself, and calculates the buffer timings 46 so that a decoder, such as the video decoder 82, can correctly manage the encoded image buffer using the thus calculated buffer timings 46 in the second set 20b of encoding parameter settings.

[0040] Fig. 7 shows an alternative method of the operation mode of the network device, namely when using replacement parameters 56 in the information 50. According to this alternative, as shown in Fig. 7, the parameter setting portion 16 simply includes the indexed set of encoding parameter settings 20a, and the reindexing or redirection 72 and the set remover 74 do not need to be performed by the reducer 66. However, instead, the reducer 66 uses the replacement parameters 56 obtained from the information 50 to adjust 88 the indexed set of encoding parameter settings 20a to become the set of encoding parameter settings 20b. According to this alternative, the reducer 66 performing steps 68, 78 and 88 to derive the reduced video data stream 62 from the original video data stream 10 is a relatively uncomplicated operation.

[0041] In other words, in the case of FIG. 7, the substitution parameters 56 may include, for example, one or more of the image size 26, the tile structure 38 and / or the buffer timing 46.

[0042] It should be noted with respect to Figures 5 and 7 that there may also be a mixture of both alternatives with information 50 including both substitution indexes and substitution parameters. For example, the coding parameter settings subject to change from set 20a to 20b may be distributed over or composed of different parameter set slices such as SPS, PPS, VPS, etc. Thus, for different ones of these slices, different processing, for example according to Figure 5 or Figure 7, may be performed.

[0043] Concerning the task of modifying 78 the position indication, it should be noted that this task must be performed relatively frequently, for example for each payload slice of the slices 34 in the payload portion 18, but the calculation of the new replacement value of the position indication 32 is relatively uncomplicated. For example, the position indication may indicate a position according to horizontal and vertical coordinates, and the modification 78 may calculate the new coordinates of the position indication by forming, for example, a subtraction between the corresponding coordinates of the original position indication 32 and the offset of the sub-region 22 relative to the top left corner of the data stream 10 and the image 12. Alternatively, the position indication 32 may indicate a position in some suitable units, for example in units of coding blocks such as tree root blocks in which the image 12 is regularly divided into rows and columns, for example using some linear scale following the above-mentioned decoding order. In such a case, the position indication is newly calculated in step 78 taking into account only the coding order of these code blocks in the sub-region 22. In this regard, it should also be noted that the reduction / extraction process just outlined for forming the reduced video data stream 62 from the video data stream 10 is also suitable for forming the reduced video data stream 62 in such a manner that the smaller images 86 of the video 84 encoded in the reduced video data stream 62 represent the sections 22 in a spatially stitched manner, and the same image content of the images 84 may be positioned within the images 12 of the sub-regions 22 in a spatially different manner.

[0044] With regard to Fig. 6, it should be noted that the video decoder 82 shown in Fig. 6 may or may not be capable of decoding the video data stream 10 to reconstruct the images 12 of the video 14. A reason why the video decoder 82 cannot decode the video data stream 10 may be that, for example, the profile level of the video decoder 82 is sufficient to handle the size and complexity of the reduced video data stream 62, but insufficient to decode the original video data stream 10. However, in principle, both data streams 62 and 10 can be fitted into one video codec by the above-mentioned appropriate adaptation of the indexed sets of coding parameter settings by re-indexing and / or parameter adjustment.

[0045] After describing a fairly general embodiment for video stream reduction / extraction with respect to a particular sub-region of an image of a video data stream to be reduced, the above motivation and problem description for such extraction with respect to HEVC is resumed below and specific examples for implementing the above-described embodiment are provided.

[0046] 1. Signaling aspects for single layer subregions

[0047] 1.1. Parameter set: The following aspects of the parameter set require adjustment when extracting spatial subsets: VPS: No normative information on single-layer coding SPS: Level Information Image size Cut or fit window information Buffering and timing information (i.e. HRD information) Potentially adding more VUI (Video Usability Information) items such as motion_vectors_over_pic_boundaries_flag, min_spatial_segmentation_idc, etc. PPS: Spatial segmentation information, i.e. tile information regarding the amount and size of tiles in horizontal and vertical directions

[0048] Signaling Embodiments Signaling 1A: For each TMCTS, the encoder can send additional unused (i.e. not activated at all) VPS, SPS and PPS in-band (i.e. as respective NAL units) and provide their mapping to the TMCTS in a Supplemental Enhancement Information (SEI) message.

[0049] Example syntax / semantics for signaling 1A SEI is shown in FIG.

[0050] Syntax element 90 is optional as it can be derived from an image parameter set identifier.

[0051] The semantics are as follows: num_extraction_information_sets_minus1 indicates the number of information sets contained in a given Signaling1A SEI that should be applied in the sub-image extraction process.

[0052] num_applicable_tile_set_identifiers_minus1 indicates the number of mcts_id values ​​of tile sets for which the following i-th information set applies to the sub-image extraction process.

[0053] mcts_identifier[i][k] indicates the value of mcts_id of the tile set for which all num_applicable_tile_set_identifers_minus1 apply the next i-th information set to the sub-image extraction process + 1.

[0054] num_mcts_pps_replacements[i] indicates the number of pps identifier replacements to signal in the Signaling1A SEI for the tile set with mcts_id equal to mcts_id_map[i].

[0055] mcts_vps_id[i] indicates that the mcts_vps_idx[i]th video parameter set is used in the sub-image extraction process for the tile set with mcts_id equal to mcts_id_map[i].

[0056] mcts_sps_id[i] indicates that the mcts_sps_idx[i]th sequence parameter set is used in the sub-image extraction process for the tile set with mcts_id equal to mcts_id_map[i].

[0057] mcts_pps_id_in[i][j] indicates the jth value of the num_mcts_pps_replacements[i] pps identifier in the slice header syntax structure of the tile set with mcts_id equal to mcts_id_map[i] that should be replaced in the sub-image extraction process.

[0058] mcts_pps_id_out[i][j] indicates the jth value of the num_mcts_pps_replacements pps identifier in the slice header syntax structure of the tile set having mcts_id equal to mcts_id_map[i] to replace the pps identifier equal to the value mcts_pps_id_in[i][j] in the sub-image extraction process.

[0059] Signaling 1B: The encoder can send VPS, SPS, and PPS for each TMCTS, and can map to all TMCTSs contained in the container-style SEI.

[0060] Example syntax / semantics for signaling 1B SEI is shown in FIG.

[0061] The yellow syntax element 92 is optional as it can be derived from the image parameter set identifier.

[0062] The semantics are as follows: num_vps_in_message_minus1 indicates the number of vps syntax structures in a given Signaling1B SEI that should be used in the sub-image extraction process.

[0063] num_sps_in_message_minus1 indicates the number of sps syntax structures in a given Signaling1B SEI that should be used in the sub-image extraction process.

[0064] num_pps_in_message_minus1 indicates the number of pps syntax structures in a given Signaling1B SEI that should be used in the sub-image extraction process.

[0065] num_extraction_information_sets_minus1 indicates the number of information sets contained in a given Signaling1B SEI that should be applied in the sub-image extraction process.

[0066] num_applicable_tile_set_identifiers_minus1 indicates the number of mcts_id values ​​of tile sets to which the i-th information set that follows applies for the sub-image extraction process.

[0067] mcts_identifier[i][k] represents all num_applicable_tile_set_identifers_minus1 plus one mcts_id value of the tile set for which the next i-th information set applies to the sub-image extraction process.

[0068] mcts_vps_idx[i] indicates that for the tile set with mcts_id equal to mcts_id_map[i] in the sub-image extraction process, the mcts_vps_idx[i]-th video parameter set signaled in the Signaling1B SEI is used.

[0069] mcts_sps_idx[i] indicates that the mcts_sps_idx[i]th sequence parameter set signaled in the Signaling1B SEI in the sub-image extraction process is used for the tile set with mcts_id equal to mcts_id_map[i].

[0070] num_mcts_pps_replacements[i] indicates the number of pps identifier replacements to signal in the Signaling1B SEI for the tile set with mcts_id equal to mcts_id_map[i].

[0071] mcts_pps_id_in[i][j] indicates the jth value of the num_mcts_pps_replacements[i] pps identifier in the slice header syntax structure of the tile set with mcts_id equal to mcts_id_map[i] that should be replaced in the sub-image extraction process.

[0072] mcts_pps_idx_out[i][j] indicates that in the sub-image extraction process, the picture parameter set with a pps identifier equal to mcts_pps_id_in[i][j] should be replaced by the mcts_pps_idx_out[i][j]-th signaled picture parameter set in the Singalling1C SEI.

[0073] Signaling 1C: The encoder may provide parameter set information related to the TMCTS that cannot be derived (essentially additional buffering / timing (HRD) parameters and mapping to the applicable TMCTS in the SEI).

[0074] Example syntax / semantics for signaling 1C SEI is shown in FIG. The HRD information of the next SEI is structured such that the extraction process can replace consecutive blocks of syntax elements in the original VPS with respective consecutive blocks of syntax elements from the SEI.

[0075] num_extraction_information_sets_minus1 indicates the number of information sets contained in a given signaling 1C SEI that are applied in the sub-image extraction process. num_applicable_tile_set_identifiers_minus1 indicates the number of mcts_id values ​​of tile sets to which the i-th information set following applies for the sub-image extraction process.

[0076] mcts_identifier[i][k] denotes all num_applicable_tile_set_identifers_minus1 plus one of the mcts_id values ​​of the tilesets for which the next i-th set of information applies to the sub-image extraction process.

[0077] mcts_vps_timing_info_present_flag[i] equal to 1 indicates that mcts_vps_num_units_in_tick[i], mcts_vps_time_scale[i], mcts_vps_poc_proportional_to_timing_flag[i] and mcts_vps_num_hrd_parameters[i] are present in the VPS. mcts_vps_timing_info_present_flag[i] equal to 0 indicates that mcts_vps_num_units_in_tick[i], mcts_vps_time_scale[i], mcts_vps_poc_proportional_to_timing_flag[i] and mcts_vps_num_hrd_parameters[i] are not present in the Signaling1C SEI.

[0078] mcts_vps_num_units_in_tick[i] is the number of i-th time units of a clock running at a frequency mcts_vps_time_scale Hz, which corresponds to one increment (called a clock tick) of the clock tick counter. The value of mcts_vps_num_units_in_tick[i] must be greater than 0. A clock tick (in seconds) is equal to mcts_vps_num_units_in_tick divided by mcts_vps_time_scale. For example, if the picture rate of a video signal is 25 Hz, mcts_vps_time_scale is equal to 27000000 and mcts_vps_num_units_in_tick is equal to 1,080,000, so a clock tick may be 0.04 seconds.

[0079] mcts_vps_time_scale[i] is the i-th number of time units passing in a second. For example, a time coordinate system that measures time using a 27 MHz clock has a vps_time_scale of 27000000. The value of vps_time_scale must be greater than 0.

[0080] mcts_vps_poc_proportional_to_timing_flag[i] equal to 1 indicates that the image order count value of each image in the CVS that is not the first image in the CVS, in decoding order, is proportional to the image output time relative to the output time of the first image in the CVS. mcts_vps_poc_proportional_to_timing_flag[i] equal to 0 indicates that the image order count value of each image in the CVS that is not the first image in the CVS, in decoding order, may or may not be proportional to the image output time relative to the output time of the first image in the CVS.

[0081] mcts_vps_num_ticks_poc_diff_one_minus1[i] plus 1 specifies the number of clock ticks that correspond to a difference in picture sequence count values ​​equal to 1. The value of mcts_vps_num_ticks_poc_diff_one_minus1[i] must be in the range of 0 to 2-2.

[0082] mcts_vps_num_hrd_parameters[i] specifies the number of hrd_parameters() syntax structures present in the i-th entry of the Signaling1C SEI. The value of mcts_vps_num_hrd_parameters must be in the range of 0 to vps_num_layer_sets_minus1 + 1.

[0083] mcts_hrd_layer_set_idx[i][j] specifies an index into the list of layer sets specified by the VPS of the ith entry in the Signaling1C SEI of layer sets to which the jth hrd_parameters() syntax structure in the Signaling1C SEI applies and which are used in the sub-image extraction process. The value of mcts_hrd_layer_set_idx[i][j] shall be in the range of (vps_base_layer_internal_flag?0:1) to vps_num_layer_sets_minus1. It is a bitstream conformance requirement that the value of mcts_hrd_layer_set_idx[i][j] shall not be equal to the value of hrd_layer_set_idx[i][k] for any value of j not equal to k.

[0084] mcts_cprms_present_flag[i][j] equal to 1 specifies that the HRD parameters common to all sublayers are present in the jth hrd_parameters() syntax structure of the ith entry of the Signaling1C SEI. mcts_cprms_present_flag[i][j] equal to 0 specifies that the HRD parameters common to all sublayers are not present in the ith hrd_parameters() syntax structure of the ith entry of the Signaling1C SEI and are derived to be the same as the (i-1)th hrd_parameters() syntax structure of the ith entry of the Signaling1C SEI. mcts_cprms_present_flag[i][0] is inferred to be equal to 1.

[0085] Since the above HRD information is VPS related, signaling of similar information for SPS VUI HRD parameters can be embodied in the same way, for example by extending the above SEI or as a separate SEI message.

[0086] It is worth noting that further embodiments of the present invention may use the mechanisms implemented by signaling 1A, 1B and 1C as an extension of other bitstream syntax structures or parameter sets, such as VUI.

[0087] 1.2. SEI Message The occurrence of any of the following SEI messages in the original video bitstream may require a reconciliation mechanism to avoid inconsistencies after TMCTS extraction: HRD related buffering period, picture timing and decode unit information SEI Pan Scan SEI * FramePackingArrangement * SEI · DecodedPictureHash SEI (Decoded image hash SEI) ·TMCTS SEI

[0088] Signaling embodiment: Signaling 2A: The encoder may provide appropriate permutations of said SEIs in relation to the TMCTS within a container-style SEI for all TMCTSs. Such signaling may be combined with the embodiment of Signaling 1C and is shown in FIG. 11. In other words, in addition to or instead of the above description, a video data stream 10 representing a video 14 may include a payload portion 18 in which an image 12 of the video is encoded, and an auxiliary enhancement information message indicating auxiliary enhancement information corresponding to the payload portion 18, or more precisely, the manner in which the image 12 of the video is encoded in the payload portion 18, and further includes a replacement auxiliary enhancement information message for replacing the information 50 including a representation 52 of a predefined sub-area 22 of the image 12 and the auxiliary enhancement information message, the replacement auxiliary enhancement information message selecting to remove 68 a portion 70 of the payload portion 18 and to modify 78 a position indication 32 in the payload portion 18 with reference to an area of ​​the image 12 outside the predefined sub-area 22 and a reduced video data stream 62 modified compared to the video data stream, to indicate a position in a manner measured from the periphery of said predefined sub-area 22 instead of said image 12, and having a reduced payload portion in which a sub-area specific picture 86 indicating the predefined sub-area 22 of the image is encoded, such that the replacement auxiliary enhancement information message indicates the replacement auxiliary enhancement information corresponding to the reduced payload portion, i.e., the manner in which the sub-area specific picture 86 is encoded in the reduced payload portion 18. The parameter setter 80a generates supplemental enhancement information messages that are subject to potential replacement by replacement SEI messages, in addition to or as an alternative to the parameter generation described above. The replacement is performed by the network device 60, in addition to or as an alternative to the redirection and / or adjustment described above.

[0089] all_tile_sets_flag equal to 0 specifies that the applicable_mcts_id[0] list is specified by wapplicable_mcts_id[i] for all tile sets defined in the bitstream. all_tile_sets_flag equal to 1 specifies that the list applicable_mcts_id[0] consists of all values ​​of nuh_layer_id present in the current access unit that are greater than or equal to the nuh_layer_id of the current SEI NAL unit, in ascending order of value. tile_sets_max_temporal_id_plus1 minus1 indicates the maximum temporal level that should be extracted in the sub-image extraction process for tilesets with mcts_id equal to an element of the array applicable_mcts_id[i].

[0090] num_applicable_tile_set_identifiers_minus1 plus 1 indicates the number of next applicable mcts ids for which the next SEI message should be used in the sub-image extraction process.

[0091] mcts_identifier[i] indicates all num_applicable_tile_set_identifiers_minus1 values ​​of mcts_id for which the next SEI message should be inserted when extracting the respective tileset with mcts_id equal to applicable_mcts_id[i] using the tileset sub-image extraction process.

[0092] 2. Sub-image extraction process:

[0093] The details of the extraction process obviously depend on the signalling scheme applied.

[0094] Constraints on the tile setup and TMCTS SEI, especially the extracted TMCTS, must be formulated to guarantee conforming output. The presence of any of the above signaling embodiments in the bitstream indicates a guarantee that the encoder will follow the constraints formulated below while creating the video bitstream.

[0095] input: Bitstream ·Target MCTS identifier MCTSIdTarget. · Target layer identifier list layerIdListTarget.

[0096] Constraints or bitstream requirements: tiles_enabled_flag is equal to 1. ·num_tile_columns_minus1> 0 || num_rows_minus1> 0. · A TMCTS SEI message with mcts_id[i] equal to MCTSIdTarget exists and is associated with every image that should be output. · A TMCTS with mcts_id [i] equal to MCTSIdTarget exists in the TMCTS SEI. · The appropriate level for the TMCTS with mcts_id[i] equal to MCTSIdTarget shall be indicated either by the TMCTS SEI syntax elements mcts_tier_level_idc_present_flag[i], mcts_tier_idc[i], mcts_level_idc[i] or by one of the signaling variables 1A or 1B above. · The HRD information of the TMCTS is present in the bitstream via one of the signaling variables 1A, 1B or 1C. · All rectangles in the TMCTS with mcts_id[i] equal to MCTSIdTarget have the same height or width, or both, in luma samples.

[0097] process: · Remove all tile NALUs that are not in the tileset associated with mcts_id[i] equal to MCTSIdTarget. -Replace / adjust parameter sets according to signaling 1X. Adjust the remaining NALU slice headers as follows:

[0098] ○ Adjust slice_segment_address and first_slice_segment_in_pic_flag to create a common image plane from all the rectangles in the tileset. ○ Adjust pps_id as needed. · In the presence of signaling 2A, it removes or replaces SEI.

[0099] As an alternative embodiment, the constraints or bitstream requirements described above as part of the extraction process can take the form of dedicated signaling in the bitstream, for example separate SEI messages or VUI indications, the presence of which is a prerequisite for them on the extraction process.

[0100] 2. Multi-layer

[0101] In some scenarios, there may be interest in layered codecs, for example to provide different qualities per region. It may be interesting to provide a larger spatial region at a lower layer quality, so that if required by the user and some specific region of the wide-angle video is not available at the higher layer, the content can be upsampled at the lower layer and presented together with the higher layer content. The degree to which the lower layer video region extends the higher layer video region should be allowed to vary depending on the usage situation.

[0102] In addition to the TMCTS SEI described, in the layered extension of the HEVC specification (i.e., Annex F), an ILCTS (Inter-layer Constrained Tile Sets) SEI message is specified that indicates similar constraints of nature for inter-layer prediction. For reference, a syntax table is shown in Figure 12. Therefore, as a further part of the present invention, a similar method extraction process is implemented for layered encoded video bitstreams that takes additional information into account.

[0103] The main difference with the signaling and processing disclosed above when considering the signaling aspects of a multi-layer sub-picture is that the target data portion of the bitstream is no longer identified by a single value of the mcts_id identifier, but instead the identifier of the layer set and, if applicable, multiple identifiers of the TMCTS in each containing layer and, if applicable, each ILCTS identifier between the contained layers form a multi-dimensional vector that identifies the target portion of the bitstream.

[0104] The present invention embodiments are variations of the single layer signaling schemes 1A, 1B, 1C and 2A disclosed above, in which the syntax element mcts_identifier[i][k] is replaced by the multi-dimensional identifier vector as described.

[0105] Furthermore, the encoder constraints or bitstream requirements are extended as follows:

[0106] 2.2. Extraction process:

[0107] input: A multidimensional identifier vector consisting of: · Target layer set layerSetIdTarget Target layer TMCTS identifier MCTSIdTarget_Lx for at least the highest layer in the set of layers with identifier layerSetIdTarget Target ILCTS identifier LTCTSIdTarget_Lx_refLy corresponding to Bitstream

[0108] Bitstream Request In addition to those defined for the single layer case:

[0109] A TMCTS SEI message with mcts_id[i] equal to MCTSIdTarget_Lx exists for each layer Lx, and an ILCTS SEI message with ilcts_id[i] equal to ILTCTSIdTarget_Lx_refLy exists for each layer Lx for any used reference layer Ly included in the bitstream and layer set layerSetIdTarget.

[0110] In order to exclude the presence of reference samples that are missing in the extracted bitstream portion, the TMCTS and ILCTS that define the bitstream portion must furthermore satisfy the following constraints:

[0111] For each reference layer A with tileset tsA associated with mcts_id[i] equal to MCTSIdTarget_LA: The tiles in layer A that make up tsA are the same tiles associated with the tileset with ilcts_id[i] equal to ILTCTSIdTarget_LA_refLy. · For each referenced layer B with tile set tsB associated with mcts_id[i] equal to MCTSIdTarget_LB: The tiles of layer B that make up tsB are entirely contained in the associated reference tile set, denoted ilcts_id[i] ILCTSIdTarget_Lx_refLB.

[0112] process For each layer x: mcts id [i] remove all tile NAL units that are not in the tile set with identifier MCTS Id Target_Lx.

[0113] Before moving on to the next aspect of the present application, a brief note must be made regarding the above-mentioned possibility that the sub-region 22 may be composed of a set of tiles, the relative position of which within the image 12 may differ from the relative position of these tiles to each other within the smaller image 86 of the video 84 represented by the reduced video data stream 62.

[0114] Fig. 13 shows an image 12 subdivided into an array of tiles 42, listed with capital letters A to I along the decoding order. For the sake of illustration only, Fig. 13 exemplarily shows one image and its subdivision into 3x3 tiles 42. Assume that the image 12 is encoded into the video data stream 10 such that the coding dependencies do not cross tile boundaries 40, and that the coding dependencies include intra-picture spatial interdependencies but are also restricted, such as e.g. temporal independence. Thus, tiles 42 in the current image only depend on themselves or on co-located tiles in previously coded / decoded images, i.e. temporal reference images.

[0115] In this situation, the sub-region 22 may be composed of a set of non-contiguous tiles 42, such as the set of tiles [D, F, G, I]. Due to their mutual independence, an image 86 of a video 84 may show a sub-region 22 such that the tiles involved are spatially arranged in the image 86 in a different way. This is illustrated in FIG. 13: the image 86 is also encoded in the reduced video data stream 62 in units of tiles 42, but the tiles forming the sub-region 22 of the image 12 are spatially arranged in the image 86 in a different positional relationship to each other. In the example of FIG. 13, the tiles 42 forming the sub-region 22 occupy tiles on opposite sides of the image 12, as if the image 12 shows a horizontal panoramic view, and these opposing side tiles actually show adjacent parts of the panoramic scene. However, in the image 86, the tiles in each tile row switch positions with respect to their relative positions in the image 12. That is, for example, tile F appears to the left with respect to tile D, compared to the mutual horizontal positions of tiles D and F in the image 12.

[0116] Before proceeding to the next aspect of the present application, it should be noted that neither the tiles 42 nor the sections 22 need to be coded into the image 12 in the manner described above where coding dependencies are limited so as not to cross their boundaries. Of course, this restriction relaxes the above-described concept of reducing / extracting the video data stream, but such coding dependencies tend to only affect small edge portions along the boundaries of the sub-regions 22 / tiles 42, depending on the application where distortion at these edge portions is acceptable.

[0117] Moreover, it should be noted that the above-described embodiment shows the possibility of extending current video codecs to newly include the described compliance points, i.e. to reduce the video stream to a reduced video stream related to a sub-region 22 of the original image 12 while only maintaining conformance, and to this end, the information 50 is exemplarily hidden typically in SEI messages, VUI or parameter set extensions, i.e. in parts of the original video data stream that can be skipped by the decoder according to likes and dislikes. However, instead, the information 50 can be conveyed in the video data stream in parts that are standard parts. That is, the new video codec can be set from the beginning to include the described compliance points.

[0118] Furthermore, for completeness, a further concrete example of the above-mentioned embodiment is described, which shows the possibility of extending the HEVC standard to implement the above-mentioned embodiment. For this purpose, a new SEI message is provided. In other words, a modification to the HEVC specification is described that allows for the extraction of a motion constrained tile set (MCTS) as an individual HEVC compliant bitstream. Two SEI messages are used and are described below.

[0119] The first SEI message, i.e., the MCTS Extraction Information Set SEI message, provides the syntax for the carriage of MCTS-specific replacement parameter sets and defines the extraction process in semantics. The second SEI message, i.e., the MCTS Extraction Information Nest SEI message, provides the syntax for MCTS-specific nested SEI messages.

[0120] Therefore, to include these SEI messages in the HEVC framework, the HEVC general SEI message syntax is modified to include the new types of SEI messages.

[0121] JPEG2025028895000002.jpg100165

[0122] Thus, the list SingleLayerSeiList is set to include payload type values ​​3, 6, 9, 15, 16, 17, 19, 22, 23, 45, 47, 56, 128, 129, 131, 132 and 134 through 153, inclusive. Similarly, the lists VclAssociatedSeiList and PicUnitRepConSeiList are extended with new SEI message type numbers 152 and 153, which of course have been chosen for illustrative purposes only.

[0123] Table D.1 of HEVC further includes a hint to the new tape in the SEI message lifetime.

[0124] JPEG2025028895000003.jpg31159

[0125] Its syntax is as follows: The MCTS Extraction Information Set SEI message syntax can be designed as follows.

[0126] JPEG2025028895000004.jpg230166

[0127] As far as semantics are concerned, the MCTS Extraction Information Set SEI message is an example of information 50 that uses substitution parameters 56 . The MCTS extraction information set SEI message provides auxiliary information to perform sub-bitstream MCTS extraction as specified below and derive an HEVC compliant bitstream from a motion constrained tile set, i.e., a set of tiles forming a fragment 84 of the whole image region. This information consists of a number of extraction information sets, each of which contains an identifier of the motion constrained tile set to which the extraction information set applies. Each extraction information set contains the RBSP bytes of the replacement video parameter set, sequence parameter set, and picture parameter set used during the sub-bitstream MCTS extraction process. Let the set of pictures associatedPicSet be the pictures from the access unit that contains the MCTS Extraction Information Set SEI message (inclusive), including the first of the following, in decoding order, up to but not including the first: - The next access unit in decoding order that contains an MCTS Extraction Information Set SEI message. - Next IRAP picture in decoding order with NoRaslOutputFlag equal to 1. - The next IRAP access unit in decoding order for which NoClrasOutputFlag is equal to 1. The scope of the MCTS Extraction Information Set SEI message is the set of images associatedPicSet. If an MCT extraction information set tile set SEI message is present for any image in associatedPicSet, then a temporally motion constrained tile set SEI message shall be present for the first image in associatedPicSet, in decoding order, and MAY be present for other images in associatedPicSet. The temporally motion constrained tile set SEI message MUST have mcts_id[] equal to mcts_identifer[] for all images in associatedPicSet. If an MCTS Extraction Information Set Tile Set SEI message is present for any image in the associatedPicSet, then the MCTS Extraction Information Set SEI message MUST be present for the first image in the associatedPicSet in decoding order, and MAY be present for other images in the associatedPicSet. For any PPS that is active for any image in the associatedPicSet, if tiles_enabled_flag is equal to 0, then the MCTS Extraction Information Set SEI message shall not be present for any image in the associatedPicSet. The MCTS Extraction Information Set SEI message shall not be present for any image in the associatedPicSet unless all PPSs that are active for any image in the associatedPicSet have the same values ​​of syntax elements num_tile_columns_minus1, num_tile_rows_minus1, uniform_spacing_flag, column_width_minus1[i] and row_height_minus1[i]. NOTE 1 - This constraint is similar to the constraint related to tiles_fixed_structure_flag being equal to 1, and it is preferable that tiles_fixed_structure_flag be equal to 1 when an MCTS Extraction Information Set SEI message is present (although this is not required). If multiple MCTS Extraction Information Set SEI messages exist for images in the associatedPicSet, they shall contain the same content. A NAL unit that contains tiles that belong to tile set tileSetA must not contain tiles that do not belong to tile set tileSetA. The number of MCTS Extraction Information Set (SEI) messages in each access unit must not exceed five. num_extraction_info_sets_minus1 plus 1 indicates the number of extraction information sets contained in the MCTS Extraction Information Set SEI message that is applied to the mcts extraction process. The value of num_extraction_info_sets_minus1 is 0 to 2. 32 Must be in the range -2. The i-th extracted information set is assigned an MCTS extracted information set identifier value equal to i. num_associated_tile_set_identifiers_minus1[i] plus 1 indicates the number of mcts_id values ​​of tilesets in the i-th extraction info set. The value of num_extraction_info_sets_minus1[i] ranges from 0 to 2. 32 Must be in the range -2. mcts_identifier[i][j] identifies the j-th tile set with mcts_id equal to mcts_identifier[i][j] associated with the ith extraction information set. The value of mcts_identifier[i][j] ranges from 0 to 2. 32 Must be in the range -2. num_vps_in_extraction_info_set_minus1[i] plus 1 indicates the number of replacement video parameter sets in the i-th extraction information set. The value of num_vps_in_extraction_info_set_minus1[i] must be in the range of 0 to 15. vps_rbsp_data_length[i][j] indicates the number of bytes vps_rbsp_data_bytes[i][j][k] of the next j-th replacement video parameter set in the i-th extraction information set. num_sps_in_extraction_info_set_minus1[i] plus 1 indicates the number of permutation sequence parameter sets in the i-th extraction information set. The value of num_sps_in_extraction_info_set_minus1[i] must be in the range of 0 to 15. sps_rbsp_data_length[i][j] indicates the number of bytes of sps_rbsp_data_bytes[i][j][k], which is the next j-th replacement sequence parameter set in the i-th extraction information set. num_pps_in_extraction_info_set_minus1[i] plus 1 indicates the number of replaced image parameter sets in the i-th extraction information set. The value of num_pps_in_extraction_info_set_minus1[i] must be in the range of 0 to 63. pps_nuh_temporal_id_plus1[i][j] specifies the temporal identifier for generating a PPS NAL unit associated with the PPS data specified in the PPS RBSP specified in pps_rbsp_data_bytes[i][j][] for the j-th replaced picture parameter set for the i-th extraction information set. pps_rbsp_data_length[i][j] indicates the number of bytes, pps_rbsp_data_bytes[i][j][k], of the next j-th replaced image parameter set set in the i-th extraction information set. mcts_alignment_bit_equal_to_zero must be equal to 0. vps_rbsp_data_bytes[i][j][k] contains the kth byte of the RBSP of the next jth replacement video parameter set in the i-th extraction information set. sps_rbsp_data_bytes[i][j][k] contains the kth byte of the RBSP of the next jth permutation sequence parameter set in the i-th extraction information set. pps_rbsp_data_bytes[i][j][k] contains the kth byte of the RBSP of the next jth replaced image parameter set in the i-th extraction information set. The sub-bitstream MCTS extraction process is applied as follows. Let the bitstream inBitstream, the target MCTS identifier mctsIdTarget, the target MCTS extraction information set identifier mctsEISIdTarget, and the target highest TemporalId value mctsTIdTarget be the inputs to the sub-bitstream MCTS extraction process. The output of the sub-bitstream MCTS extraction process is the sub-bitstream outBitstream. The bitstream conformance requirement for an input bitstream is that any output sub-bitstream that is the output of the process specified in this section along with the bitstream is a conforming bitstream. The output sub-bitstreams are derived as follows. The bitstream outBitstream is set to be identical to the bitstream inBitstream. The lists ausWithVPS, ausWithSPS and ausWithPPS are set to consist of all access units in outBitstream that contain VCL NAL units of type VPS_NUT, SPS_NUT and PPS_NUT. Remove all SEI NAL units that have nuh_layer_id equal to 0 and that contain non-nested SEI messages. NOTE 2 – A "smart" bitstream extractor may include appropriate non-nested SEI messages in the extracted sub-bitstream if the SEI messages applicable to the sub-bitstream are present as nested SEI messages in the original bitstream's mcts_extraction_info_nesting(). -Remove all types of NAL units from outBitstream: - A VCL NAL unit that contains a tile that does not belong to a tileset with mcts_id[i] equal to mctsIdTarget. - A non-VCL NAL unit having type VPS_NUT, SPS_NUT, or PPS_NUT. - Insert into all access units in list ausWithVPS in outBitstream num_vps_in_extraction_info_minus1 [mctsEISIdTarget] plus 1 NAL units of type VPS_NUT generated from the VPS RBSP data in the mctsElSldTarget-th MCTS extraction information set, i.e., vps_rbsp_data_bytes [mctsEISIdTarget] [j][] for all values ​​of j in the range from 0 to num_vps_in_extraction_info_minus1 [mctsEISIdTarget]. For each generated VPS_NUT, nuh_layer_id is set equal to 0 and nuh_temporal_id_plus1 is set equal to 1. - Insert into all access units in the list ausWithSPS in the outBitstream num_sps_in_extraction_info_minus1[mctsEISIdTarget] plus 1 NAL unit with type SPS_NUT generated from the SPS RBSP data in the mctsEISIdTarget-th MCTS extraction information set, i.e., sps_rbsp_data_bytes[mctsEISIdTarget][j][] for all values ​​of j in the range from 0 to num_sps_in_extraction_info_minus1[mctsEISIdTarget]. For each generated SPS_NUT, nuh_layer_id is set equal to 0 and nuh_temporal_id_plus1 is set equal to 1. - Insert into all access units in the list ausWithPPS in outBitstream NAL units with type PPS_NUT generated from the mctsEISIdTarget-th MCTS extraction information set, i.e., the PPS RBSP data in pps_rbsp_data_bytes[mctsEISIdTarget][j][] for all values ​​of j in the range 0 to num_pps_in_extraction_info_minus1[mctsEISIdTarget] inclusive, where pps_nuh_temporal_id_plus1[mctsEISIdTarget][j] is less than or equal to mctsTIdTarget. For each generated PPS_NUT, nuh_layer_id is set equal to 0 and nuh_temporal_id_plus1 is set equal to pps_nuh_temporal_id_plus1[mctsEISIdTarget][j] for all values ​​of j in the range from 0 to num_pps_in_extraction_info_minus1[mctsEISIdTarget], where pps_nuh_temporal_id_plus1[mctsEISIdTarget][j] is less than or equal to mctsTIdTarget. - Remove all NAL units with TemporalId greater than mctsTIdTarget from outBitstream. -For each remaining VCL NAL unit in outBitstream, adjust the slice segment headers as follows: - For the first VCL NAL unit in each access unit, set the value of first_slice_segment_in_pic_flag equal to 1, and set it equal to 0 otherwise. - Set the value of slice_segment_address according to the tile set defined in the PPS where pps_pic_parameter_set_id is equal to slice_pic_parameter_set_id.

[0128] The MCTS Extract Information nested SEI message syntax can be designed as follows:

[0129] JPEG2025028895000005.jpg129166

[0130] With regard to semantics, it should be noted that MCTS extraction information nested SEI messages may be present in addition to or instead of MCTS extraction information set SEI messages to form information 50.

[0131] The MCTS Extract Information Nested SEI message carries nested SEI messages and provides a mechanism for associating the nested SEI messages with bitstream subsets that correspond to one or more motion constrained tile sets. In the sub-bitstream MCTS extraction process specified in the semantics of the MCTS extraction information set SEI message, nested SEI messages contained in the MCTS extraction information nested SEI message may be used to replace non-nested SEI messages in the access unit that contains the MCTS extraction information nested SEI message. all_tile_sets_flag equal to 0 indicates that the mcts_identifier list consists of mcts_identifier[i]. all_tile_sets_flag equal to 1 indicates that the list mcts_identifier[i] consists of all values ​​of mcts_id[] in temporal_motion_constrained_tile_sets SEI messages that are present in the current access unit.

[0132] num_associated_mcts_identifiers_minus1 plus 1 indicates the number of subsequent mcts_identifiers. The value of num_associated_mcts_identifiers_minus1[i] is 0 to 2. 32 Must be in the range -2. mcts_identifier[i] indicates the tile set with mcts_id equal to mcts_identifier[i] associated with the next nested SEI message. The value of mcts_identifier[i] ranges from 0 to 2. 32 Must be in the range -2. num_seis_in_mcts_extraction_seis_minus1 plus 1 indicates the number of next nested SEI messages. mcts_nesting_zero_bit must be equal to 0.

[0133] It has already been indicated above that the evaluation or generation of the information 50, i.e. the information guiding the parameters and / or SEI adaptation, can alternatively take place in the outer encoder 80, i.e. outside the place where the actual encoding of the image 12 into the stream 10 takes place. According to such an alternative, the data stream 10 may be transmitted accompanied by the original parameters 20a and / or the original SEI message relating only to the unreduced stream 10. Optionally, information regarding one or more supported sub-regions 22 of the image 12 may be present in the video stream 10, but this is not mandatory, since the evaluation of the information 50 may be based solely on evaluating the tile structure of the stream 12 to determine the one or more sub-regions. In doing so, the burdensome task of evaluating the information 50 is transferred from the encoder site to a location closer to the client, or even to a user site, such as immediately upstream of the final decoder 82, but avoiding the need for transmission of the complete, i.e. unreduced, data stream 10, by ceasing to transmit portions 70 of the payload portion 18, referring to regions of the image 12 outside the desired sub-region 22. The original encoding parameter set 20a and / or the SEI message for the unreduced data stream 12 will of course be transmitted. The network entity 60 performing the actual reduction or removal 68 of the portion 70 can be located immediately upstream of the entity performing the evaluation of the information 50 as well. For example, the streaming device specifically downloads only the part of the payload portion 18 of the data stream 10, which does not belong to the portion 70. For this purpose, some download prescriptions can be used, such as a manifest file. For this purpose, the DASH protocol can be used. The evaluation of the information 50 can in fact be performed in such a network device located before the decoder, as a mere preparation for the actual parameter adjustment of the parameters according to FIG. 3 or FIG. 5, and / or for the SEI replacement, respectively.Overall, the network device may include an interface for receiving a reduced data stream 10, which includes payload portion 18 with fewer portions 70, but still has parameters 20a that parameterize the complete encoding of image 12 in payload portion 18 including portion 70, according to the alternatives described above. The position indication 32 may still be the original position indication. That is, the received data stream is in fact erroneously encoded. The received data stream corresponds to data stream 62 shown in FIG. 5 with parameters 20b, but still with the original parameters 20a, and still with the wrong position indication 32. The index 54 in that stream may refer to parameters 20a in that stream and may not have been modified relative to the original encoding of the unreduced data stream. In fact, the unreduced original data stream may simply differ from the original data stream received by the omission of portions 70. Optionally, one or more SEI messages are included, which refer to the original encoding, for example the size of original image 12 or other characteristics of the complete encoding. At the output of such a network device, a data stream 62 is output, which is decoded by a decoder 82. Between this input and the output is connected a module that adapts the SEI messages and / or parameters to fit the sub-area 12 where the inbound data stream 10 has already been subjected to reduction, i.e. such a module must perform task 74, i.e. adjusting parameter 20a to become parameter 20b, and task 78, i.e. adjusting the position indication 32 to refer precisely to the periphery of the sub-area 22. The knowledge of the sub-area 22 where the reduction has been performed may be internal to such a network device, in the latter case of specifically restricting the download of the stream 10 to the part of the payload portion 18 with reference to the sub-area 22, or it may be externally supplied in the case of another network device that may be present upstream of the network device, assuming the task of reduction 68.For a network device including a module between the input and the output, the same statements apply as made above with respect to the network device of Fig. 4, for example in terms of implementation in software, firmware or hardware. To summarise the embodiment just outlined, the embodiment concerns a network device arranged to receive a data stream including a portion of the payload portion in which an image of a video is encoded. This portion corresponds to the result when the portion 70 is excluded from the payload portion 18 referring to an area of ​​the image outside the predetermined sub-area 22 of the image. The image 12 of the video is encoded in the payload portion 18 in a parameterised manner without exclusion, using the encoding parameter settings of the parameterised portion of the data stream. That is to say, the parameterised portion of the received data stream correctly parameterises the encoding of the image 12 in the payload portion 18 if the portion 70 is not left. Additionally or alternatively, the image 12 of the video is encoded in the payload portion 18 in such a way as to fit without exclusion the auxiliary enhancement information indicated by the auxiliary enhancement messages of the data stream. That is to say, the SEI messages optionally included by the received data stream actually coincide with the unreduced payload portion 18. The network device modifies the received data stream by modifying 78 the position indication 32 in the payload portion to indicate a position measured from the periphery of the predefined sub-region 22 instead of the image 12, adjusts the encoding parameter settings of the parameter setting portion and / or adjusts the auxiliary enhancement information message such that the modified data stream has a sub-region specific image 84 indicating part of the payload portion 18, i.e. all but 70, or in other words the predefined sub-region of the image, coded to be correctly parameterized using the thus adjusted encoding parameter settings and / or coded to match the auxiliary enhancement information indicated by the adjusted auxiliary enhancement information message after the adjustment. The generation of the parameters 20b in addition to the parameters 20a has been done by the encoder 80 according to the previous embodiment to result in a data stream carrying both parameter settings 20a and 20b.Here, in the above alternative, the parameters 20a are changed to the corresponding parameters 20b on the fly. The adjustment of the parameters 20a to the parameters 20b requires the use of knowledge regarding the sub-region 22 to modify the parameters 20a. For example, the image size 26 in the configuration 20a corresponds to the size of the complete image 12, but the image size should indicate the size of the sub-region 22 or the image 86, respectively, after the adjustment of the configuration 20b. Similarly, the tile structure 38 in the configuration 20a corresponds to the tile structure of the complete image 12, but the tile structure 38 should indicate the tile structure of the sub-region 22 or the image 86, respectively, after the adjustment of the configuration 20b. Similar statements apply, for example, with respect to the buffer size and timing 46, but not exclusively. If there are no retransmitted SEI messages in the received incoming data stream, no adjustment is necessary. Alternatively, instead of adjusting the SEI messages, they can simply be aborted.

[0134] It should be noted that with respect to the above embodiment, the adaptation of the auxiliary enhancement information may be related to buffer size and / or buffer timing data. In other words, the type of information in any present SEI that is matched or different between the original SEI and the replacement SEI to fit the decomposed or reduced video stream may be related, at least in part, to the buffer size and / or buffer timing data. That is, the SEI data in stream 10 may have buffer size and / or buffer timing data related to the full encoding, while the replacement SEI data, which is conveyed in addition to the former one as described with respect to FIG. 1 or generated on the fly as described in the previous paragraph, has buffer size and / or buffer timing data related to the reduced stream 62 and / or reduced image 86.

[0135] The following description relates to a second aspect of the present application, namely, a concept for enabling more efficient transport of video data that does not fit the usual rectangular image shape of a video codec. As before, with respect to the first aspect, the following description starts with a kind of introduction, i.e., an exemplary description of an application in which such a problem may arise, in order to motivate the advantages resulting from the embodiments described below. However, it should also be noted that this preliminary description should not be understood as limiting the breadth of the embodiments described later. Moreover, it should be noted that the aspects of the present application described next can also be combined in an advantageous manner with the embodiments described above. Details in this regard are also given below.

[0136] The problem explained next arises from the different projections used for panoramic videos, especially when processes such as the sub-region extraction mentioned above are applied.

[0137] For example, in the following description we use the so-called cubic projection. Cubic projection is a special case of rectilinear projection, also called gnomonic projection. This projection describes the transformation approximated for most conventional camera systems / lenses when an image representation of a scene is acquired. Straight lines in the scene are mapped to straight lines in the resulting image, as shown in Figure 14.

[0138] A cubic projection applies a rectangular projection to map the perimeter of a cube onto six faces, each of which has a viewing angle of 90° x 90° from the center of the cube. The result of this type of cube projection is shown as image A in Figure 15. Other arrangements of the six faces in a common image are possible as well.

[0139] In order now to derive a more coding-friendly representation of the image A thus obtained (i.e., with fewer unused image regions 130 and rectangular shapes), the image patches can be displaced within the image, for example as shown in FIG. 16, resulting in image B.

[0140] From a system point of view, it is important to have an understanding of how the various image patches in image B (Fig. 16) are spatially related to the original (world-view direction) continuous representation of image A, i.e. additional information to derive image A (Fig. 15) given the representation of image B. In particular, under the above-mentioned circumstances, processing such as the above-mentioned sub-region extraction in the coding domain of Figs. 1-13 is performed on the server side or on a network element. In this case, only the part of image B shown via the isolated ROI 900 in Fig. A is available at the end device. The end device must be able to remap the relevant region of a given (partial) video to the correct position and area expected by the end display / rendering device. The server or network device, which modifies the coded video bitstream according to the above-mentioned extraction process, can generate, add or adjust the respective signaling displacement signaling depending on the modified video bitstream.

[0141] Thus, the embodiments described later provide signaling indicating within a video bitstream (rectangle) group of samples of image B. Furthermore, the displacement of each group of samples with respect to the samples of image B in the horizontal and vertical directions. In a further embodiment, the bitstream signaling includes explicit information about the resulting image size of image A. Furthermore, default luma and chroma values ​​of samples not covered by the displaced sample group or samples initially covered by the displaced sample group. Furthermore, some of the samples of image A may be initialized with sample values ​​of the corresponding samples of image B.

[0142] An exemplary embodiment is shown in the syntax table of FIG.

[0143] A further embodiment utilizes tile structure signaling for indication of samples belonging to a displaced sample group.

[0144] With reference to Fig. 18, an embodiment of a data stream 200 according to the second aspect is shown. The data stream 200 has a video 202 encoded therein. The embodiment of Fig. 18 may be modified to refer to one image encoded only in the data stream 200, but to facilitate understanding of those examples in which an embodiment according to the second aspect is combined with any of the embodiments of the other aspects of the present application, Fig. 18 shows an example in which the data stream 200 is a video data stream rather than an image data stream. As just mentioned, the data stream 200 has an image 204 of the video 202 encoded therein.

[0145] In addition, however, the video data stream 200 contains displacement information 206. The displacement information has the following meaning: In practice, the data stream 200 must convey image content in an image 204 with a non-rectangular perimeter. Fig. 18 shows an example of such an image content in reference number 200. That is, the image content 208 indicates the image content that the data stream 200 should convey in one timestamp, i.e. in one image 204. However, the perimeter 210 of the actual image content is non-rectangular. In the example of Fig. 18, the actual image content corresponds rather to a six-sided rollout of a cube 212, whose sides are distinguished from one another by using the normal distribution of the numbers 1 to 6 in Fig. 18, six on each of the six faces of the cube 212, i.e. the number of opposite faces equals a total of seven. Each face thus represents one sub-area 214 of the actual image content 208, and may represent, for example, the appropriate projection of the sixth of a complete three-dimensional panoramic scene into the respective sub-area 214, which according to this example is of square shape. However, as already mentioned above, the sub-regions 214 may be of different shapes and may be arranged within the image content 208 in a manner other than a regular arrangement of rows and columns. In any case, the actual image content shown in 208 is non-rectangular in shape, and therefore the smallest possible rectangular target image area 216 that would completely contain the cropped panoramic content 208 will have an unused portion 130, i.e., a portion that is not occupied by actual image content associated with the panoramic scene.

[0146] Thus, in order to "not waste" image area within the image 204 of the video 202 carried in the data stream 200, the image 204 carries the complete real-world image content 208 such that the spatial relative positions of the sub-regions 214 are changed relative to their positions within the target image region 216.

[0147] As shown in FIG. 18, FIG. 18 shows an example in which four sub-regions, namely sub-regions 1, 2, 4 and 5, should not be displaced when copying image 204 undistorted or congruently to target image region 216, but sub-regions 3 and 6 must be displaced. Illustratively, in FIG. 18, the displacement is a pure translational displacement that can be described by a two-dimensional vector 218, but according to alternative embodiments, more complex displacements can be selected, such as displacements that further include, for example, scaling and / or reflection (mirroring) and / or rotation of the respective sub-regions when transitioning between image 204 on the one hand and target image region 216 on the other hand. The displacement information 206 can indicate, for each of a set of at least one predetermined sub-region of image 204, the displacement of the respective sub-region in target image region 216 relative to an undistorted or undistorted copy of image 204 to target image region 216. For example, in the example of FIG. 18, the displacement information 206 can indicate the displacement of a set of sub-regions 214 that only encompasses sub-regions 3 and 6. Alternatively, the displacement information 206 can indicate a displacement for a subregion 214 or a set of displaced subregions 214 relative to some default point of the target region 216, such as its upper left corner. By default, the remaining subregions in the example image 204 of FIGS. 18,1,2,4 and 5 can be treated as remaining intact upon undistorted or congruent copying onto the target image 216, for example, relative to the default point.

[0148] 19 shows that the displacement information may include a count 220 of the sub-regions 214 to be displaced, a target image region size parameter indicating the size of the target image region 216, 222, and, for each of the n displaced sub-regions, coordinates 224 describing the displacement 218 of each sub-region 214 to be displaced, for example by measuring the displacement to the aforementioned default point or its relative position, when mapping the sub-regions 214 in their original arrangement in the image 204 onto the target region 216. In addition to the coordinates 224, the information 206 may include a scaling 226 for each sub-region 214 to be displaced, i.e. an indication of how each displaced sub-region 214 should be scaled according to the respective coordinates 224 in the target image region 216 when mapping the respective sub-region 214. The scaling 226 may result in an enlargement or reduction with respect to the non-displaced sub-regions 214. Alternatively, a horizontal and / or vertical reflection and / or rotation may be signaled to each sub-region 214. Additionally, the displacement information 206 may include the coordinates of the top left and bottom right corners of each region, again measured relative to a default point or respective corner if mapping onto the region of interest 216 without displacing the respective subregions 214, for each subregion to be displaced or positioned within the region of interest 216. This allows positioning, scaling and reflection to be signaled. Thus, the subregions 214 of the transmitted image 204 may be freely displaced relative to their original relative positioning within the image 204.

[0149] The displacement information 206 may have a scope, i.e. validity, for a time interval of the video 202 larger than one timestamp or one image 204, as for example the case of a sequence of images 204 or the entire video 202. Furthermore, Fig. 18 shows that the data stream 200 may also optionally contain default filling information 228 indicating the default filling with which the portions 130 of the target image region should be filled at the decoding side, i.e. that these portions 130 are not covered by any of the sub-regions 214, i.e. either the displaced or non-displaced sub-regions 214 of the image 204. In the example of FIG. 18, for example, subregions 1, 2, 4 and 5 form a non-displaced portion of image 204, which is shown unhatched in FIG. 18, while the remainder of image 204, i.e., subregions 3 and 6, which are shown hatched in FIG. 18, and all six of these subregions do not cover the shaded portion 130 of target image region 216 after subregions 3 and 6 have been displaced according to information 206 such that portion 130 is filled according to default filling information 228.

[0150] An encoder 230 suitable for generating the data stream 200 is shown in Fig. 20. The encoder 230 simply accompanies or provides the data stream 200 with information 206 indicating the displacement required to fill the target image region with the image 204 to be encoded in the data stream 200. Fig. 20 further shows that the encoder 230 can generate the data stream 200 so that it can be reduced in a compliance-preserving manner, for example, as the data stream 10 of the embodiment of Figs. 1 to 13. In other words, the encoder 230 may be implemented to realize the encoder 80 of Fig. 1. For the sake of illustration, Fig. 20 shows the case where the image 204 is subdivided into an array of sub-regions 214, i.e. a 2x3 array of sub-regions according to the example of Fig. 18. The image 204 can correspond to the image 12, i.e. an image of the unreduced data stream 200. Fig. 20 exemplarily shows sub-regions 22 with respect to which the encoder 230 can reduce the data stream 200. As explained with respect to Fig. 1, there may be several such sub-regions 22. By this means, the network device can reduce the data stream 200 such that it simply extracts a portion of the data stream 200 to obtain a reduced video data stream in which the image 86 simply represents the sub-region 22. The data stream 200 can be reduced and the reduced video data stream associated with the sub-region 22 is shown in FIG. 20 using the reference numeral 232. The image 204 of the unreduced video data stream 200 represents the image 204, whereas the image of the reduced video data stream 232 simply represents the sub-region 22. The encoder 230 can additionally provide the video data stream 200 with displacement information 206' specific to the sub-region 22 in order to provide a receiver, such as a decoder receiving the reduced video data stream 232, with the ability to fill the target image region with the content of the image of the reduced video data stream 232. A network device, such as 60 of FIG. 4, can remove the displacement information 206 and simply carry over the displacement information 206' to the reduced video data stream 232 as a result of reducing the video data stream 200 to reduce the video data stream 232.By this means a receiver, such as a decoder, is able to fill in the image content of the reduced video data stream 232 related to the content of the sub-region 22 onto the target image region 216 shown in FIG.

[0151] Again, it should be emphasized that Figure 20 should not be understood as being limited to reducible video data streams: if the video data stream 200 is reducible, different concepts may be used than those presented above with respect to Figures 1 to 13.

[0152] Fig. 21 shows a possible reception. The decoder 234 receives the video data stream 200 or the reduced video data stream 232 and reconstructs its image, i.e. an image that simply shows the sub-area 22 based on the image 204 or the reduced video data stream 232, respectively. The receiver is a decoder and comprises, besides the decoding core 234, a displacer 236 that uses the displacement information of 206 for the video data stream 200 or 206' for the reduced video data stream to fill the target image area 216 based on the image content. The output of the displacer 236 is thus the filled target image area 216 for each image of the respective data stream 200 or 232. As mentioned above, some parts of the target image area remain unfilled by the image content. Optionally, a renderer 238 may be connected after the displacer 236 or to the output of the displacer 236. The renderer 238 applies an injective projection to the target image formed by the target image region 216 - or at least a sub-region thereof within the fill region of the target image region - to form an output image or output scene 240 that corresponds to the currently viewed scene section. The injective projection performed by the renderer 238 may be the inverse of a cubic projection.

[0153] Thus, the above embodiment allows for rectangular region-by-region packing of image data, such as panoramic or semi-panoramic scenes. A specific syntax example is given below: Below, an example syntax in the form of pseudocode called RectRegionPacking(i) is presented that specifies how a source rectangular region of the projected frame, i.e. 216, is packed into a destination rectangular region of the packed frame, i.e. 204. Horizontal mirroring and rotation of 90 degrees, 180 degrees, or 270 degrees is indicated, and vertical and horizontal resampling is inferred from the width and height of the region.

[0154] aligned(8)class RectRegionPacking(i)[ unsigned int(32)proj_reg_width [i]; unsigned int(32)proj_reg_height [i]; unsigned int(32)proj_reg_top [i]; unsigned int(32)proj_reg_left [i]; unsigned int(8)transform_type [i]; unsigned int(32)packed_reg_width [i]; unsigned int(32)packed_reg_height [i]; unsigned int(32)packed_reg_top [i]; unsigned int(32)packed_reg_left [i]; ]

[0155] The semantics are as follows:

[0156] proj_reg_width[i], proj_reg_height[i], proj_reg_top[i] and proj_reg_left[i] are in pixels with width and height in the projected frame, i.e. 216, equal to proj_frame_width and proj_frame_height, respectively. i is the index in the case of the respective region, i.e. tile 214, in comparison with FIG. 18. proj_reg_width[i] specifies the width of the i-th region in the projected frame. proj_reg_width[i] must be greater than 0. proj_reg_height[i] specifies the height of the i-th region in the projected frame. proj_reg_height[i] must be greater than 0. proj_reg_top[i] and proj_reg_left[i] specify the topmost sample row and leftmost sample column in the projected frame. Values ​​lie in the range from 0 or greater, inclusively indicating the top-left corner of the projected frame, to proj_frame_height and proj_frame_width, respectively, exclusive. proj_reg_width[i] and proj_reg_left[i] must be constrained such that proj_reg_width[i] + proj_reg_left[i] is less than proj_frame_width. proj_reg_height[i] and proj_reg_top[i] must be constrained such that proj_reg_height[i] + proj_reg_top[i] is less than proj_frame_height. If the projected frame 216 is stereoscopic, proj_reg_width[i], proj_reg_height[i], proj_reg_top[i], and proj_reg_left[i] are such that the regions identified by these fields on the projected frame lie within a single constituent frame of the projected frame.transform_type[i] specifies the rotation and mirroring applied to the i-th region of the projected frame to map to the packed frame, and therefore the mapping must be reversed to map the respective region 214 back to region 216. Of course, it can indicate a mapping from image 204 to target region 216. If transform_type[i] specifies both rotation and mirroring, the rotation is applied after the mirroring. Of course, the reverse is also possible. According to an embodiment, the following values ​​are specified, although other values ​​may be reserved:

[0157] 1: no transformation, 2: mirror horizontally, 3: rotate 180 degrees counterclockwise, 4: mirror horizontally then rotate 180 degrees counterclockwise, 5: mirror horizontally then rotate 90 degrees counterclockwise, 6: rotate 90 degrees counterclockwise, 7: mirror horizontally then rotate 270 degrees counterclockwise, 8: rotate 270 degrees counterclockwise. Note that the values ​​correspond to the EXIF ​​image orientation tag.

[0158] packed_reg_width[i], packed_reg_height[i], packed_reg_top[i], and packed_reg_left[i] specify, respectively, the width, height, topmost sample row, and leftmost sample column of the region within the packed frame, i.e., the region covered by a tile 214 in image 204. The rectangle specified by packed_reg_width[i], packed_reg_height[i], packed_reg_top[i], and packed_reg_left[i] shall not overlap with the rectangle specified by packed_reg_width[j], packed_reg_height[j], packed_reg_top[j], and packed_reg_left[j] for any value of j in the range 0 to i-1, inclusive.

[0159] To summarize and generalize the above example, the embodiments further described above may differ in that for each region or tile 214 of the image 214, two rectangular regions are indicated, i.e. the region that each region or tile 214 covers within the target region 216 and the rectangular region that each region or tile 214 covers within the image region 204, including mapping rules for mapping the image content, i.e. reflection and / or rotation, of each region or tile 214 between these two regions. Scaling can be signaled by signaling a pair of regions of different sizes.

[0160] In the following, the third aspect of the present application is described. The third aspect relates to an advantageous concept of distributing access points to a video data stream. In particular, access points related to one or more sub-regions of an image encoded in the video data stream are introduced. Advantages resulting therefrom are described below. As with the other aspects of the present application, the description of the third aspect is accompanied by an introduction that explains the problems that arise. As with the description of the first aspect, this introduction exemplarily refers to HEVC, but this context should also not be interpreted as limiting the embodiments described later to refer only to HEVC and its extensions.

[0161] In the context of the TMCTS system described above, tile-specific random access points may provide clear benefits: random access in tiles at different time instances allows a more even distribution of bitrate peaks across pictures in a video sequence. All or a subset of the mechanisms for picture-specific random access in HEVC can be transferred to tiles.

[0162] One picture-specific random access mechanism is the indication of an intra-coded picture or access unit a) in presentation order or b) where subsequent pictures in coding and presentation order inherit prediction dependencies to the previous picture samples of the intra-coded picture. In other words, a reference picture buffer reset is indicated either instantly in case of b) or from the first subsequent picture on in case of a). In HEVC, such access units are indicated on the Network Abstraction Layer (NAL)) via specific NAL unit types, i.e. so-called Intra Random Access Point (IRAP) access units like BLA, CRA (both above category a) or IDR (above category b). The embodiments described further below may use a NAL unit header level indication, e.g. via a new NAL unit type or an SEI message for backward compatibility, which indicates to a decoder or a network intermediate box / device that a given access unit contains at least one intra-coded slice / tile that satisfies condition a) or b), i.e. some form of reference picture buffer reset is applied on a slice / tile basis. Furthermore, slices / tiles can be identified by slice header level indication for the picture on the encoder side in addition to or instead of NAL unit type signaling. The pre-decoding operation allows to reduce the DPB size required for post-extraction decoding.

[0163] For this to happen, the constraints expressed with fixed_tile_structure enabled must be satisfied: samples of a previous tile of a given access unit should not be referenced by the same tile (and not other tiles) of the current image.

[0164] According to some embodiments, the encoder can constrain the coding dependency via temporal prediction between sub-regions, such that for each sub-region where RA occurs, the image region used as a reference for temporal prediction in the reference image is extended by the image region covered by further sub-regions if these further sub-regions are subjected to RA in the same way. These slices / tiles / sub-regions are indicated in the bitstream, for example at the NAL unit or slice level or on the SEI message. Such a structure prevents the extraction of sub-regions, but mitigates the penalty of constrained temporal prediction. The type of sub-region random access (allows extraction or not) must be distinguishable from the bitstream indication.

[0165] Another embodiment exploits the above signaling opportunity by adopting a specific coding-dependent structure, where image-wise random access points exist at specific points in time and with a coarse time granularity that allows instantaneous random access without drift with existing state-of-the-art signaling.

[0166] However, at finer temporal granularity, the coding structure allows tile-wise random access that distributes the bitrate load of intra-coded image samples over time towards a less fluctuating bitrate behavior. For backward compatibility, this tile-wise random access can be signaled via SEI messages, keeping the respective slices as non-RAP pictures.

[0167] In the sub-picture bitstream extraction process, the NAL unit type indicated by the above SEI message indicating tile-based random access within such a stream structure is changed to picture-wise random access, signaling instantaneous random access opportunities in each picture of the extracted sub-bitstream, if necessary.

[0168] A video data stream 300 according to an embodiment of the third aspect of the present application is described with respect to Fig. 22. The video data stream 300 has encoded therein a sequence of images 302, i.e. videos 304. As with the other previous embodiments, the temporal order in which the images 302 are shown may correspond to a presentation time order that may or may not coincide with the decoding order in which the images 302 are encoded in the data stream 300. That is, although not described with respect to the other previous figures, the video data stream 300 may be subdivided into a sequence of access units 306, each access unit 306 being associated with a respective one of the images 302, and the order in which the images 302 are associated with the sequence of access units 306 corresponds to the decoding order.

[0169] The pictures 302 are encoded into the video data stream 300 using temporal prediction, i.e., predictively coded pictures between the pictures 302 are coded using temporal prediction that is based on one or more temporal reference pictures that precede each picture in decoding order.

[0170] Instead of having just one type of random access picture, the video data stream 300 includes at least two different types, as described below. In particular, normal random access pictures are pictures where no temporal prediction is used, i.e. each picture is coded in a way that is independent of any previous picture in the decoding order. For such normal random access pictures, the stopping of temporal prediction concerns the whole picture area. According to the embodiments described below, the video data stream 300 may or may not include such normal picture-by-picture random access pictures.

[0171] As just explained, random access pictures do not depend on previous pictures in the decoding order. They therefore allow random access to the decoding of the video data stream 300. However, encoding pictures without temporal prediction means encoding penalties in terms of compression efficiency. Thus, normal video data streams produce bit rate peaks, i.e. bit rate maxima, at random access pictures. These problems can be solved by the above-mentioned embodiments.

[0172] According to the embodiment of FIG. 22, the video data stream 300 comprises a first set of one or more images of type A encoded in the video data stream 300 forming a first set of one or more first random access points that suspend temporal prediction in at least a first image sub-region A, and a second set of one or more images of type B encoded in the video data stream 300 forming a second set of one or more second random access points of the video data stream 300 while suspending temporal prediction in a second image sub-region B different from the first image sub-region A.

[0173] In the case of Fig. 22, the first and second image sub-regions A and B are indicated using hatching and do not overlap each other, as shown in Fig. 23a, so that sub-regions A and B adjoin each other along a common boundary 308 such that sub-regions A and B cover the entire image area of ​​the image 302. However, this is not necessarily the case. As shown in Fig. 23b, sub-regions A and B may partially overlap, or as shown in Fig. 23c, the first image region A may actually cover the entire image area of ​​the image 302. In the case of Fig. 23c, the images of type A are image-wise random access points where the temporal prediction is completely switched off, i.e. they are coded in the data stream 300 without temporal prediction throughout the entire respective image. For completeness, Fig. 23d shows that sub-region B does not have to be located inside the image area 302, but can also adjoin the outer image border 310 of the two images 302. Fig. 23e shows that in addition to images of types A and B, there is an image of type C with an associated sub-region C, which together can completely cover the image area of ​​the image 302.

[0174] The consequence of restricting the regions in the images B and A of FIG. 22 where temporal prediction is suspended to sub-regions A and B is as follows: Usually, the bit rate for coding the image 302 into the video data stream 300 is large for the images forming the random access points, since temporal prediction is refrained from being used throughout the entire respective image region, while prediction from previous images (in decoding order) is broken for subsequent images (at least in presentation order). For the images (pictures) of types A and B of FIG. 22, the avoidance of the use of temporal prediction is only used in sub-regions A and B, respectively, so that the bit rate peaks 312 of these images A and B are relatively low compared to the image-wise random access point images. However, as will be explained below, the reduction of the bit rate peaks 312 occurs at a relatively low cost, at least with respect to the random access rate of the complete image, except for the coding-dependent constraints at the boundaries of the sub-regions. This is illustrated in FIG. 22: Here, let the curve 314 represent a function showing the temporal evolution of the bit rate depending on the time t. As explained, the peaks 312 at the time of images A and B are lower than the peaks resulting from image-by-image random access images. Compared to the case of simply using image-by-image random access images, the image-related random access rate of the video data stream 300 corresponds to the rate at which sub-region-related random access images covering the entire image region are traversed, which in the case of FIG. 22 is the rate at which at least one image of type B and at least one image of type A are encountered. Even the presence of a normal image-by-image random access image in the video data stream 300, i.e., image A, as in the example of FIG. 23c, is advantageous over a normal video data stream in which such images are simply distributed in time across the video data stream. In particular, in such a case, the presence of a sub-region-related random access image, i.e., image B, in the case of FIG. 23c, can be utilized to be able to reduce the image rate of image A, which comes with a high bit rate peak, as will be explained in more detail below.However, compensating for the increased random access latency by interspersing images of type B in the video data stream 300 between images of type A allows for sub-region restricted random access to the video data stream, filling the time until the next image unit random access point is encountered, i.e., the next image A.

[0175] Before proceeding to a description of a decoder that utilizes a special type of random access image within the video data stream 300, some remarks are made with respect to sub-region B and / or sub-region A, whereby the way image 302 is encoded into the video data stream 300 by taking into account sub-regions beyond the interruption of temporal prediction within sub-regions A and B while applying temporal prediction within the same image outside of sub-regions A and B.

[0176] FIG. 22 illustrates, using a dashed box 316, a video encoder configured to encode the image 302 into the video data stream 300 to include the just outlined images A and B, respectively. As already outlined above, the video encoder 316 may be a hybrid video encoder using motion compensated prediction for encoding the image 302 into the video data stream 300. The video encoder 316 may use any GOP (Group of Pictures) structure for encoding the image 302 into the video data stream 300, such as an open GOP structure or a closed GOP structure. With regard to the sub-region related random access images A and B, this means that the video encoder 316 interposes between the images 302 an image whose sub-region, i.e. A or B, does not depend on any previous image in the decoding order. It will be explained later that such sub-regions B and / or A may correspond to the sub-regions 22 according to the embodiment of FIGS. 1 to 13, i.e. sub-regions to which the video data stream 300 can be reduced. However, it should be noted that this is merely an example and that reducibility is not a necessary property of the video data stream 300, although it would be advantageous if image 312 were encoded in the video data stream 300 such that it conforms to the boundaries of sub-regions B and / or A, similar to the discussion given above with respect to the boundaries of sub-region 22 in relation to Figures 1-13.

[0177] In particular, although the reach of spatial coding dependency mechanisms when coding the images 302 in the video data stream 300 is usually short, it is advantageous if the sub-region-related random access images, i.e. images A and B of FIG. 22, are coded into the video data stream 300 such that the coding dependency for coding the respective sub-region B / A does not cross the boundaries of the respective sub-regions, so as not to introduce coding dependencies outside or spatially neighboring the respective sub-regions. That is, within the respective sub-region B / A, the sub-region-related random access images A and B are coded without temporal prediction and spatial coding dependencies to the parts of the respective images outside the respective sub-region A / B. In addition to this, it is advantageous if the images between the random access images A and B are also coded into the video data stream 300 taking into account the section boundaries of the section in which the immediately preceding section-specific random access image forms a sub-region-specific random access point.

[0178] For example, in Fig. 22, image B forms a sub-region specific random access point with respect to sub-region B, and therefore it is advantageous if image 302 follows this image B and precedes the first occurrence of the next sub-region random access image, i.e. image A. - This sequence is exemplarily indicated using brace brackets 317. - Image 302 is coded taking into account the boundaries of section B. In particular, it is advantageous if the spatial and temporal prediction and coding dependencies for coding these images in the video data stream 300 are constrained such that these images are coded in the video data stream 300 such that section B does not depend on any part of these images or image B itself that is outside section B. With regard to the motion vectors, for example, the video encoder 316 constrains the motion vectors available for coding section B of the images between images B and A such that they do not point to image B and to parts of the images between image B and image A that extend beyond the sub-region of reference image B. Beyond that, no temporal reference image for temporally predicting sub-region B of image 317 should be upstream with respect to image B, since image B forms a random access point with respect to sub-region B. The spatial dependency for coding sub-region B of intermediate images between images B and A is similarly restricted, i.e. so as not to introduce dependencies in adjacent parts outside sub-region B. This restriction may also be relaxed depending on the application, and even beyond this, see the description of figures 1 to 13 for possible countermeasures against drift errors. Similarly, the restriction just described with respect to images between random access images while the predecessor is a sub-region specific random access image such as image B may only be applied with respect to temporal prediction, where spatial dependencies have less significant impact with respect to drift errors.

[0179] The argument raised in the immediately preceding paragraph concerns the restriction of the coding dependency for coding the immediate successor (in terms of the decoding order) image 317 of a sub-region-specific random access image B only with respect to the coding of the image 317 in sub-region B, i.e. the immediately preceding random access image, i.e. the sub-region for which image B forms a sub-region-specific random access point. A question to be addressed separately is the question of whether the coding dependency of a coded image 317 outside section B, i.e. sub-region A in the case of FIG. 22, should be restricted so as to render the coding of the outer part of image 317 dependent on sub-region B. That is to say, the question is whether sub-region A of image 317 should be coded such that its spatial coding dependency is restricted so as not to reach section B of the same image, for example, and whether the temporal coding dependency for coding sub-region A of image 317 should be restricted so as not to reach sub-region B of a reference image, which is either one of the preceding images B or image 317 in the coding / decoding order. More precisely, it should be noted that the reference image used for coding sub-region A of image 317 can belong on the one hand to one of the images coded / decoded before image 317, and on the other hand to the leading random access image B or be located upstream (in decoding order) with respect to image B. In the case of a reference image located upstream with respect to image B, the temporal coding dependency with respect to coded sub-region A of image 317 is still restricted not to reach sub-region B. Rather, the question addressed here is whether the coding dependency for coding sub-region A of image 317 is or is not restricted not to reach section B, as far as the intra or spatial coding dependency is concerned, which is a temporal coding dependency with respect to any previously coded / decoded image of image 317, where the reference image is one of images B. Both options have merit.If subregion B of image B and subregion boundaries of image 317 are also followed when encoding subregion A of image 317, i.e. if the encoding dependency for encoding subregion A is limited and does not reach subregion B, then subregion A continues to be encoded in a manner independent of section B, thus forming a subregion from which data stream 300 can be extracted or reduced as described above. The same applies to subregion B when considering the same situation for encoding subregion B of an image immediately following subregion-specific random access image A. If the reducibility of specific subregions such as subregion A and / or subregion B is not so important, it can be beneficial in terms of encoding efficiency if the immediately preceding encoding dependency reaches the subregion with respect to which the directly preceding section-wise random access image can form a subregion-specific random access point. In that case, other sub-regions, such as for example sub-region A in the preceding discussion, would no longer be able to be reduced, but the coding efficiency would increase, since the video encoder 316 would have less restrictions on the exploitation of redundancy by selecting motion vectors to cross the boundaries of sub-region B, such that the video encoder 316 would temporally predict part of sub-region A of image 317 based on sub-region B of either of these images or image B. For example, the data stream 300 would no longer be able to be reduced with respect to region A if the following were done: image 317 is coded in the video data stream 300 in an image region A outside the second image sub-region B using temporal prediction that at least partially references a second image sub-region B of a reference image among images 317 and image B. That is, in the case of FIG. 22, the complete image region would be available to reference the temporal predictive coding region A of image 317, i.e. A but also B. An image 317' following image A in terms of decoding order, i.e. an image 317' between A and a subsequent image of type B not shown in FIG. 22, is encoded in the video data stream 300 while restricting temporal prediction in an image area A outside the second image sub-area B so as not to reference the second image sub-area B of the preceding reference image in terms of the decoding order between images A and 317'.That is, in the case of Fig. 22, region A is simply available because it is referenced by temporal predictive coding region A of picture 317'. Similarly for sub-region B: picture B attaches to sub-region B within sub-region B, and the coding of B of picture B can use sub-region A as well as A. Then stream 300 would no longer be reducible to sub-region B.

[0180] Before proceeding with a description of a decoder configured to decode the video data stream 300 of Fig. 22, it should be noted that the video encoder 316 may be configured to provide signal processing 319 to the video data stream 300 indicating a spatial subdivision of the corresponding images into sub-regions A and B, to the extent that it relates to the entire video 304 or encompasses the sequence of images 302, or according to any of the alternatives as illustrated in Figs. 23a-23e. The encoder 316 also provides the data stream 300 with signaling 318 indicating images A and B, i.e., signaling 318 marking certain images that are sub-region specific random access points for any of the sub-regions indicated by the signaling 319. That is, the signaling 319 signals a constant spatial subdivision among the images for which the signaling 319 is valid, and the signaling 318 distinguishes the sub-region specific random access images from the other images and associates these sub-region random access images with one of the sub-regions A and B.

[0181] It is possible to make no special distinction in the video data stream 300 between pictures B and A on the one hand, and to make a distinction among other pictures as far as picture type is concerned. In the example of FIG. 22, pictures B and A are "just" sub-region specific random access pictures, and therefore not real picture unit random access pictures such as IDR. Thus, as far as the picture type signaling 320 in the video data stream 300 is concerned, the video data stream 300 does not distinguish between pictures B and A on the one hand, and other pictures that are temporally predicted on the other hand. For example, signaling 318 may be included in the slices or in the slice headers of these slices, indicating that picture 302 is coded in the stream 300 in that unit, whereby the picture regions corresponding to the respective slices form section B. Alternatively, a combination of signaling 320 and 318 may be used to indicate to the decoder that a picture is a sub-region specific random access picture and that the same picture belongs to a specific sub-region. For example, signaling 320 may be used to indicate that a specific picture is a sub-region specific random access picture, but without revealing which sub-region each picture represents a sub-region specific random access point. The latter indication is performed by signaling 318, which indicates which of the sub-regions of the picture subdivision is signaled by signaling 319, and which associates a picture signal with a sub-region specific random access picture by signaling 320. However, signaling 320, which may be a syntax element of NAL unit type, can instead distinguish or differentiate between temporally predicted pictures such as B-pictures and P-pictures, picture-wise random access pictures such as IDR-pictures, and sub-region-wise random access points as between pictures A and B, by using different values ​​for the different sub-regions, i.e. pictures A and B, respectively.

[0182] Further signaling 321 may be inserted by the video encoder 316 to signal for a particular sub-region whether the data stream 300 is reducible for the respective sub-region or not. The signaling 321 may be signaled in a manner that allows one of the sub-regions in the data stream 300 to be signaled as a sub-region in which the data stream 300 is reducible, while other sub-regions do not form such sub-regions in which the data stream 300 is reducible. Alternatively, the signaling 321 may simply allow a binary signaling of reducibility for all sub-regions, i.e., the signaling 321 may signal that all sub-regions are sub-regions in which the data stream 300 is reducible, or alternatively, the signaling 321 may signal that the data stream 300 is not reducible for any of these sub-regions. However, the signaling 321 still leaves the effect that sub-regions such as sub-regions A and B in the example of FIG. 22 are each treated as being completely independently coded sub-regions with respect to whether data stream 300 can be reduced, or not, in which case the asymmetric coding dependencies described above are used across sub-region boundaries, as previously described.

[0183] It should be noted that although sub-regions B and A have been shown thus far to be contiguous areas, sub-regions B and A may alternatively be non-contiguous areas, such as sets of tiles of image 302. With regard to the specific processing with respect to tiles into which image 302 may be encoded into data stream 300 according to this example, reference is made to the descriptions of Figures 1 through 13.

[0184] With respect to FIG. 23c, note that since picture A is a picture-by-picture random access picture, the picture type of picture A is different from that of the other temporally predicted pictures and therefore picture A will be signaled within picture type signaling 320.

[0185] FIG. 24 shows a video decoder 330 configured to decode a video data stream 300 with the aid of images B and A. The video data stream 300 of FIG. 24 is shown exemplarily in correspondence with the description of FIG. 22. That is, the sub-region-related random access images B and A are interspersed between the images 302 of the video 304 encoded in the video data stream 300. The video decoder 330 is configured to wait for the occurrence of the next random access image when randomly accessing the video data stream 300, that is, to wait for a sub-region-related image or an image-by-image random access image. In the case of a video data stream 300 that does not contain an image-by-image random access image, the video decoder 330 may not even respond to such an image. In any case, the video decoder 330 resumes decoding the video data stream 300 as soon as it encounters the first sub-region-related random access image, which in the example of FIG. 24 is image B. Starting from this image B, the video decoder 330 starts reconstructing, decoding and outputting an image 322 showing only sub-region B. Alternatively, the video decoder 330 will decode, reconstruct and output the portions of these images 322 outside of sub-region B, with or without signaling accompanying these images that indicates to a presentation device, such as a display device, that these portions of images 322 outside of sub-region B are missing a reference image for this outer sub-region and therefore will experience drift error.

[0186] The video decoder 330 continues to decode the video data stream 300 in this manner until it encounters the next random access image, which is image A in the example of FIG. 24. Since sub-region A represents a random access point for the remaining image area, i.e., the area of ​​image 302 outside sub-region B, the video decoder 330 fully decodes, reconstructs and outputs images 302 after image A. That is, the functionality described in relation to FIG. 24 provides the user with the opportunity to get the advantage of watching the video 304 sooner, at least for sub-region B, i.e., for the random access image B associated with the first encountered sub-region. Then, after encountering the next sub-region-related random access image for the sub-region covering the remaining image 302, the video decoder 330 can provide the complete image 302 without drift error. In the example of FIG. 23e, this is the case after encountering the first of each of the sub-region-specific random access images for each of sub-regions A, B and C.

[0187] Figure 25 shows another mode of operation of the video decoder 330. Here, the video decoder 330 starts decoding and reconstructing sub-region B since it encounters the first sub-region specific random access image, here image B, but the video decoder 330 waits until it encounters enough random access images so that the sub-region covers the entire image area of ​​image 302, until the video decoder 330 actually outputs image 302 in its entirety. In this example, this is the case when image A and image B are encountered, i.e., from image A the video decoder 330 outputs image 302, but sub-region B should have been available from image B.

[0188] FIG. 26 shows a network device 322 receiving a video data stream 300 including a sub-region specific random access image. However, now the video data stream 300 is reducible with respect to one or more sub-regions of the sub-region specific random access image. For example, FIG. 26 shows the case where the video data stream 300 is reducible with respect to sub-region B. With respect to the reducibility and the corresponding functionality of the network device 322, it should be noted that this functionality may or may not be configured to correspond to the description of FIGS. 1 to 13. In either case, the image B is shown in the video data stream 300 using the signaling 318 described above. When the network device 322 reduces the video data stream 300 to relate only to the image B, it extracts it from the video data stream 300 or reduces the video data stream 300 to a reduced video data stream 324, whose image 326 forms a video 328 that simply shows the content of the sub-region B of the video 304, i.e. the content of the unreduced video data stream 300. However, as per this measure, since sub-region B no longer represents a sub-region with respect to picture 326, reduced video data stream 324 no longer includes signaling 318. Rather, the picture of video 328 corresponding to sub-region B of sub-region specific random access picture B of the video data stream is signaled in reduced video data stream 324 to be a picture-wise random access picture, such as an IDR picture, by picture type signaling 320. There are various ways to do this. For example, network device 322, if configured to correspond to network device 60, can change picture type signaling 320 in the NAL unit header of the corresponding NAL unit when reducing video data stream 300 towards its reduced video data stream 324, in addition to the redirection and / or parameter set revisions described above with respect to Figures 5 and 7.

[0189] For the sake of completeness, Fig. 27 shows a network device 231 configured to process the data stream 20 of Fig. 20. However, Fig. 20 shows that the information 206' may or may not already be present in the reducible video data stream 200, such as in the information 50. As already mentioned above, via this network device 231 it can be configured to discard the displacement information 206 when reducing the video data stream 200 to the reduced video data stream 232, by simply carrying over the sub-region specific displacement information 206' from the video data stream 200 to the reduced video data stream 232, or the network device 231 can form a readjustment of the displacement information 206, so as to become sub-region specific displacement information 206, based on knowledge of the position of the sub-region 22 with respect to the image of the reducible video data stream 200.

[0190] Thus, the above description has revealed processes and signaling for, for example, extraction of tile sets with temporal motion and inter-layer prediction constraints. Extraction or spatial subsets of encoded video bitstreams using single or multi-layer video coding has also been described.

[0191] With regard to the above description, it should be noted that any encoder, decoder or network device shown may be embodied or implemented in hardware, firmware or software. If implemented in hardware, the respective encoder, decoder or network device may be implemented, for example, in the form of an application specific integrated circuit (ASIC). If implemented in firmware, the respective device may be implemented as a field programmable array, and if implemented in software, the respective device may be a processor or computer programmed to perform the described functions.

[0192] Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with blocks or apparatus corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of the corresponding block or item or feature of the corresponding apparatus. Some or all of the method steps may be performed by (or used in) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0193] The encoded data stream or signal of the present invention can be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless transmission medium or a wired transmission medium such as the Internet. Although the insertion or encoding of some information into the data stream has been described above, this description is at the same time understood as a disclosure that the resulting data stream includes the respective information, flag syntax elements, etc.

[0194] Depending on specific implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be carried out using a digital storage medium, such as a floppy disk (floppy is a registered trademark), DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperates (or can cooperate) with a programmable computer system so that the respective method is executed. The digital storage medium can thus be computer readable.

[0195] Some embodiments according to the present invention comprise a data carrier having electronically readable control signals operable in cooperation with a programmable computer system to perform one of the methods described herein.

[0196] Generally, the embodiments of the present invention may be implemented as a computer program product having a program code operative to perform one of the methods when the computer program product runs on a computer, The program code may for example be stored on a machine readable carrier.

[0197] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0198] In other words, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0199] A further embodiment of the inventive methods is therefore a data carrier (or digital storage medium or computer readable medium) having recorded thereon the computer program for performing one of the methods described herein. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory.

[0200] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals can for example be adapted to be transmitted via a data communication connection, for example the Internet.

[0201] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0202] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0203] Further embodiments according to the present invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0204] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware device.

[0205] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0206] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0207] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0208] The methods described herein, or any components of the apparatus described herein, may be performed at least in part by hardware and / or software.

[0209] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein are obvious to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details shown by the description and explanation of the embodiments herein.

Claims

1. A video data stream representing a video, a parameter setting portion indicating encoding parameter settings; a payload portion in which the image of the video is parameterized and encoded using the first set of encoding parameter settings, the first set being indexed by an index contained in the payload portion; Including, The video data stream includes: a display of a predetermined sub-region of the image; substitution parameters for adjusting the first set of encoding parameter settings to produce a second set of encoding parameter settings, the encoding parameter settings defining buffer sizes and timing; information including The second set of encoding parameters comprises: removing a portion of the payload portion that falls outside the predetermined sub-region of the image; and modifying a location indication within the payload portion such that a location measured from a perimeter of the predetermined sub-region rather than the image indicates a location including a reduced payload portion encoding a sub-region specific image parameterized to indicate the predetermined sub-region of the image using the second set of encoding parameter settings. selecting a reduced video data stream that is modified when compared to the video data stream by The information is included in an SEI message, a VUI, or a parameter set extension. Video data stream.

2. 1. An encoder for encoding a video into a video data stream, comprising: a parameter setter configured to determine encoding parameter settings and to generate a parameter setting portion of the video data stream indicative of said encoding parameter settings; an encoding core configured to encode images of the video, parameterized using the first set of encoding parameter settings, into a payload portion of the video data stream, the first set being indexed by an index included in the payload portion; Equipped with The encoder comprises: a display of a predetermined sub-region of the image; substitution parameters for adjusting the first set of encoding parameter settings to produce a second set of encoding parameter settings, the encoding parameter settings defining buffer sizes and timing; configured to provide information in the video data stream including The second set of encoding parameters comprises: removing a portion of the payload portion that falls outside the predetermined sub-region of the image; and modifying a position indication within the payload portion to indicate a position measured from a perimeter of the predetermined sub-region rather than from the image; a reduced video data stream modified compared to the video data stream by is selected to include a reduced payload portion encoding a sub-region specific image parameterized to illustrate the sub-region of the image using the second set of encoding parameter settings; The encoder is configured to insert the information into a SEI message, a VUI, or a parameter set extension. Encoder.

3. 1. A decoder for decoding a video data stream, the video data stream comprising: a parameter setting portion indicating encoding parameter settings; a payload portion in which an image of a video is parameterized and encoded using said first set of encoding parameter settings, said first set being indexed by an index contained in said payload portion; Equipped with The decoder comprises: a display of a predetermined sub-region of the image; substitution parameters for adjusting the first set of encoding parameters to produce a second set of encoding parameter settings, the encoding parameter settings defining buffer sizes and timing; configured to read information from the video data stream including performing redirection and / or adjustment such that the second set of encoding parameter settings is indexed by an index of the payload portion; removing a portion of the payload portion that falls outside the predetermined sub-region of the image; and modifying the position indicia on the payload portion to indicate a position measured from a perimeter of the predetermined sub-region rather than from the image. reducing the video data stream to a reduced video data stream modified by The decoder is configured to read the information from a SEI message, a VUI, or a parameter set extension of the video data stream. decoder.

Images