Hypothetical reference decoder

Abstract

To provide a video encoder, a video decoder, a network node, an apparatus for managing a coded picture buffer (CPB) of a video decoder, and an apparatus for splicing video data streams together.SOLUTION: A video encoded video data stream 14 includes first timing information relating to a first coded picture buffer (CPB) removal time to be applied at the decoder side when buffering a full version 15 of the video data stream, and second timing information relating to a second CPB removal time to be applied at the decoder side for buffering a reduced version of the video data stream, which differs from the video data stream resulting from removal of a portion of the video data stream from the full version of the video data stream.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The hypothetical reference decoder HRD and its use for checking the conformance of bitstreams and decoders are typically specified in video coding standards such as AVC (Advanced Video Codec) and HEVC (High Efficiency Video Coding).

[0002] Embodiments according to the present invention relate to a video stream, a video encoder, a video decoder, a network node, an apparatus for managing a coded picture buffer (CPB) of a video decoder, and an apparatus for splicing video data streams together. [Background technology]

[0003] The following provides an introduction to the background of the invention. To do this, an HRD buffer model is specified, consisting of a Virtual Stream Scheduler (HSS), a Coded Picture Buffer (CPB), a decoding process (considered instantaneous), a Decoded Picture Buffer (DPB), and an output cropping process, as shown in Figure 2.

[0004] The model defines the timing and bit rate at which the bitstream is fed into the coded picture buffer, the time at which its decoding units (either AUs, access units or VCL NAL (VCL: Video Coding Layer, NAL: Network Abstraction Layer) units in the case of low latency operating mode) are removed from the CPB and instantly decoded, and the output time at which the pictures are output from the DPB.

[0005] By doing so, it is also possible to define the CPB size required for the decoder to avoid buffer overflow (more data being sent to the decoder than can be held in the CPB) or underrun (where less data is sent to the decoder (at a lower bitrate than required) and the required decoding units are not in adequate time at the decoder for decoding).

[0006] State-of-the-art coding standards specify different parameters to describe the bitstream and HRD requirements as well as the buffer model.

[0007] For example, the hrd_parameter defined in HEVC for each sublayer consists of one or more pairs of Bitrate(i) and CPBsize(i), which indicate that no overflow or underflow will occur when the stream scheduler supplies a bitrate of Bitrate(i) to a CPB of size CPBsize(i).

[0008] Along with the hrd_parameter syntax element, there is additional timing information in the bitstream that specifies the removal time of a picture from the CPB, indicating when the VCL NAL units belonging to that picture are sent for decoding. For ease of understanding, sub-image processing is omitted from the description.

[0009] The relevant information is present in the buffering period SEI (Supplemental Enhancement Information) message with InitialCPBRemovalDelay(i), InitialCPBRemovalOffset(i), and AuCPBRemovalDelay, and in the picture timing SEI message with AuCPBRemovalDelay. In the most basic operation, only InitialCPBRemovalDelay(i) and AuCPBRemovalDelay are used.

[0010] In that case, the first access unit decoded is the random access point RAP and its corresponding buffering period SEI message and time 0 is defined as the time when the first bit of the random access point enters the CPB. Then, at time InitialCPBRemovalDelay(i), the picture corresponding to the random access point is removed from the CPB, and further removal of non-RAP pictures from the CPB occurs at InitialCPBRevovalDelay(i)+AuCPBRemovalDelay (HEVC defines some parameters to convert delay to time, namely ClockTick, but this is ignored here for simplicity).

[0011] When the next RAP comes, the removal time is calculated as before for the non-RAP images, i.e. InitialCPBRemovalDelay(i)+AuCPBRemovalDelay This new value is then used as the anchor for further deltas up to another RAP, i.e. anchorTime=InitialCPBRemovalDelay(i)+AuCPBRemovalDelay

[0012] Then, the image removal is anchorTime+AuCPBRemovalDelay, and anchorTime is updated in the buffering SEI message in the next RAP. For example, anchorTime=anchorTime+AuCPBRemovalDelay. The described mode of operation is the simplest one. There are further cases that need to be considered. In the current situation, there are some drawbacks that the present invention overcomes, which are detailed as follows: Summary of the Invention

[0013] One embodiment according to the present invention relates to a video data stream in which video is encoded and comprises first timing information relating to a first Coded Picture Buffer (CPB) removal time that is applied on the decoder side when buffering a full version of the video data stream, and second timing information relating to a second CPB removal time that is applied on the decoder side for buffering a reduced version of the video data stream that differs from the video data stream resulting from removal of a portion of the video data stream from the full version of the video data stream.

[0014] Another embodiment according to the invention relates to a video data stream in which video is encoded and which comprises first HRD timing information data relating to a minimum CPB supply bitrate and second HRD timing information data relating to a maximum CPB supply bitrate, wherein the first HRD timing information data and the second HRD timing information data enable determination of a third HRD timing information of an actual CPB supply bitrate by linearly interpolating between the first and second HRD timing information data.

[0015] Another embodiment according to the invention relates to a video data stream comprising: a sequence of image portions, each image portion having an image of encoded video, the image portions comprising a first type image portion and a second type image portion that serve as a CPB removal time reference; first timing information for each image portion relating to the respective image portion's first CPB removal time, the first CPB removal time measuring the time elapsed since the removal of a preceding first type image portion; second timing information for each of the predetermined first type image portions relating to the respective predetermined first type image portion's second CPB removal time when resuming decoding of the video data stream after the respective predetermined first type image portion, the second CPB removal time measuring the time elapsed since the CPB arrival of the first bit of the respective predetermined first type image portion; and a splice point indication indicating an image portion for which the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold, so as to serve as the last image portion before a splice point with another video data stream.

[0016] Another embodiment according to the present invention relates to a video encoder for encoding video into a video data stream and providing the video data stream with first timing information relating to a first CPB removal time to be applied on the decoder side when buffering a full version of the video data stream, and second timing information relating to a second CPB removal time to be applied on the decoder side for buffering a reduced version of the video data stream, which differs from the video data stream resulting from removal of a portion of the video data stream from the full version of the video data stream.

[0017] Another embodiment according to the present invention relates to a video encoder for encoding video into a video data stream and providing the video data stream with first HRD timing information relating to a minimum CPB supplied bitrate and second HRD timing information relating to a maximum CPB supplied bitrate, wherein the first HRD timing information and the second HRD timing information enable determination of third HRD timing information of an actual CPB supplied bitrate by linearly interpolating between the first and second HRD timing information.

[0018] Another embodiment according to the invention is a video encoder for encoding video into a video data stream such that the video data stream comprises a sequence of image portions, each image portion having an image of the encoded video, the video encoder classifying the image portions into first type image portions and second type image portions that serve as CPB removal time references, and providing in the video data stream first timing information for each image portion relating to a first CPB removal time of the respective image portion, the first CPB removal time measuring an elapsed time since removal of a preceding first type image portion, and a time when to resume decoding of the video data stream after each predetermined first type image portion. and second timing information for each of the predetermined first-type image portions relating to a second CPB removal time of the image portion of said type, the second CPB removal time measuring the elapsed time from the CPB arrival of the first bit of the respective predetermined first-type image portion; checking each of the predetermined first-type image portions as to whether a time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold so as to serve as the last image portion before a splice point with another video data stream; and providing a splice point indication in the video data stream to indicate those predetermined image portions for which the time difference between their final CPB arrival and their CPB removal exceeds the predetermined threshold.

[0019] Another embodiment according to the invention relates to a network node for forwarding a video data stream to remove a portion of the video data stream from the video data stream.

[0020] Another embodiment according to the invention relates to an apparatus for managing a CPB of a video decoder that decodes a video data stream in a CPB-buffered manner, the apparatus managing the CPB according to first timing information when a portion is constituted by the video data stream and according to second timing information when a portion is removed.

[0021] Another embodiment according to the present invention relates to an apparatus for managing a CPB of a video decoder that decodes a video-encoded video data stream, the apparatus deriving, from the video data stream, first HRD timing information related to a minimum CPB supply bitrate, second HRD timing information related to a maximum CPB supply bitrate, determining third HRD timing information of an actual CPB supply bitrate by linearly interpolating between the first and second HRD timing information, and managing the CPB using the third HRD timing information.

[0022] Another embodiment according to the present invention relates to an apparatus for managing a CPB of a video decoder that decodes a video data stream in a CPB-buffered manner, the apparatus checking whether a concatenation flag of a first predetermined first-type image portion indicates that a splice-in has occurred in the first predetermined first-type image portion and determining a time to remove the first predetermined first-type image portion from the CPB, wherein the determination is based on first timing information of the first predetermined first-type image portion if the concatenation flag of the first predetermined first-type image portion indicates that a splice-in has not occurred in the first predetermined first-type image portion, and based on second and third timing information of the first predetermined first-type image portion if the concatenation flag of the first predetermined first-type image portion indicates that a splice-in has occurred in the first predetermined first-type image portion.

[0023] Another embodiment according to the invention is an apparatus for splicing together a first video data stream and a second video data stream, each of which comprises a sequence of image portions, each image portion having an image of the coded video, the image portions comprising a first type image portion and a second type image portion serving as a CPB removal time reference; first timing information for each image portion relating to a first CPB removal time of the respective image portion, the first CPB removal time measuring an elapsed time since removal of a preceding first type image portion; and first timing information for a predetermined first type image portion relating to a second CPB removal time of each predetermined first type image portion when resuming video data stream decoding after each predetermined first type image portion. and second timing information for each of the image portions of the first predetermined first type, wherein the second CPB removal time measures the elapsed time from the CPB arrival of the first bit of the respective image portion of the first predetermined first type, the second video data stream comprising a concatenation flag and third timing information for a first image portion of the first predetermined first type, the concatenation flag being set to a second state indicating the second video data stream at the first image portion of the first predetermined first type continuing the second video data stream, the third timing information indicating a third CPB removal time for the first image portion of the first predetermined first type that is useful for determining a CPB removal time when splicing in the second video data stream at the first image portion of the first predetermined first type, and the third CPB removal time measures the elapsed time from the most recent CPB removal of the non-discardable image portion.The apparatus of the present invention checks whether the splice point indication of the first video data stream indicates that, for a predetermined image portion, the time difference between its CPB arrival and its CPB removal exceeds a predetermined threshold, so that the predetermined image portion can serve as the last image portion before the first video data stream is spliced with the second video data stream; if so, sets the concatenation flag of the first predetermined first type image portion of the second video data stream to a first state indicating that the second video data stream has been spliced in at the first predetermined first type image portion; and concatenates the first and second video data streams at the predetermined image portion and the first predetermined first type image portion, respectively, to obtain a spliced video data stream.

[0024] The above concepts can be implemented by methods according to embodiments of the present invention. These methods are based on the same considerations as the decoder, encoder, device, and data stream described above. However, it should be noted that the methods can be supplemented by any of the features, functions, and details described herein, as well as the decoder, encoder, device, and data stream. Also, the methods can be supplemented by the features, functions, and details of the decoder, encoder, device, and data stream, either individually or in combination.

[0025] Finally, the concepts can also be used to generate coded data streams according to embodiments of the present invention, which data streams can be supplemented by features, functions and details of decoders, encoders, devices and methods, either individually or in combination. Embodiments in accordance with the present invention will now be described with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 2 illustrates an encoded data stream according to an embodiment of the present application. [Figure 2]1 is a flowchart of a hypothetical reference decoder according to an embodiment of the present application; [Figure 3] FIG. 10 illustrates the state of a coded image buffer including removal times according to an embodiment of the present application. [Figure 4] FIG. 10 illustrates the state of a coded image buffer including removal times according to an embodiment of the present application. [Figure 5] FIG. 10 illustrates the state of a coded image buffer including removal times according to an embodiment of the present application. [Figure 6] FIG. 10 illustrates the state of a coded image buffer including removal times according to an embodiment of the present application. [Figure 7] FIG. 10 illustrates the state of a coded image buffer including removal times according to an embodiment of the present application. [Figure 9] FIG. 10 illustrates the state of a coded image buffer including removal times according to an embodiment of the present application. [Figure 8] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 10] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 11] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 20] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 21] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 22] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 23] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 24a] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 24b] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 26] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 27] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 28] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 29] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 30] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 31] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 32] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 33] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 34] FIG. 2 illustrates an exemplary signaling syntax according to an embodiment of the present application. [Figure 12] FIG. 2 illustrates a data stream structure for different frame rates according to an embodiment of the present application. [Figure 13] FIG. 2 illustrates a data stream structure for different frame rates according to an embodiment of the present application. [Figure 14] FIG. 2 illustrates a data stream structure for different frame rates according to an embodiment of the present application. [Figure 16] FIG. 2 illustrates a data stream structure for different frame rates according to an embodiment of the present application. [Figure 18] FIG. 2 illustrates a data stream structure for different frame rates according to an embodiment of the present application. [Figure 15] FIG. 10 illustrates exemplary removal time values when different frame rates are provided in the same bitstream, according to an embodiment of the present application. [Figure 17] FIG. 10 illustrates exemplary removal time values when different frame rates are provided in the same bitstream, according to an embodiment of the present application. [Figure 19]FIG. 10 illustrates exemplary removal time values when different frame rates are provided in the same bitstream, according to an embodiment of the present application. [Figure 25] FIG. 1 illustrates coded data streams being stitched together according to an embodiment of the present application. [Figure 35] FIG. 10 illustrates a table of values for CPB size calculation according to an embodiment of the present application. [Figure 36] FIG. 10 illustrates a table of values for CPB size calculation according to an embodiment of the present application. [Figure 37] FIG. 10 illustrates a table of values for CPB size calculation according to an embodiment of the present application. [Figure 38] 1 is a graph of an exemplary CPB fill level according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0027] In the drawings, like reference numbers indicate like elements and features.

[0028] In the following, the considerations underlying the present invention are discussed and some solutions are described. In particular, some details are disclosed, which can be optionally introduced into any of the embodiments disclosed herein. Specifically, the following description begins with a brief presentation of the problems that arise and are faced when dealing with splicing, stream extraction, stripping of scalable bitstreams, and bitstream provisioning in various transmission conditions. Next, specific fixes for the problems are presented, followed by embodiments that utilize the corresponding fixes. The following issues are examined and addressed:

[0029] Splicing: Add AuCPBRemovalDelay so that the removal time of the CPB of the image is used, while the removal time of the previous RAP with buffering SEI is used as an anchor.

[0030] However, when splicing occurs, it is not easy to know what the previous value of anchorTime was at the splicing point, so in such cases, the derivation of the CPB removal time for a RAP with a spliced buffering SEI is performed differently.

[0031] To solve this problem, HEVC includes two additional parameters: concatenation_flag, which indicates whether splicing occurred in the RAP containing the buffering period SEI message, and AuCPBRemovalDelta, which refers to the delta in time to the previous non-discardable picture instead of referring to the previous RAP with the buffering period SEI message. This therefore eliminates the need to calculate any value and rewrite the buffering period SEI message at the splicing point. Then, the CPB removal time from the RAP where splicing occurs is as follows: Removal(previousNonDiscardable)+AuCPBRemovalDelta

[0032] This assumes a seamless splicing scenario. However, it cannot guarantee that the CPB status is the same at the splicing point of the original bitstream (when the RAP with the SEI message results in an unspliced RAP in the original bitstream during the buffering period of splicing) and when splicing. More specifically, Figure 3 illustrates the described problem, where the top bitstream in frame #3 is spliced with the bottom bitstream from frame #2 onwards.

[0033] Therefore, (frame t af 3) as mentioned above Removal(previousNonDiscardable)+AuCPBRemovalDelta Instead of having a removal time of Removal(previousNonDiscardable)+InitialCPBRevovalDelay(i)+ taf 2-t rm 2 The removal time is used instead.

[0034] That is, the removal time of spliced RAP is Removal(previousNonDiscardable)+SplicingDelta where SplicingDelta is AuCPBRemovalDeta and InitialCPBRevovalDelay(i)+t af 2-t rm It is set to a maximum value of 2. If the second value happens to be greater than the first value, for example as shown in FIG. 3, non-seamless splicing occurs.

[0035] AuFinalArrivalTime and the removal time of another image following a NonDiscardable image are unknown, therefore both values taf X and t rm If the difference in X is not the same as that for NonDiscardable images, the derived removal time is incorrect (InitialCPBRemovalDelay(i)+t af Xt rm X) possibility, this splicing operation only works if the first bitstream ends with a corresponding NonDiscardable picture.

[0036] CPB Fullness and Delay: Another problem that arises in the HRD model is that a time delay is required for the removal of the first access unit in order to utilize the CPB buffer.

[0037] When the first RAP after splicing enters a CPB, the removal of that AU is delayed until after the InitialCPBRemovalDelay(i) time, as discussed previously. By feeding a CPB at Bitrate(i) during InitialCPBRemovalDelay(i), a given CPB fullness, e.g., CPB A is achieved. Then, in a further RAP with a buffering period SEI message, A CPB greater than B It is impossible to achieve this. The reason for this can be seen by mathematically checking the earliest time an AU can enter the CPB.

[0038] initArrivalTime[n]=Max(AuFinalArrivalTime[n-1],initArrivalEarliestTime[n]), where initArrivalEarliestTime[n]=RemovalTime[n]-InitCpbRemovalDelay(i)

[0039] This means that if the AU following the buffering period SEI message cannot enter the CPB earlier than its removal time InitialCPBRemovalDelay(i), then it is not possible to supply Bitrate(i) to the CPB during InitialCPBRemovalDelay(i). A Only by achieving CPB fullness can CPB A CPB greater than B This means that it is not possible to achieve

[0040] To solve this problem, it is conceivable that the virtual sender (or HSS in the figure) delays the scheduling of the first RAP with a buffered SEI message by a given time offset InitialCPBRemovalOffset(i), as shown in Figure 4. It's important to mention that this only works with VBR, not CBR as in CBR below. initArrivalTime[n]=AuFinalArrivalTime[n-1] This changes the scheduling as follows:

[0041] initArrivalEarliestTime[n]=RemovalTime[n]InitCpbRemovalDelay(i)-InitialCPBRemovalOffset(i)

[0042] This is CPB B This means that the CPB size of can correspond to the size achieved by supplying Bitrate(i) to the CPB for InitCpbRemovalDelay(i)+InitialCPBRemovalOffset(i).

[0043] Dropping AUs: The HEVC HRD model also supports dropping some NAL units. More specifically, "discardable" pictures associated with RAP pictures can be dropped. "Discardable" pictures refer to RASL (Random Access Skip Leading) pictures, i.e., pictures that precede the RAP picture in presentation order in an open GOP (Group of Pictures) structure but follow it in decoding order, and also refer to pictures preceding the RAP in decoding order. Therefore, when randomly accessing the corresponding RAP picture, the RASL picture cannot be decoded. Therefore, when splicing such a bitstream with another bitstream or starting a session at that RAP position, transmitting the RASL picture is a waste of resources, and in some cases, these pictures are dropped before transmission. Obviously, when the bitstream is modified, the timing information, i.e., the initial arrival time and removal time of the pictures, changes.

[0044] Thus, the buffering period SEI message includes alternative timings for when RASL pictures associated with the RAP having the buffering period SEI message are dropped. AltInitCpbRemovalDelay(i) ·AltInitialCPBRemovalOffset(i)

[0045] Temporal scalability: When dropping sublayers, the timing information also changes. For example, if the original bitstream has 60 Hz, the nominal removal time distance between two consecutive pictures in decoding order is expected to be 1 / 60 second. Dropping every other picture results in a distance of 1 / 30 second. This means that the picture timing SEI messages need to be replaced. Furthermore, InitCpbRemovalDelay(i) and InitialCPBRemovalOffset(i) also need to be changed. Therefore, the buffering period SEI messages need to be replaced. For such operations, additional picture timing SEI messages and buffering period SEI messages are typically included in so-called nesting SEI messages. When the middleware performs sublayer bitstream extraction (e.g., dropping every other frame to obtain a 30 Hz bitstream from a 60 Hz bitstream), the original picture timing SEI messages and buffering period SEI messages are replaced with the corresponding picture timing SEI messages and buffering period SEI messages from the nesting SEI message.

[0046] Bitrate Variation: HRD parameters such as initial_removal_delay and initial_removal_delay_offset, as well as bitrate and CPB_size, are typically provided for several values. The bitrate at which a given bitstream is provided in the CPB may vary, and therefore several values that result in a valid HRD model may be provided.

[0047] However, there are some aspects that have not been adequately considered. There can only be bitrates that result in CBR. If the appropriate Bitrate is not known a priori, the bitstream cannot provide a valid HRD model (parameters for it).

[0048] As outlined above, splicing non-Discardable points is currently a difficult task to achieve, and therefore the embodiments described below aim to achieve exactly the possibility of splicing even at non-Discardable points. A first simple modification is to correct the formula so that it does not always use a non-discardable image as the anchor, but instead uses the last image received after splicing, i.e. Removal(previousNonDiscardable)+SplicingDelta

[0049] where SplicingDelta is the sum of AuCPBRemovalDelta and InitialCPBRevovalDelay(i)+t af 2-t rm Set to a maximum value of 2 from, -Removal(previousNonDiscardable)+AuCPBRemovalDelta

[0050] -Removal(lastPictureBeforeSplicing)+InitialCPBRevovalDelay(i)+t af 2-t rm 2. Here, Removal(lastPictureBeforeSplicing) is actually t rm 2 Change to the maximum value of (in the example above).

[0051] Still, as can be seen in the formula, the splicer must split the previous image into the last image before splicing, e.g., t rm If we decide to consider it as 1, the maximum value of both cases will always be Removal(previousNonDiscardable)+AuCPBRemovalDelta. This means that the splicer will change the value of AuCPBRemovalDelta accordingly, and (t af 2-t rm Unless one ensures that the difference in 2) is such that seamless switching is possible, it will prevent seamless splicing at any AU.

[0052] It is possible to change the value of AuCPBRemovalDelta during the buffering period SEI. However, (t af X and t rm X) is not so easy to keep track of, so the splicer af X and t rm X) is not easy to track.

[0053] Since seamless splicing at different points is desirable, some signaling to indicate to the splicer whether a given image allows this or not would be desirable. This can be done on the encoder side by (t af X and t rm X) values and calculate the difference (t rm Xt af This can be achieved by making sure that X) is not smaller than a given value. The signaling can be done, for example, in the picture timing SEI associated with the picture that can be used as the last one before the splicing point. An example can be seen in Figure 26. Alternatively, even if the image is non-discardable as promised, subsequent discardable images will still meet the requirements. An example can be seen in Figure 27.

[0054] (t rm Xt afThe requirement for the minimum value that X) must meet is related to the initial_removal_delay of the spliced bitstream. In the buffering period SEI there should be some indication as to at what value the picture can be used for seamless splicing. An example can be seen in Figure 28. Therefore, as a result of consideration of the possible modifications just outlined, the following embodiments were evaluated.

[0055] In Figure 25 and the signaling examples of Figures 26, 27, and 28, it can be seen that a video data stream can comprise a sequence of image portions 23a, such as access units, and each image portion can have an image 10a of the encoded video 12a. Figure 25 depicts two data streams 14a and 14b and shows the splicing of streams 14a and 14b at splicing point 92, such that the trailing end of the portion of stream 14a preceding, i.e., to the left of, the splicing point abuts the leading end of the portion of stream 14b following, i.e., to the right of the splicing point 92. The result of the splicing performed by a splicing device, such as a server, a video conferencing system, or a client-server streaming system, i.e., the spliced data stream, is shown at 14c.

[0056] The image portions may comprise first-type image portions, which are shown cross-hatched, or simply hatched in some figures, and referred to above as mark images. An example for this is an AU that has or comprises a buffering period SEI. These first-type image portions serve as CPB removal time references, i.e., their CPB removal is used as a reference anchorTime for other timings given as time offsets, such as AuCPBRemovalDelay.

[0057] The image portion may further comprise an image portion of a second type, shown without hatching in some figures, such as an image without a buffering period SEI.

[0058] The data streams 14a and 14b additionally comprise first timing information 95, which compares AuCPBRemovalDelay or au_cpb_removal_delay_minus1 for each image portion and indicates the first CPB removal time of the respective image portion, which measures the elapsed time since the removal of the preceding first type image portion, i.e., the one that precedes it in coding or bitstream order as depicted in Figure 25, specifically the one immediately preceding it.

[0059] The data streams 14a and 14b further comprise second timing information for comparing the initial_removal_delay, which may be present in the picture timing syntax of the buffering period SEI, for example, before a splice point indication discussed below, in each of the predetermined first-type picture portions. These predetermined first-type picture portions are simply hatched and are shown with cross-hatching in FIG. 25 to distinguish them from other first-type picture portions that are specific first-type picture portions, such as an AU with a buffering period SEI indicated to be a RAP, or in other words, an AU with a buffering period SEI indicated to be a RAP. The second timing information informs the second CPB removal time of each predetermined first-type picture portion when resuming decoding of the video data stream after the respective predetermined first-type picture portion. The second CPB removal time measures the elapsed time from the CPB arrival of the first bit of each predetermined first-type picture portion, i.e., the arrival of the CPB at the decoder.

[0060] At least one of the data streams, 14a in FIG. 25, i.e., the one to be split or separated to obtain another stream, here 14b, added at the splicing point 92, may further comprise a splice point indication 94, 94′, e.g., spliceable_flag or following_pic_spliceable_flag, which indicates the time between its final CPB arrival and its CPB removal as indicated above, so as to be able to function as the last picture portion before the splice point 92 with the other video data stream 14b, i.e., to be suitable for enabling splicing. rm #-t af # indicates an image portion where the time difference exceeds a predetermined threshold, e.g., no RAP. It should be noted that the threshold value can be calculated, for example, as follows:

[0061] The removal times are assumed to be equidistant and equal to 1 / frame rate. The desired removal time for a newly spliced AU, such as AU23b' in Figure 25, is t rm # + 1 / frame rate, t rm # is the removal time of the previous AU or image part, and is indicated by 23'a.

[0062] t rm #+1 / frame rate is t af # Must be greater than or equal to +initial_removal_delay, t af # is the final arrival time of the CPB of each AU 23b'. Therefore, the "predetermined threshold" is initial_removal_delay-1 / frame rate.

[0063] And in the variant of max_val_initial_removal_delay_for_seamless_splicing, i.e., where this threshold is explicitly indicated in at least one of the data streams, for example, data stream 14a to which another stream 14b is added, t rm#+1 / frame rate is t af #+max_val_initial_removal_delay_for_ _seamless_splicing or more. Then, the "predetermined threshold" is max_val_initial_removal_delay_for_seamless_splicing-1 / frame rate.

[0064] That is, the video data stream 14a may also comprise an indication 99 of a maximum second CPB removal time value, which indicates that as long as the second CPB removal time of the first-type picture portion 23b′ at the start of another video data stream 14b, which is spliced with the video data stream 14a at the splice point 92, is less than the maximum second CPB removal time value, the time difference t between its final CPB arrival and its CPB removal. rm #-t af # indicates that concatenating another video data stream 14b with any of the image portions indicated by the splice point indicators as exceeding a predetermined threshold will result in seamless splicing.

[0065] Note that the example buffering period SEI shown above includes an indication 99, a concatenation flag 95, and a third CPB removal time 98. This illustrates an example where both streams 14a and 14b carry the same type of information data. In other examples, this may be different.

[0066] The splice point indications 94, 94' represent for each image portion the time difference t between its final CPB arrival and its CPB removal. rm #-t af There may further be a flag 94 present in each image portion or each set of image portions, for example without RAP, that indicates whether the # exceeds a predetermined threshold.

[0067] Alternatively or additionally, the splice point indication 94, 94' may further comprise, for each image portion or each set of image portions, a flag 94' present for those image portions that are not RAPs, for example, which flag indicates for each subsequent image portion of the first type a time difference t between its final CPB arrival and its CPB removal. rm #-t af # indicates whether or not the value exceeds a predetermined threshold.

[0068] The sequence of image portions may further comprise images of the video 12a that have been coded using temporal inter-prediction 90, such as motion compensated prediction, such that the image portions comprise non-discardable image portions, which are shown without underlining in some figures, such as Figure 25, and discardable image portions, which are shown underlined therein.

[0069] These non-discardable image portions are decodable in a stand-alone manner, while discardable image portions are not required to decode non-discardable image portions, but may require non-discardable image portions to be decodable themselves. Note that if a discardable image portion is not decoded, the video can be decoded without issue in the future.

[0070] That is, discardable image portions, for example, do not serve as reference images for temporal inter-prediction, while non-discardable images may serve as reference images for temporal inter-prediction. A data stream coded using hierarchical temporal scalability represents an example of such image portions, where image portions of the lowest (base) temporal layer may be non-discardable, while others may be discardable. Another example is an open GOP coded data stream, where RASL images may be non-discardable.

[0071] The video data stream 14c has a time difference t between its final CPB arrival and its CPB removal. rm #-t afThe predetermined image portion 23a', for which the splice point indication 94;94' indicates that # exceeds a predetermined threshold, is followed by and spliced at splice point 92 so as to abut the first predetermined first type image portion 23b' from the splice-in video data stream 14b.

[0072] The video data stream 14c includes a concatenation flag 96 and third timing information 98 at the first predetermined first-type image portion, and AuCPBRemovalDelta or au_cp_removal_delay_delta_minus1 are compared. The concatenation flag 96 and third timing information 98 may have already been present in the spliced-in data stream 14b before splicing, but the splicing device may have reset the flag 96 from a state indicating no splicing to a state indicating splicing and / or set the third timing information 98. That is, the concatenation flag 96 is set to a first state indicating that the video data stream is spliced at the first predetermined first-type image portion 23b', and the third timing information indicates a third CPB removal time for the first predetermined first-type image portion 23b', which is useful for determining the CPB removal time when splicing at the first predetermined first-type image portion 23b'. The third CPB removal time measures the elapsed time since the most recent CPB removal of a non-disposable image portion. For example, since the removal of the most recent non-discardable image portion of the video data stream that has been encoded into the spliced data stream 14c.

[0073] Optionally, each of the predetermined first type image portions, shown hatched in some figures, may comprise a concatenation flag 96 and third timing information 98, the concatenation flag being settable to a first and a second state, the second state indicating that the video data stream is not spliced at the respective predetermined first type image portion.

[0074] A video encoder according to this embodiment can encode video into a video data stream in the following manner. For example, assume that such an encoder encodes data stream 14a. It can also be configured to encode data stream 14b, which can be interpreted in the same way as described above. The encoder performs the encoding so that video data stream 14a comprises a sequence of image portions 23a, each image portion having an image 10a of encoded video 12a. The video encoder can then classify the image portions into a first type of image portion that serves as a CPB removal time reference and the second type of image portion described above. The encoder can provide first and second timing information to the video data stream and check, for each predetermined image portion (e.g., these may include discardable image portions), whether the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold. If so, each predetermined image portion can serve as the last image portion before splice point 92 with another video data stream 14b. The video encoder may further provide splice point indications 94, 94' in the video data stream 14a.

[0075] The video encoder may further use, for each predetermined image portion, as a predetermined threshold value determined based on the second CPB removal time of the first type of predetermined image immediately preceding the respective predetermined image portion, for example, the one simply hatched in some of the figures preceding 23a'.

[0076] The video encoder may also use, for each predetermined image portion, a value determined based on the maximum second CPB removal time value as a predetermined threshold value, and write an indication 99 of the maximum second CPB removal time value in the first type of predetermined image immediately preceding each predetermined image portion, for example, the one simply hatched in the figure preceding 23a'.

[0077] The following describes how the spliced data stream described above can be processed within a decoder, such as by an HRD 46 therein, or in other words, by a device 46 for managing a CPB 48 of a video decoder 44. The video decoder 44 decodes the video data stream 14c so that it is buffered by the CPB, i.e., receives the image portion via the CPB at the removal time, and the image portion arrives completely at the CPB at the aforementioned final arrival time. The device can check whether the concatenation flag 96 of the first predetermined first-type image portion 23b' indicates that a splice-in has occurred at the first predetermined first-type image portion. The device can then determine the time to remove the first predetermined first-type image portion 23b' from the CPB.

[0078] The time to be removed can be determined based on the first timing information 95 of the first predetermined first type image portion 23b' if the concatenation flag of the first predetermined first type image portion indicates that no splice-in has occurred in the first predetermined first type image portion.

[0079] The time to be removed can then be determined based on the second timing information, e.g., initial_removal_delay, of the first predetermined first type image portion 23b' and the third timing information 98 of the first predetermined first type image portion when the concatenation flag 96 of the first predetermined first type image portion indicates that a splice-in has occurred in the first predetermined first type image portion.

[0080] Optionally, the device calculates a time difference t between the latest CPB removal of the non-disposable image portion plus the third CPB removal time of the first predetermined first type image portion, and the CPB removal of the predetermined image portion 23 a′ that the first predetermined first type image portion subsequently abuts plus the second CPB time minus the time difference t between the final CPB arrival and CPB removal of the predetermined image portion 23 a′. rm #-t afThis can be done by determining the maximum value between # and #. This time is then used to remove the first predetermined first type image portion 23b' from the CPB.

[0081] Also according to this embodiment, an apparatus is described for splicing together two video data streams, here 14a and 14b to generate 14c, depicted in Figure 25, each comprising a sequence of image portions 23a,b, each image portion having an image 12a,b of the encoded video 12a,b, as described above. The image portions also comprise first and second types of image portions and first and second timing information as described above.

[0082] Next, the second video data stream comprises a concatenation flag 96 and third timing information 98 for the first predetermined first-type image portion. The concatenation flag 96 is set to a second state indicating the second video data stream and the first predetermined first-type image portion continues the second video data stream, and the third timing information indicates a third CPB removal time for the first predetermined first-type image portion that is useful for determining a CPB removal time when splicing in the second video data stream at the first predetermined first-type image portion, and the third CPB removal time measures the elapsed time since the most recent CPB removal of the non-disposable image portion.

[0083] The device can then check whether the splice point indication 94; 94' of the first video data stream indicates that for a given image portion without, for example, a RAP, the time difference between its CPB arrival and its CPB removal exceeds a predetermined threshold, so that the first video data stream can serve as the last image portion before being spliced with the second video data stream.

[0084] For example, in one variant, the predetermined threshold is determined based on the initial_removal_delay of the buffering SEI of the first stream 14a or based on max_val_initial_removal_delay_for_seamless_splicing].

[0085] If the result of this check is positive, i.e., "yes", the device sets the concatenation flag 96 of the first predetermined first type image portion of the second video data stream to a first state indicating that the second video data stream has been spliced in with the first predetermined first type image portion, and can concatenate the first and second data streams of the predetermined image portion 26a' and the first predetermined first type image portion 26b', respectively, to obtain a spliced video data stream.

[0086] Also, if so, the device can rewrite the third timing information 98 to measure the elapsed time since the most recent CPB removal of a non-disposable image portion of the first video data stream in the video data stream spliced instead in the second video data stream.

[0087] Also, if so, the device can further check whether the elapsed time since the most recent CPB removal of a non-discardable image portion before the first predetermined first type image portion has changed when considering the spliced video data stream compared to the second video data stream, and if so, can rewrite the third timing information 98 to measure the elapsed time since the most recent CPB removal of a non-discardable image portion of the first video data stream in the spliced video data stream. In general, the device may check the following checks and perform configuration and connection if both checks are positive:

[0088] First, whether the upper limit equal to or determined from the second CPB removal time of the nearest first-type predetermined image portion in the first video data stream 14a preceding each predetermined image portion is greater than the second CPB removal time of the first predetermined first-type image portion 23b' in the second video data stream. The nearest predetermined image portion of the first type in the first stream 14a is shown simply hatched in some of the figures.

[0089] The second CPB removal time of the nearest first type predetermined image portion may also include an additional CPB supply deferral time of the nearest first type predetermined image portion, which measures the elapsed time by which the CPB arrival of the first bit of the nearest first type predetermined image portion is delayed. Each predetermined image portion is simply hatched in some of the figures preceding, for example, 23a'.

[0090] The second check is whether the maximum second CPB removal time value 99 shown in the most recent predetermined first type image portion is greater than the second CPB removal time of the first predetermined first type image portion 23b' of the second video data stream.

[0091] For example, in one variant, the predetermined threshold may be only the initial_removal_delay of the buffering SEI of the first stream. Alternatively, the threshold may be initial_removal_delay+initial_removal_delay_offset, i.e., the CPB delivery deferral time. Further alternatively, a maximum value max_val_initial_removal_delay_for_seamless_splicing can be transmitted, as described in the second option.

[0092] For example, AuCPBRemovalDelta only needs to be rewritten if the original value, which is the distance to the non-discardable image in the original second bitstream 14b, is different from the distance in the spliced bitstream 14c.

[0093] For example, it may be assumed that both bitstreams have the same frame rate and that the following bitstreams, shown in decoding order, are concatenated, i.e., spliced: Second bitstream: RAP1, B0, B1 (non-discardable), B2, RAP2 (splicing_point) First bitstream: rap1, b0 (non-discardable), b1, b2, b3, b4 Splice with each other: rap1, b0 (cannot be discarded), b1, b2, b3, RAP2 (splicing_point)

[0094] Originally, the buffering period SEI of RAP2 includes concatenation_flag set to 0 and AuCPBRemovalDelta equal to 2 / frame rate.

[0095] In the spliced bitstream, concatenation_flag is set to 1 and AuCPBRemovalDelta is equal to 4 / framerate.

[0096] However, if the spliced bitstream is rap1, b0 (non-discardable), b1, RAP2 (splicing_point), then concatenation_flag is set to 1 and AuCPBRemovalDelta is equal to 2 / frame rate. Therefore, in the first case, AuCPBRemovalDelta needs to be rewritten, but in the second case, it is not necessary.

[0097] For streams that are spliced and that contain an initial removal delay smaller than max_val_initial_removal_delay_for_seamless_splicing, seamless splicing can be achieved if a flag in the picture timing SEI indicates so. If the initial removal delay of the spliced stream is higher, it is clearly not feasible to know whether splicing is possible or not.

[0098] While the above embodiments have been related to the problem of enabling splicing at non-discardable points, the following description addresses the problem of how available removal time can also be made available for removable image parts such as DRAPs. In particular, the subsequent embodiments relate to providing timing information (removal time) when dropping some AUs is performed, such as in the case of RASL images with an open GOP structure when random access or splicing scan is performed. In other words, whether removal time is also applicable to dependent random access points, DRAPs, images.

[0099] The idea here is to modify the initial_removal time of the buffering period SEI of the RAP. In the following different cases are discussed and examples of removal times of AUs are given.

[0100] Figure 1 illustrates the inventive concept of the present application. A video 12 is encoded and a first CPB removal time t is applied at the decoder side when buffering a complete version 15 of the video data stream 14. rm (1) and a second CPB removal time t applied at the decoder side to buffer a reduced version 20 of the video data stream 14, which differs from the video data stream due to removal 22 of a portion 24 of the video data stream from the full version of the video data stream. rm (2) 1 and 2. A video data stream is depicted having second timing information 18 relating to the video data stream.

[0101] Optionally, the second timing information 18 includes a second CPB removal time t rm (2) The first CPB removal time t rm (1) The timing correction information on how to correct the second CPB removal time t rm (2) An example of such timing correction information is, for example, init_removal_delay_correction_offset.

[0102] Furthermore, the first timing information 16 may, for each image, increment the first CPB removal time t for the image relative to the preceding marked image in decoding order 26. rm (1) can be signaled.

[0103] The increment can be exemplarily signaled by AuCPBRemovalDelay or au_cpb_removal_delay_minus1. Mark images may for example be at clean random access points CRA and are images that include a buffering period SEI. These are typically RAP images, but may also be other required images, for example temporal layer 0 images.

[0104] In other words, each picture portion 23 of the video data stream comprises an increment that measures the delay of its removal from the CPB 48 relative to the RAP picture that precedes it along the coding order 26. A picture portion 23 is also called an access unit AU, such as in the case of HEVC. Optionally, the marked image is marked by a buffering period message of the video data stream in an image portion associated with the marked image.

[0105] Furthermore, the device 46 for managing the CPB 48 of the video decoder 44 that decodes the video data stream can manage the CPB according to the first timing information 16 if the portion is constituted by the video data stream, and according to the second timing information 18 if the portion 24 is removed. The complete bitstream is shown in Figure 5. The initial removal time of the clean random access point CRA is 8.

[0106] In Figure 7, the RASL image is removed. The initial removal time of the CRA is 7. The removal times of subsequent images have a delta offset of 4 compared to the previous case.

[0107] Figure 6 shows the removed RASL image and the TRAIL (normal trailing) image up to (but not including) TRAIL4, which is the DRAP. The initial removal of the CRA is now 11, so that the CPB level is the same as if the bitstream had been there from the beginning.

[0108] One option would be to add alternative timings for different possibilities to drop a frame: one for removing the RASL image, one for removing the first DRAP, one for removing the second DRAP, etc.

[0109] However, this increases the size of the buffering period SEI and requires the receiver to indicate in which case this applies, i.e. whether the RASL is removed, all AUs up to the first DRAP, all AUs up to the second DRAP, etc. This can be done in the SEI.

[0110] However, it also requires that the encoding of all DRAPs be done so that the buffering SEI can be written properly, which causes additional delay at the encoder / transmitter side.

[0111] Alternatively, a single value is indicated in the buffering SEI message, which can be modified by a subsequent SEI, e.g., a picture timing SEI message for the next non-removed AU. Then, when the AU removal is performed, the flag aus_since_rap_removed_flag only needs to be set. An example of this can be seen in Figure 8.

[0112] Here, in combination with the above figures, the second timing information 18 is a second CPB removal time t rm (2) a first CPB removal time t indicated by the first timing information for a first image portion 28′ of the video data stream for a first predetermined image CRA and preceding portion 24′ of the video data stream to generate rm (1) or this first CPB removal time t rm (1) The first timing replacement information for replacing the second CPB removal time t rm (2) It can be seen that it is possible to define

[0113] The first timing correction information 19 can be signaled, for example, by init_removal_delay_correction_offset.

[0114] Further, the first timing correction information 19 or first timing substitution information is signaled in the video data stream in a second image portion 30′ of the video data stream following portion 24′ of the video data stream and associated with a second predetermined image TRAIL1.

[0115] Note that SEI messages may also be treated as buffered in the CPB and may therefore need to be taken into account when determining the first and second timing information, which may be calculated and conveyed once for the SEI message conveying the included first and second timing information and once for the SEI message being dropped. Optionally, the second image portion may also comprise signaling 32 indicating whether the portion has been removed.

[0116] The first timing correction information 19 or the first timing substitution information can then be signaled in the video data stream regardless of the signaling 32 indicating that the portion has been removed or the signaling 32 indicating that the portion has not been removed.

[0117] The first predetermined image may be a mark image, for example an Intra Random Access Point IRAP or a Dependent Random Access Point DRAP, image, ie one that serves as a timing reference, as detailed above. The second predetermined image may also be a TRAIL or DRAP image.

[0118] Additionally, the second timing information 18 may include a second CPB removal time t for a set of one or more third image portions 30′. rm (2) a first CPB removal time t associated with a third predetermined image and indicated by the first timing information for a set of one or more third image portions 31′ of the video data stream comprising a second image portion 28′ according to the portion 24′ of the video data stream to generate rm (1) , and a second timing correction information 21 about how to correct the second CPB removal time t rm (2)can be defined and the second timing correction information 21 can be signaled in the video data stream in the second image portion 30'. The second timing correction information 21 can be signaled, for example, by cpb_removal_delay_offset.

[0119] The second image portion may also be provided with signaling 32 indicating whether the portion has been removed or not, and the second timing correction information 21 may be signaled in the video data stream regardless of the signaling 32 indicating that the portion has been removed or the signaling 32 indicating that the portion has not been removed.

[0120] It is one option for the set of one or more third image portions 31' of the video data stream to extend to a fourth image portion relative to the mark image, where the mark image can serve as a starting point for the timing reference. Finally, the marked image can be marked with a buffering period message in the video data stream within the image portion associated with the marked image.

[0121] Note that in the case shown for DRAP, the initial removal of CRA is 11 instead of 8, as it would be when the full bitstream is considered and therefore the CPB fill level is higher than in the original case. This can lead to potential problems and buffer overflows.

[0122] Another option would be to allow (in the case of DRAP) that the removal times from the CPB are not equidistant for all frames. More specifically, RAP and DRAP can have non-equidistant distances, as do all other AUs.

[0123] In this case, the RAP buffering period SEI message can have two timings: one for the entire bitstream or when the RAP is removed, and one for the DRAP. In the second case, the RAP is not output, so the removal time of the RAP can be such that the final arrival time of the RAP is equal to its removal time. The DRAP AU then has a second removal time that accumulates the buffer to the desired level. In this case, the final arrival time of the RAP is the earliest arrival time of the DRAP for both the CBR and VBR cases. The described solution is shown in Figure 9. An exemplary signaling syntax is shown in FIGS.

[0124] In one embodiment, the second timing information 18 is based on the second CPB removal time t t , which is determined by the third timing correction information 33 and / or the fourth timing correction information 35 in terms of the correction method, or by the third timing substitution information 33 or the fourth timing substitution information 35. rm (2) can be defined.

[0125] The third timing correction / replacement information 33 is a second CPB removal time t rm (2) a first CPB removal time t indicated by first timing information for a first image portion 28′ of the video data stream preceding portion 24″ of the video data stream relative to a first predetermined image CRA to generate rm (1) This relates to a method for modifying or replacing

[0126] The third timing correction / replacement information 33 can be signaled, for example, by drap_operation_initial_removal_delay. It should also be noted that cpb_drap_operation_removal_offset relates to the arrival time, i.e., the time at which a particular image portion, i.e., a marked image marked by a buffering SEI message, enters or is delivered to the CPB. Delivery can be the responsibility of a device different from the device for managing the CPB as described in this application. Compliance with the arrival time can be up to the transmitting intermediate network node.

[0127] The fourth timing correction / replacement information 35 includes a second CPB removal time t for a set of one or more third image portions 30 ″. rm (2) a first CPB removal time t indicated by the first timing information for a set of one or more third image portions 31″ of the video data stream following the portion 24″ of the video data stream in relation to the third predetermined image, to generate rm (1) This relates to a method for modifying or replacing

[0128] The fourth timing correction / replacement information 35 may be signaled, for example, by cpb_drap_operation_removal_delay_offset. In general, the device 46 for managing the CPB 48 of a video decoder is also capable of performing the modifications or substitutions detailed above.

[0129] Optionally, the third timing correction / replacement information 33 may be signaled in the video data stream within a first image portion 28' of the video data stream, and the fourth timing correction / replacement information 35 may be signaled in the video data stream within a second image portion 30'' of the video data stream that precedes in terms of encoding order 26 one of the set of one or more third image portions.

[0130] Further, the first image portion 28' may include signaling 36 indicating whether the distance between the second CPB removal times of the first and second image portions may deviate from the distance between the second CPB removal times of a consecutive pair of second and third image portions, and the third timing correction information 33 or the third timing replacement information 33 may be signaled in the video data stream within the first image portion 28' of the video data stream, subject to the signaling 36 indicating that the distance between the second CPB removal times of the first and second image portions may deviate from the distance between the second CPB removal times of a consecutive pair of second and third image portions.

[0131] The video encoder can check whether the second predetermined image is a DRAP that references the first predetermined image that is a RAP, set signaling 36 to indicate whether the second predetermined image is a DRAP that references the first predetermined image that is a RAP, and encode third timing correction / replacement information 33 in the video data stream within first image portion 28′ of the video data stream if the second predetermined image is a DRAP that references the first predetermined image that is a RAP.

[0132] Note that typically, au_cpb_removal_delay_minus1 in the pic_timing SEI message increases steadily between successive image portions, i.e., by the same distance. However, if the first image portion 28' moves away from the image encoded therein and is not presented or output, it is acceptable for the normal temporal distance to be interrupted until the DRAP and remain valid thereafter, since there is an image to be output. In other words, in the example, au_cpb_removal_delay_minus1 in the pic_timing SEI message increases steadily by the same distance. While this is typically the case, it is not prohibited for it to be otherwise. The example shows that this is the case for a full bitstream that does not originally remove anything. However, when removing everything between the RAP and the DRAP, this is no longer the case, but this is not a problem when using the DRAP function, since the RAP is not output or displayed at all. Additionally, second image portion 30'' may include signaling 38 indicating whether portion 24' has been removed.

[0133] The device 46 for managing the CPB 48 of the video decoder 44 according to the invention is able to postpone the removal of the first image portion of the video data stream after checking the signaling 32;38.

[0134] Alternatively, a network node 42 for transporting a video data stream according to the invention may set the signaling 32; 36 to indicate that the portion is to be removed from the video data stream. Network node 42 can also remove, or drop, video data stream portion 24 from the video data stream.

[0135] As can be seen in FIG. 5, optionally, the second timing information can define the second CPB removal time such that a first CPB fill level 39a'; 39a'' on the decoder side that appears on the decoder side after buffering a first image portion of the video data stream preceding the portion of the video data stream and that follows the portion of the video data stream when performing buffer removal of the second image portion according to the first CPB removal time is equal to a second CPB fill level 39b'; 39b'' on the decoder side that appears on the decoder side after buffering the first and second image portions of the video data stream when performing buffer removal of the second image portion according to a second CPB buffer time having an unbuffered portion. The video encoder can set the first and second timing information so that the video data stream complies with the above.

[0136] More generally, the second timing information may define a second CPB removal time either for a portion of a sequence of one or more RASL images or for a portion of a sequence of images preceding a particular DRAP. The video data stream may also be provided with signalling 32, 35 indicating whether or not the portion has been removed.

[0137] The offsets in the DRAP used for random access are used to calculate the removal times of subsequent AUs. Obviously, instead of indicating the offsets in the picture timing SEI message, it is mandatory that the DRAP includes a buffering period SEI message and that these offsets are indicated in the buffering period SEI message.

[0138] The problem addressed by the subsequently described embodiments is how to represent information regarding temporal scalability and picture buffer timing in picture timing SEI and buffering period SEI messages in such a coding environment. Accordingly, embodiments are described that refer to alternative information within the picture for temporal scalability in picture timing SEI and buffering period SEI messages. We begin by presenting an embodiment that refers to alternative information within the picture for temporal scalability in picture timing SEI and buffering period SEI messages, namely, an embodiment that utilizes an offset for timing in the picture timing SEI messages.

[0139] 12 to 14 show some examples of removal time values when different frame rates are provided in the same bitstream (ie, temporal scalability).

[0140] The table in Figure 17 shows that the deltaTime for frame "blue" is 0, 1 / 60, 3 / 120, 2 / 60, and 6 / 120. For "red frame," it is 3 / 120 and 6 / 120. Blue frames are referenced by those rows in the table that contain values of 30, 60, and 120 fps. Red frames are referenced by those rows in the table that contain only values of 60 and 120 fps, but not 30 fps.

[0141] In the case of an open GOP, where the highest frame rate is shown in Figure 16, and also shown in Figure 15, the following applies: In the table in Figure 15, we can see that the deltaTime for frames "blue" is 0, 1 / 60, and 3 / 120. For "red frames," it is 1 / 120 and 2 / 120. Again, blue frames are referenced by those rows in the table that contain values of 30, 60, and 120 fps. Red frames are referenced by those rows in the table that contain only values of 60 and 120 fps, but not the 30 fps value.

[0142] In summary, there are some repeating patterns in the removal times deltaTime for different frame rates due to the GOP structure of the coded bitstream.

[0143] Another aspect to consider is scene cuts, as they interrupt the described pattern. Different locations for scene cuts will result in different values. For simplicity, only one location is shown in Figure 18. As can be seen from the table in Figure 19, clearly the GOPs affected by scene cuts do not follow a pattern.

[0144] FIG. 20 shows an example signaling structure referencing a picture timing SEI message containing different values for different frame rates (i.e., target TemporalId).

[0145] In the embodiment shown in FIG. 14, the video data stream is such that the images 12 of the video are arranged at different hierarchical levels 50, where for each hierarchical level all images at each hierarchical level depend solely on images at that level or at lower hierarchical levels. i The images are coded in groups of pictures 50 using temporal inter prediction in a temporally scalable manner, so that the images are grouped into groups of pictures 50. In the figure, only the beginning of one GOP is shown.

[0146] Portion 24 also consists of images belonging to the highest hierarchical level 503 or to one or more hierarchical levels 502 excluding the highest hierarchical level 503 and the lowest hierarchical level 501 immediately below the highest hierarchical level 503 in hierarchical order 52 .

[0147] The first timing information 16 then calculates, for each picture, the first CPB removal time t for the picture by an increment relative to the preceding marked picture 53 of the current group in decoding order. rm (1) This order is indicated by the order of decimal values used when labeling the images.

[0148] Again, the increment can be exemplarily signaled by AuCPBRemovalDelay or au_cpb_removal_delay_minus1. The current GOP is a picture relative to the previous picture of the GOP of which the respective picture is part. Furthermore, the marked image can be marked with a buffering period message of the video data stream in the image portion associated with the marked image.

[0149] In one embodiment, the second timing information 18 is signaled in the video data stream in each of the fifth image portions 23 of the video data stream for the fifth images 54 associated with one or more hierarchical levels other than the highest hierarchical level by fifth timing correction information 19 or fifth timing substitution information 19 for substitution. rm (2) can be defined.

[0150] The fifth timing correction information 19 can be signaled, for example, by au_cpb_removal_pattern_offset, and the fifth timing replacement information 19 can be signaled, for example, by au_cpb_removal_delay_minus1[i], where i is not equal to the current hierarchical level to which the picture to which pic_timing is transmitted belongs.

[0151] The fifth timing correction / replacement information 19 includes a second CPB removal time t rm (2) the first CPB removal time t indicated by the first timing information for each fifth image portion 23 to generate rm (1) This relates to a method for modifying or replacing

[0152] The first timing information may be signaled, for example, by au_cpb_removal_delay_minus1[i], or u_cpb_removal_delay_minus1, where i is equal to the current hierarchical level to which the picture to which pic_timing is transmitted belongs.

[0153] If portion 24 comprises a highest hierarchical level up to the respective hierarchical level above the respective hierarchical level, fifth timing correction / replacement information 19 is signaled for each hierarchical level between the highest and lowest hierarchical levels, i.e., between 501 and 502 in the illustrated example, and for the hierarchical levels between the highest and lowest hierarchical levels above the respective hierarchical levels, i.e., between 503 of 502 and 503 and 502 of 501 in the illustrated example. As an example, the numerical value is indicated by num_sub_layer_cpb_removal_delays_minus1 if buffering_period_pattern_flag is 1, and num_sub_layer_cpb_removal_delays_minus1+1 if buffering_period_pattern_flag is zero.

[0154] 21 and 22 show example signaling structures in which patterns can alternatively be defined within each buffering period SEI and these can be indicated in the picture timing SEI message. As mentioned above, when a pattern does not apply, e.g., due to a scene cut, a value can be signaled explicitly (or in the form of some offset).

[0155] According to this embodiment, the second timing information 18 is adjusted by the fifth timing correction information 19, e.g., au_cpb_removal_pattern_offset, to adjust the second CPB removal time t rm (2)The fifth timing correction information 19 indicates how to modify the first CPB removal time by an offset value, and the video data stream comprises data 60 indicating a list of representative offset values assumed by the offset value indicated by the fifth correction information for modifying the first CPB removal time in one or more groups of pictures. The fifth timing correction information 19 then signals the offset value by a pointer that points within the list of representative offset values.

[0156] More generally, in the above embodiment, the video data stream is reduced by portion 24, i.e., resulting in reduced version 20, and the video data stream comprises first and second timing information 18. In other words, the second timing information is not nested, such that if a particular layer level is dropped at any intermediate network device, the second timing information does not nest and replaces the corresponding first timing information. Rather, the first and second timing information remain unchanged at the decoder side, and the exact timing information is selected to be used by the CPB processing depending on signaling sent within the video data stream, configured for the intermediate network device to describe which portion has been removed, or more precisely, which previous highest layer level has been removed / dropped.

[0157] Another example of an embodiment of referencing alternative information within a picture for temporal scalability in picture timing SEI messages and buffering period SEI messages is a delta in the buffering period SEI message.

[0158] As can be seen in Figure 23, different values of vcl_initial_cpb_removal_delay[i] and vcl_initial_cpb_removal_offset[i] should also obviously be indicated in the buffering period SEI message, since one is not enough for all possible frame rates.

[0159] Another embodiment refers to the consideration of multiple bit rates. As mentioned above, the current way to support different transmission rates or CPB provisioning rates is to indicate several values for bit rate, CPB size, and respective initial removal time and offset.

[0160] The problem with this current solution is that all potential CPB-supplied bitrates must be known in advance, and the rate control at the encoder must take all of them into account to ensure that the information provided is valid.

[0161] While this allows a certain degree of flexibility, where the mentioned parameters can be changed in many ways, in some practical scenarios, taking into account many potential CPB supply rates may be desirable from the transmitting side but difficult to implement on the encoder side. Under such circumstances, a "fairly" restrictive rate control, where some parameterization of the HRD model is performed, may be useful, meaning that two extremes, a maximum and a minimum bitrate, are considered, and everything in between can be linearly interpolated. While such a solution is more restrictive than a flexible solution in terms of HRD modeling, it allows the transmitting side to use any rate between the maximum and minimum bitrates, and still guarantees no underflow or overflow, with the CPB size and initial removal time being calculated based on the transmitted values for the maximum and minimum values. The hrd_parameters are divided into three parts: cbr_parameters_present_flag ·vbr_min_parameter_present_flag ·vbr_max_parameter_present_flag

[0162] Any other bit rate between the two given in vbr_min_parameter_present_flag and vbr_max_parameter_present_flag can be calculated. Example signaling is shown in Figures 24a, 29, and 30.

[0163] Similarly, the buffering period SEI message also includes the CBR, minimum, and maximum values, if present in hrd_parameters, as shown in Figures 24b and 31.

[0164] According to this, the HRD timing information data, for example the first HRD timing information data, the second HRD timing information data, or any of the plurality of HRD timing information data, and each HRD timing information data comprises one or more of the following: - an indication 104 of the CPB supply bit rate to which each HRD timing information relates; - an indication 106 of the CPB size required for each CPB supply bitrate to which each HRD timing information relates;

[0165] - a representation 108 of the CPB removal time of each of the marked image portions of the video data stream, measuring the elapsed time from the CPB arrival of the first bit of the respective marked image portion for each CPB supply bit rate to which the respective HRD timing information relates - where the marked image portions are those having a buffering period SEI such as RAP, each of which is shown by cross-hatching in another suitable figure, while another image is shown by plain hatching in another suitable figure, etc.

[0166] - an indication 110 of the CPB delivery deferral time of each of the mark picture portions of the video data stream, measuring the elapsed time by which the CPB arrival of the first bit of the respective mark picture portion is delayed for the respective CPB delivery bit rate to which the respective HRD timing information relates; If there is a minimum and a maximum, any value in between results in, for example, a linear interpolation of the minimum and maximum values.

[0167] Nevertheless, in some cases, more flexibility in not limiting the HRD model to a linear interpolation model would be desirable (e.g., decreasing initial_removal_delay rather than linearly, increasing the bit rate). Thus, depending on the scenario, the encoder can choose to select whether to use a mode with linear interpolation or one with multiple values. Example signaling is shown in Figures 32, 33, and 34.

[0168] As shown in Figures 24, 29, 31, 32, and 34, a video stream may comprise first HRD timing information data 70a, b relating to a minimum CPB supply bit rate and second HRD timing information data 72a, b relating to a maximum CPB supply bit rate. The first HRD timing information data 70a, b and the second HRD timing information data 72a, b then enable determination of a third HRD timing information of an actual CPB supply bit rate by linearly interpolating between the first and second HRD timing information data. The actual CPB supply bit rate is, for example, the bit rate at which the video data stream is actually supplied to the CPB.

[0169] The video data stream may further comprise a flag 74, e.g., hrd_interpolation-mode_flag, indicating whether a first HRD timing information data 70a, b and a second HRD timing information data 72a, b are present, or whether multiple HRD timing information data 76 are present for each of the required set of CPB supply bit rates.

[0170] The video encoder can then decide to provide the video data stream with first HRD timing information 70 and second HRD timing information 72, or to provide the video data stream with multiple HRD timing information data 76 for each of the required sets of CPB supply bit rates.

[0171] The video encoder may optionally perform a determination based on a feasibility check of whether linear interpolation between the first and second HRD timing information results in a situation where there are no CPB overflows or underflows for all CPB supply bitrates between the minimum and maximum CPB supply bitrates. The device 46 for managing the CPB 48 of the video decoder 44 can then use the third HRD timing information to manage the CPB.

[0172] Similarly, the device can check a flag 74 in the video data stream, for example hrd_interpolation-mode_flag, and, depending on the flag, derive first HRD timing information data 70a, b and second HRD timing information data 72a, b from the video data stream, or derive multiple HRD timing information data 76 for each of a desired set of CPB supply bit rates from the video data stream. FIG. 38 shows three example operating points. The interpolated case uses the following linear interpolation formula: -BR interpolated =BR min +(BR max -BR min )*alpha -IND interpolated =IND min +(IND max -IND min )*alpha(where IND=InitialRemovalDelay) In addition, IND min (for minimum bitrate) is IND maxNote that this is larger than (for the highest bitrate). Figure 37 is IND min and the following abbreviations are used: RmD = Removal Delay Afinal = Final arrival time ArrivalEar = Earliest possible arrival time of the first bit of the image due to the frame rate ArrivalInit = actual arrival time of the first bit of the image It can be seen that the maximum CPB size is 15. Figure 36 is IND max And the required CPB size for this second case is 12,1125.

[0173] For a linearly calculated initial removal delay and bitrate with alpha equal to 0.3, Figure 35 shows that the required CPB size in this case is 13,87912. If linear interpolation is performed, the calculated CPB size will correspond to 12,97875. As can be seen, the values are different.

[0174] Therefore, if an encoder wishes to adhere to the constraint that the involved HRD parameters are linearly derived with full flexibility within their maximum and minimum values, the rate control in the encoder should take this into account when determining the size of the image so that the CPB size constraints during interpolation are met.

[0175] This is not always possible, so one option would be to signal that interpolation is possible only if the encoder processes it, and otherwise signal all the actual values used at a discrete number of operating points.

[0176] As mentioned above, the above concepts can be adopted by video streams, video encoders, video decoders, network nodes, devices for managing the coded picture buffer (CPB) of a video decoder, and devices for splicing video data streams together.

[0177] Similarly, the concepts can be implemented by corresponding methods according to embodiments of the present invention. These methods are based on the same considerations as the above-mentioned video stream, video encoder, video decoder, network node, apparatus for managing a coded picture buffer (CPB) of a video decoder, and / or apparatus for splicing video data streams together. However, it should be noted that the methods may also be supplemented by any of the features, functions, and details described herein with respect to the video stream, video encoder, video decoder, network node, apparatus for managing a coded picture buffer (CPB) of a video decoder, and / or apparatus for splicing video data streams together. Also, the methods may be supplemented by the features, functions, and details of the video stream, video encoder, video decoder, network node, apparatus for managing a coded picture buffer (CPB) of a video decoder, and / or apparatus for splicing video data streams together, either individually or in combination.

[0178] Finally, the concepts can also be used to generate coded data streams according to embodiments of the present invention, which data streams can also be supplemented by features, functions, and details of methods, video encoders, video decoders, network nodes, apparatuses for managing a coded picture buffer (CPB) of a video decoder, and / or apparatuses for splicing video data streams together, either individually or in combination.

[0179] In conclusion, the embodiments described herein can be optionally supplemented by any of the key features or aspects described herein, but it should be noted that the key features and aspects described herein can be used individually or in combination, and can be introduced into any of the embodiments described herein, either individually or in combination.

[0180] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a device or portions thereof correspond to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding apparatus or portion thereof or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0181] Depending on specific implementation requirements, the implementation of the present invention can be implemented in hardware or software. The implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored, which cooperates (or can cooperate) with a programmable computer system to execute the respective method. Thus, the digital storage medium can be computer-readable.

[0182] Some embodiments according to the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0183] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer. The program code may, for example, be stored on a machine-readable carrier. Further embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0184] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0185] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer-readable medium) comprising, and having recorded thereon, a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0186] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals may for example be adapted to be transmitted via a data communication connection, for example via the Internet.

[0187] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0188] Further embodiments according to the invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0189] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0190] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0191] The devices described herein, or any components of the devices described herein, may be implemented at least in part in hardware and / or software.

[0192] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0193] The methods described herein, or any part of the methods described herein, may be performed at least in part by hardware and / or software.

[0194] The above-described embodiments merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details presented by the description and explanation of the embodiments herein.

Claims

1. 1. A video encoder for encoding video into a video data stream such that the video data stream comprises a sequence of image portions, each image portion having an image of the video encoded therein; indicating a maximum initial removal delay in a first buffering period (BP) supplemental enhancement information (SEI) message; indicating a first flag in a picture timing SEI message corresponding to a first picture portion of the video data stream, the first flag equal to 1 indicating that if the first picture portion is followed by a second picture portion having a second BP SEI message with a concatenation flag equal to 1 and an initial removal delay less than the maximum initial removal delay, a nominal coded picture buffer (CPB) removal time based on the elapsed time since removal of a previous non-discardable picture portion is to be applied; a video encoder configured to:

2. The video encoder of claim 1 , wherein the second BP SEI message indicates an elapsed time since removal of the previous non-discardable image portion.

3. The video encoder of claim 1 , wherein the first image portion, the second image portion, and the previous non-discardable image portion are access units.

4. 1. A method of encoding video into a video data stream such that the video data stream comprises a sequence of image portions, each image portion having an image of the video encoded therein; indicating a maximum initial removal delay in a first buffering period (BP) supplemental enhancement information (SEI) message; indicating a first flag in a picture timing SEI message corresponding to a first picture portion of the video data stream, the first flag equal to 1 indicating that if the first picture portion is followed by a second picture portion having a second BP SEI message with a concatenation flag equal to 1 and an initial removal delay less than the maximum initial removal delay, a nominal coded picture buffer (CPB) removal time based on the elapsed time since removal of a previous non-discardable picture portion is to be applied; A method for providing the above.

5. The method of claim 4 , wherein the second BP SEI message indicates an elapsed time since removal of the previous non-discardable image portion.

6. The method of claim 4 , wherein the first image portion, the second image portion, and the previous non-discardable image portion are access units.

7. When executed by at least one processor, indicating a maximum initial removal delay in a first buffering period (BP) supplemental enhancement information (SEI) message; indicating a first flag in a picture timing SEI message corresponding to a first picture portion of the video data stream, the first flag equal to 1 indicating that if the first picture portion is followed by a second picture portion having a second BP SEI message with a concatenation flag equal to 1 and an initial removal delay less than the maximum initial removal delay, a nominal coded picture buffer (CPB) removal time based on the elapsed time since removal of a previous non-discardable picture portion is to be applied; instructions to cause the at least one processor to encode video into the video data stream such that the video data stream comprises a sequence of image portions by performing Each image portion has an image of the video encoded therein. Non-transitory computer-readable medium.

8. The non-transitory computer-readable medium of claim 7 , wherein the second BP SEI message indicates an elapsed time since removal of the previous non-discardable image portion.

9. The non-transitory computer-readable medium of claim 7 , wherein the first image portion, the second image portion, and the previous non-disposable image portion are access units.

10. 1. A program for encoding video into a video data stream such that the video data stream comprises a sequence of image portions, each image portion having an image of the video encoded therein; The program is provided to a computer system, indicating a maximum initial removal delay in a first buffering period (BP) supplemental enhancement information (SEI) message; indicating a first flag in a picture timing SEI message corresponding to a first picture portion of the video data stream, the first flag equal to 1 indicating that if the first picture portion is followed by a second picture portion having a second BP SEI message with a concatenation flag equal to 1 and an initial removal delay less than the maximum initial removal delay, a nominal coded picture buffer (CPB) removal time based on the elapsed time since removal of a previous non-discardable picture portion is to be applied; Execute program.

Images