Hypothetical reference decoder
By introducing a variety of CPB removal time information and cascading flags into the video encoding standard, seamless switching and buffer state consistency problems are solved at the clipping point, and efficient video data flow management and bit rate adaptation are achieved to ensure that Continuity of video quality.
Patent Information
- Application Number
- CN202411320524.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-01
- Filing Date
- 2020-02-27
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-02-27
AI Technical Summary
Existing video encoding standards, such as HEVC, are difficult to ensure seamless switching and consistency of CPB state at the clipping point, especially in non-seamless clipping scenarios, which may cause buffer overflow or underload and fail to be effective. Process the case of bit rate variants and image discarding.
The first and second types of CPB removal time information are introduced, the actual CPB feed bit rate is determined by linear interpolation, and the cascade marking and splice point indicator are used in the image part to ensure the accuracy and buffering of splice points. The device is filled with a degree of fullness, adapting to different frame rates and bitstream structures.
Seamless switching at the clipping point is achieved, buffer overflow or underloading is avoided, the continuity and quality of the video data stream is ensured, and the changes in different bitstreams and frame rates are adapted to.
Smart Images

Figure CN118945376A_ABST
Abstract
Description
Technical Field
[0001] This application is a divisional application of application number: 202080032895.0, and the invention name is: Hypothetical Reference Decoder. Hypothetical reference decoder HRD and its use for checking bitstream and decoder consistency are usually specified in video coding standards, such as AVC (Advanced Video Codec) and HEVC (High Efficiency Video Coding).
[0002] Embodiments according to the present invention relate to a video stream, a video encoder, a video decoder, a network node, a device for managing a coded picture buffer CPB of a video decoder, and a device for splicing video data streams together. Background Art
[0003] Hereinafter, an introduction to the background technology will be provided.
[0004] To this end, a HRD buffer model is specified, which consists of a hypothetical stream scheduler (HSS), a coded picture buffer (CPB), a decoding process (which is considered to be instantaneous), a decoded picture buffer (DPB), and an output clipping process, as shown in Figure 2 as shown in .
[0005] The model defines the timing and bit rate for feeding the bitstream into the coded picture buffer, the time when its decoding unit (access unit AU or VCL NAL (VCL: Video Coding Layer, NAL: Network Abstraction Layer) in the case of low-latency operation mode) is removed from the CPB and decoded instantaneously, and the output time of the picture from the DPB.
[0006] By doing so, it is also possible to define the CPB size that is needed by the decoder to avoid buffer overflow (more data is sent to the decoder than can be stored in the CPB), or to avoid underrun where less data is sent to the decoder (at a lower bit rate than required), and the required decoding units are not at the decoder at the right time to decode.
[0007] Current state-of-the-art decoding standards specify different parameters to describe the bitstream as well as the HRD requirements and buffer models.
[0008] For example, the hrd_parameter defined per sublayer in HEVC consists of one or more pairs of Bitrate(i) and CPBsize(i), indicating that if the stream scheduler feeds a CPB of size CPBsize(i) at a bitrate of Bitrate(i), no overflow or underflow will occur.
[0009] In conjunction with the hrd_parameter syntax element, there is additional timing information in the bitstream specifying the removal time of a picture from the CPB, which indicates when the VCL NAL units belonging to that picture are sent for decoding.
[0010] For ease of understanding, the description of sub-image processing is omitted.
[0011] The relevant information is present in the buffering period SEI (Auxiliary Enhancement Information) message with InitialCPBRemovalDelay(i), InitialCPBRemovalOffset(i) and AuCPBRemovalDelay, and in the picture timing SEI message with AuCPBRemovalDelay.
[0012] For the most basic operation, only InitialCPBRemovalDelay(i) and AuCPBRemovalDelay are used.
[0013] In this case, the first access unit decoded is a random access point RAP with its corresponding buffering period SEI message, and time 0 is defined as the time when the first bit of the random access point enters the CPB. Then at time InitialCPBRemovalDelay(i), the picture corresponding to the random access point and other non-RAP pictures are removed from the CPB, and the removal of the CPB occurs at InitialCPBRevovalDelay(i)+AuCPBRemovalDelay (HEVC defines some parameters to convert delay into time, namely ClockTick, but this is ignored here for simplicity).
[0014] When the next RAP comes, the removal time is calculated as before for non-RAP pictures, i.e. InitialCPBRevovalDelay(i)+AuCPBRemovalDelay, and this new value is used as an anchor for further increments until another RAP, i.e.
[0015] anchorTime=InitialCPBRemovalDelay(i)+AuCPBRemovalDelay
[0016] And then, the removal of the picture becomes anchorTime+AuCPBRemovalDelay, and anchorTime is updated at the next RAP with buffered SEI message,
[0017] anchorTime=anchorTime+AuCPBRemovalDelay, and so on.
[0018] The described mode of operation is the simplest mode of operation. There are other situations that need to be considered.
[0019] Under the present circumstances, the present invention overcomes several disadvantages. The details are as follows. Summary of the invention
[0020] An embodiment according to the present invention relates to a video data stream in which a video is encoded, and includes first timing information about a first coded picture buffer (CPB) removal time to be applied at a decoder side when a complete version of the video data stream is buffered, and second timing information about a second CPB removal time to be applied at the decoder side to buffer a reduced version of the video data stream, wherein the reduced version differs from the video data stream by removing a portion of the video data stream from the complete version of the video data stream.
[0021] A further embodiment according to the present invention relates to a video data stream in which a video is encoded, comprising first HRD timing information data associated with a minimum CPB feed bit rate, second HRD timing information data associated with a maximum CPB feed bit rate, wherein the first HRD timing information data and the second HRD timing information data allow determination of third HRD timing information for an actual CPB feed bit rate by linear interpolation between the first and second HRD timing information data.
[0022] A further embodiment according to the present invention relates to a video data stream, comprising: a sequence of image parts, each of which has an image of a video encoded therein, the image parts comprising a first type of image part and a second type of image part used as a CPB removal time reference; first timing information in each image part, which is about a first CPB removal time of the corresponding image part, the first CPB removal time measuring the elapsed time from the removal of a previous first type of image part; and second timing information in each of the predetermined first type of image parts, which is about a second CPB removal time of the corresponding predetermined first type of image part in the case of resuming the decoding of the video data stream forward from the corresponding predetermined first type of image part, the second CPB removal time measuring the elapsed time from the arrival of the CPB of the first bit of the corresponding predetermined first type of image part; a splicing point indication, which indicates an image part whose time difference between the final CPB arrival and its CPB removal exceeds a predetermined threshold, so that the image part can be used as the last image part before the splicing point with another video data stream.
[0023] A further embodiment according to the invention relates to a video encoder, which encodes a video into a video data stream, and provides the video data stream with first timing information and second timing information, wherein the first timing information is about a first CPB removal time to be applied at a decoder side in case a complete version of the video data stream is buffered, and the second timing information is about a second CPB removal time to be applied at the decoder side to buffer a reduced version of the video data stream, wherein the reduced version differs from the video data stream by removing a part of the video data stream from the complete version of the video data stream.
[0024] A further embodiment according to the present invention relates to a video encoder, which encodes a video into a video data stream, and provides the video data stream with first HRD timing information related to a minimum CPB feed bit rate and second HRD timing information related to a maximum CPB feed bit rate, wherein the first HRD timing information and the second HRD timing information allow determination of third HRD timing information for an actual CPB feed bit rate by linear interpolation between the first and second HRD timing information.
[0025] According to a further embodiment of the present invention, reference is made to a video encoder for encoding a video into a video data stream so that the video data stream comprises a sequence of image portions, each image portion having an image of the video encoded therein, wherein the video encoder classifies the image portions into a first type of image portion used as a CPB removal time reference and a second type of image portion; provides the video data stream with first timing information in each image portion, which is about a first CPB removal time of the corresponding image portion, the first CPB removal time measuring the elapsed time from the removal of a previous first type of image portion, and second timing information in each of the predetermined first type of image portions, which is about a first CPB removal time of the corresponding image portion from the removal of the previous first type of image portion. The method comprises the following steps: determining a second CPB removal time of a corresponding predetermined first type image portion in the case of forwardly resuming the decoding of a video data stream with a predetermined first type image portion, the second CPB removal time measuring the elapsed time from the arrival of the CPB of the first bit of the corresponding predetermined first type image portion; for each of the predetermined image portions, checking whether the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold value, so that the predetermined image portion can be used as the last image portion before a splicing point with another video data stream; and providing a splicing point indication to the video data stream, which is used to indicate these predetermined image portions whose time difference between their final CPB arrival and their CPB removal exceeds the predetermined threshold value.
[0026] A further embodiment according to the invention relates to a network node for forwarding a video data stream, which is arranged to remove a portion of the video data stream from the video data stream.
[0027] A further embodiment according to the present invention relates to a device for managing the CPB of a video decoder, which decodes a video data stream in a manner buffered by the CPB, and manages the CPB according to first timing information when the video data stream includes the part, and manages the CPB according to second timing information when the part has been removed.
[0028] A further embodiment according to the present invention relates to an apparatus for managing a CPB of a video decoder, which decodes a video data stream in which a video is encoded, the apparatus deriving first HRD timing information related to a minimum CPB feed bit rate from the video data stream, deriving second HRD timing information related to a maximum CPB feed bit rate, determining third HRD timing information for an actual CPB feed bit rate by linearly interpolating between the first and second HRD timing information, and managing the CPB using the third HRD timing information.
[0029] A further embodiment according to the present invention relates to a device for managing a CPB of a video decoder, the video decoder decoding a video data stream in a manner of buffering through the CPB, the device checking whether a concatenation flag in a first predetermined first type image portion indicates that a splicing has occurred at the first predetermined first type image portion, and determining a time to remove the first predetermined first type image portion from the CPB. If the concatenation flag in the first predetermined first type image portion indicates that a splicing has not occurred at the first predetermined first type image portion, the determination is made based on first timing information of the first predetermined first type image portion, and if the concatenation flag in the first predetermined first type image portion indicates that a splicing has occurred at the first predetermined first type image portion, the determination is made based on second and third timing information of the first predetermined first type image portion.
[0030] A further embodiment according to the invention relates to an apparatus for splicing a first video data stream with a second video data stream, each of the data streams comprising: a sequence of image parts, in each image part an image having a video encoded therein, the image parts comprising a first type of image part and a second type of image part used as a CPB removal time reference; first timing information in each image part, which is about a first CPB removal time of the corresponding image part, the first CPB removal time measuring the elapsed time from the removal of a previous first type of image part; and second timing information in each of the predetermined first type of image parts, which is about the corresponding predetermined first type of image part in case of resuming the decoding of the video data stream from the corresponding predetermined first type of image part onwards. A second CPB removal time of a first type image portion, the second CPB removal time measures the elapsed time from the arrival of the CPB of the first bit of the corresponding predetermined first type image portion, wherein the second video data stream includes a cascade flag in the first predetermined first type image portion and third timing information, the cascade flag is set to indicate to the second video data stream that the first predetermined first type image portion continues the second state of the second video data stream, and the third timing information indicates a third CPB removal time of the first predetermined first type image portion, which is used to determine the CPB removal time when the second video data stream is spliced in at the first predetermined first type image portion, and the third CPB removal time measures the elapsed time from the most recent CPB removal of the non-discardable image portion. The apparatus of the present invention checks whether a splicing point indication in a first video data stream indicates that a time difference between a CPB arrival and a CPB removal thereof for a predetermined image portion exceeds a predetermined threshold value, so that the predetermined image portion can be used as the last image portion of the first video data stream before the first video data stream to be spliced with a second video data stream; if so, a concatenation flag in a first predetermined first-type image portion of the second video data stream is set to a first state, wherein the first state indicates that the second video data stream has been spliced in at the first predetermined first-type image portion; and the first data stream is concatenated with the second data stream at the predetermined image portion and the first predetermined first-type image portion, respectively, so as to obtain a spliced video data stream.
[0031] The concepts described above can be implemented by methods according to embodiments of the present invention. These methods are based on the same considerations as the decoders, encoders, devices and data streams described above. However, it should be noted that the methods can be supplemented by any of the features, functionality and details described herein also with respect to the decoders, encoders, devices and data streams. In addition, the methods can be supplemented individually and in combination by the features, functionality and details of the decoders, encoders, devices and data streams.
[0032] Finally, the concepts can also be used to generate coded data streams according to embodiments of the invention. The data streams can also be supplemented by features, functionalities and details of decoders, encoders, devices and methods, individually and in combination. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Embodiments according to the present invention will now be described with reference to the accompanying drawings, in which:
[0034] Figure 1 shows an encoded data stream according to an embodiment of the present application,
[0035] Figure 2 A flowchart for a hypothetical reference decoder according to an embodiment of the present application is shown.
[0036] Figures 3 to 7 as well as Fig. 9 shows the state of the coded picture buffer including the removal time according to one embodiment of the present application,
[0037] Figure 8 , Fig.10 , Fig.11 , Figure 20 to Figure 2 4 and Figures 26 to 34 shows an exemplary signaling syntax according to an embodiment of the present application,
[0038] Figure 12 to Figure 14 , Fig.16 as well as Fig.18 shows a data stream structure for different frame rates according to an embodiment of the present application,
[0039] Fig.15 , Fig.17 as well as Fig.19 FIG. 2 shows an exemplary removal time value when different frame rates are provided under the same bit stream according to an embodiment of the present application.
[0040] Fig.25 shows the coded data streams spliced together according to an embodiment of the present application,
[0041] Fig.35 , Fig.36 as well as Fig.37 A table showing calculated values of CPB size according to an embodiment of the present application, and
[0042] Fig.38 A graph showing exemplary CPB fill levels according to an embodiment of the application.
[0043] Throughout the drawings, similar reference numerals indicate similar elements and features. DETAILED DESCRIPTION
[0044] In the following, some considerations on which the present invention is based will be discussed and several solutions will be described. In particular, several details will be disclosed, which can be optionally introduced into any of the embodiments disclosed herein. In particular, the following description begins with a simple presentation of the problems faced when splicing, stream extraction, stripping scalable bitstreams and bitstream feeding occur and handle the situations under various transmission conditions. Then, some corrections to the problems are presented by the subsequent presented embodiments utilizing the corresponding corrections.
[0045] Check and resolve the following issues.
[0046] Splicing : The removal time of the previous RAP with buffered SEI is used as an anchor to add AuCPBRemovalDelay so that the removal time of the CPB of the image is used.
[0047] However, when splicing occurs, the previous value of anchorTime cannot be easily known at the splicing point. Therefore, in this case, the derivation of the CPB removal time of the spliced RAP with buffered SEI is performed differently.
[0048] To address this issue, HEVC includes two additional parameters: concatenation_flag, which indicates whether a splice has occurred at a RAP that includes a buffering period SEI message; and AuCPBRemovalDelta time, which does not refer to the previous RAP with a buffering period SEI message, but refers to the time delta to the previous non-discardable picture. Thus, it prevents the situation where any value must be calculated and the buffering period SEI message rewritten at the splice point. The CPB removal time from the RAP where the splice occurred will then be
[0049] Remove(previousNonDiscardable)+AuCPBRemovalDelta.
[0050] This assumes a seamless splicing scenario. However, there is no guarantee that the CPB state is the same at the splice point of the original bitstream (when the RAP splice result with the buffering period SEI message has not been spliced, but is in its original bitstream) and at the time of splicing. More specifically, Figure 3 The described problem is shown where the bitstream at the top of frame #3 is spliced with the bitstream at the bottom from frame #2 onward.
[0051] Therefore, instead of having a removal time as discussed above (which would have been earlier than when the frame had reached t af 3:00
[0052] Removal(previousNonDiscardable)+AuCPBRemovalDelta,
[0053] Instead use the removal time
[0054] Removal(previousNonDiscardable)+InitialCPBRevovalDelay(i)+t af 2-t rm 2.
[0055] That is, the removal time of the spliced RAP is:
[0056] Removal(previousNonDiscardable)+SplicingDelta
[0057] Where SplicingDelta is set to AuCPBRemovalDeta and InitialCPBRevovalDelay(i)+t af 2-t rm The maximum value of 2.
[0058] In the case where the second value happens to be greater than the first value, for example Figure 3 As shown in , a non-seamless cut will occur.
[0059] This splicing operation works only when the first bitstream ends at the corresponding NonDiscardable picture, because AuFinalArrivalTime and the removal time of the additional pictures after the NonDiscardable picture are unknown, and therefore if the two values t af X and t rm If the difference in X is different from the NonDiscardable image, the exported removal time may be wrong (InitialCPBRemovalDelay(i)+t af Xt rm X).
[0060] CPB Fullness and Delay Another problem that occurs with the HRD model is that in order to utilize the CPB buffer, a time delay is required for the removal of the first access unit.
[0061] When the first RAP enters the CPB after splicing, the removal of this AU is delayed until InitialCPBRemovalDelay(i) time, as already discussed. A given CPB fullness, such as CPBA, is achieved by feeding the CPB at Bitrate(i) during InitialCPBRemovalDelay(i). Afterwards, a higher CPB fullness than the CPB cannot be achieved at other RAPs with buffering period SEI messages. A Large CPB B The reason for this can be seen mathematically, examining the earliest time that an AU can enter the CPB.
[0062] initArrivalTime[n]=Max(AuFinalArrivalTime[n-1], initArrivalEarliestTime[n]), where initArrivalEarliestTime[n]=RemovalTime[n]-InitCpbRemovalDelay(i)
[0063] This means that if an AU with a buffering period SEI message cannot enter the CPB before its removal time InitialCPBRemovalDelay(i), it is impossible to achieve a buffering period greater than CPB. A CPB B This is because feeding the CPB at Bitrate(i) during InitialCPBRemovalDelay(i) only achieves CPB A CPB fullness.
[0064] To solve this problem, the idea is to assume that the sender (or HSS in the figure) delays the scheduling of the first RAP with the buffered SEI message by a given time offset InitialCPBRemovalOffset(i), such as Figure 4 as shown in .
[0065] It is important to mention that this only applies to VBR and not CBR, because in CBR,
[0066] initArrivalTime[n]=AuFinalArrivalTime[n-1].
[0067] In this case, the schedule changes to the following
[0068] initArrivalEarliestTime[n]=RemovalTime[n]-InitCpbRemovalDelay(i)-
[0069] InitialCPBRemovalOffset(i)
[0070] This means that CPB B The CPB size may correspond to the size achieved by feeding the CPB at Bitrate(i) for InitCpbRemovalDelay(i)+InitialCPBRemovalOffset(i).
[0071] AU discard : The HRD model in HEVC also supports the discarding of some NAL units. More specifically, "discardable" images associated with RAP images can be discarded. "Discardable" images refer to RASL (Random Access Skip Preamble) images, that is, images that are located before the RAP image in the presentation order but after the RAP image in the decoding order in the open GOP (Group of Pictures) structure, and also refer to images that are located before the RAP in the decoding order. Therefore, when random access is performed at the corresponding RAP image, the RASL image cannot be decoded. Therefore, when splicing this bitstream with another bitstream or when starting a session at the RAP position, sending RASL images is a waste of resources, and in some cases these images are discarded before sending.
[0072] Obviously, the timing information, ie the initial arrival time of the pictures as well as the removal time, changes when the bitstream is modified.
[0073] Therefore, the buffering period SEI message contains alternative timing for the case where a RASL picture associated with a RAP with the buffering period SEI message is discarded:
[0074] o AltInitCpbRemovalDelay(i)
[0075] o AltInitialCPBRemovalOffset(i)
[0076] Time Scalability: When a sublayer is discarded, the timing information also changes. For example, if the original bitstream has 60Hz, it is expected that the distance in the nominal removal time between two consecutive images in decoding order will be 1 / 60 second. And when every other image is discarded, the distance will be 1 / 30 second. This means that the image timing SEI message will need to be replaced. In addition, InitCpbRemovalDelay(i) and InitialCPBRemovalOffset(i) will also need to be changed. Therefore, the buffer period SEI message will need to be replaced. For this operation, additional image timing SEI messages and buffer period SEI messages are usually included in the so-called nested SEI messages. When the middleware performs sublayer bitstream extraction (for example, discarding every other frame to obtain a 30Hz bitstream from a 60Hz bitstream), the original image timing SEI message and buffer period SEI message are replaced by the corresponding image timing SEI message and buffer period SEI message from the nested SEI message.
[0077] Bit rate variants : HRD parameters such as initial_removal_delay and initial_removal_delay_offset, as well as Bitrate and CPB_siz are usually provided for several values. The idea is that the bitrate at which the bitstream is fed into the CPB can vary, and therefore several values are provided that produce a valid HRD model.
[0078] However, there are some aspects that have not been properly considered:
[0079] o There may only be bit rates that produce CBR
[0080] o If the appropriate bitrate is not known a priori, the bitstream cannot provide a valid HRD model (its parameters).
[0081] As outlined above, splicing at nonDiscardable points is a difficult task to achieve so far, and therefore, the embodiments in the following description aim to accurately achieve the possibility of splicing even at nonDiscardable points. The first simple correction is to correct the formula to not always use the non-discardable image as anchor, but to use the last image received after the splice point. That is, from the following formula:
[0082] Removal(previousNonDiscardable)+SplicingDelta
[0083] Where SplicingDelta is set to AuCPBRemovalDelta and InitialCPBRevovalDelay(i)+t af 2-t rm Maximum value of 2
[0084] Change to the maximum of:
[0085] -Removal(previousNonDiscardable)+AuCPBRemovalDelta
[0086] -Removal(lastPictureBeforeSplicing)+InitialCPBRevovalDelay(i)+t af 2-t rm 2, where Removal(lastPictureBeforeSplicing) is actually t rm 2. (In the example above)
[0087] Also, as can be seen in this formula, if the splicer decides to treat the previous image as the last image before splicing, such as t rm 1, the maximum of the two cases will always be Removal(previousNonDiscardable)+AuCPBRemovalDelta. Unless the splicer changes the value of AuCPBRemovalDelta accordingly and determines (t af 2-t rm The difference of 2) makes seamless switching feasible, which would otherwise prevent seamless splicing at any AU.
[0088] It is possible to change the value of AuCPBRemovalDelta during the buffering period SEI. However, the splicer keeps track of (t af X and t rm X) is not straightforward, because keeping track of (t af X and t rm X) is not that simple.
[0089] Since it is desirable to make seamless splices at different points, some signaling to the splicer indicating whether a given picture allows seamless splicing would be desirable. This can be done by checking at the encoder side (t af X and t rm X) value, and ensure that the difference (t rm Xt afX) does not become less than a given value. This signaling may, for example, appear in a picture timing SEI associated with a picture that can be used as the last picture before the splicing point. An example may be Fig.26 Seen in.
[0090] Or, even in the case of nonDiscardable images, subsequent discardable images are guaranteed to satisfy the requirements. Examples can be found at Fig. 27 Seen in.
[0091] Due to (t rm Xt af X) has to meet a minimum requirement on the initial_removal_delay of the spliced bitstream. There should be some indication at the buffering period SEI about what value a picture can use for seamless splicing. Examples can be found in Fig.28 Seen in.
[0092] Therefore, as a result of the thinking about possible modifications that have just been outlined, the following examples were evaluated.
[0093] exist Fig.25 and Fig.26 , Fig. 27 and Fig.28 In the signalling example of FIG. 2 , it can be seen that the video data stream may comprise a sequence 23a of picture portions, such as access units, in each picture portion a picture 10a of a video 12a is encoded. Fig.25 Two data streams 14a and 14b are depicted, and streams 14a and 14b are shown spliced at a splice point 92 such that the tail end of the portion of stream 14a before (i.e., to the left) the splice point 92 abuts the front end of the portion of stream 14b after (i.e., to the right) the splice point 92. The result of the splicing performed by a splicing device such as a server, a video conferencing system, or a master-slave streaming system is shown at 14c, i.e., the spliced data stream.
[0094] The picture parts may include first type picture parts, which are shown with cross hatching or simply hatching in some figures and are referred to as marked pictures above. A corresponding example thereof is an AU with or including a buffering period SEI. These first type picture parts are used as CPB removal time references, i.e. their CPB removal is used as a reference anchorTime for other timings given as time offsets such as AuCPBRemovalDelay.
[0095] The image portions may further include a second type of image portions, which are shown unhatched in some of the figures, such as images not including any buffer period SEI.
[0096] The data streams 14a and 14b additionally comprise first timing information 95, in each picture portion AuCPBRemovalDelay or au_cpb_removal_delay_minus1 being compared which informs about the first CPB removal time of the respective picture portion. The first CPB removal time measures the elapsed time since the removal of the previous picture portion of the first type, i.e. Fig.25 The picture part which is located before in the coding or bit stream order shown in , in particular the most immediately preceding picture part.
[0097] The data streams 14a and 14b further comprise second timing information, comparing initial_removal_delay in each of the predetermined first type picture parts, said initial_removal_delay possibly being present in the picture timing syntax of the buffering period SEI, such as for example before the splice point indication discussed below. These predetermined first type picture parts are Fig.25 In the example, the second timing information is indicated with cross-hatching in order to distinguish it from other first type image parts that are simply shown with hatching and are specific first type image parts, such as, for example, an AU with a buffering period SEI indicated as RAP, or in other words, an AU including a buffering period SEI and indicated as RAP. The second timing information notifies a second CPB removal time about the corresponding predetermined first type image part in the case of resuming the decoding of the video data stream from the corresponding predetermined first type image part onwards. The second CPB removal time measures the elapsed time from the arrival of the CPB of the first bit of the corresponding predetermined first type image part, that is, arriving at the CPB of the decoder.
[0098] At least one of the data streams (in Fig.25 The image portion is shown as 14a in the figure, i.e., division or splicing should be performed in order to obtain another stream (here 14b) attached to its splice point 92) may further include a splice point indication 94, 94', such as a spliceable_flag or a following_pic_spliceable_flag, which indicates the picture portion (e.g., no RAP), the time difference between the final CPB arrival of the picture portion and the CPB removal (such as the t indicated above). rm #-t af #) exceeds a predetermined threshold, so that it can be used as the last image portion before the splicing point 92 with another video data stream 14b, that is, it is suitable for allowing splicing.
[0099] It should be noted that the threshold value may be calculated, for example, as follows:
[0100] Assume that the removal time is equal and equal to 1 / frame rate. Then, the newly spliced AU (such as Fig.25The expected removal time of AU 23b') in rm #+1 / frame rate, where t rm # is the removal time of the immediately preceding AU or image part pointed to by 23'a.
[0101] t rm #+1 / Frame rate should be equal to or greater than t af #+initial_removal_delay, where t af # is the CPB final arrival time of the corresponding AU23b'. Therefore, the "predetermined threshold" is initial_removal_delay-1 / frame rate.
[0102] And, in the case of a variant of max_val_initial_removal_delay_for_seamless_splicing, ie a variant in which this threshold is explicitly indicated in the last data stream (such as the data stream 14a to which the other stream 14b is to be appended), t rm #+1 / frame rate is equal to or greater than t af #+max_val_initial_removal_delay_for__seamless_splicing. Then, the "predetermined threshold" is max_val_initial_removal_delay_for_seamless_splicing-1 / frame rate.
[0103] That is, the video data stream 14a may also include an indication 99 of a maximum second CPB removal time value, which indicates that as long as the second CPB removal time of the starting first type image portion 23b' of the further video data stream 14b concatenated with the video data stream 14a at the splicing point 92 is lower than the maximum second CPB removal time value, the further video data stream 14b is concatenated to the time difference t between its final CPB arrival and its CPB removal. rm #-t af #Any of the image portions indicated by the splice point indication as exceeding a predetermined threshold will result in a seamless splice.
[0104] It should be noted that the buffering period SEI example given above includes an indication 99, a concatenation flag 95 and a third CPB removal time 98. This represents an example where both streams 14a and 14b carry the same kind of information data. In other examples, this data may be different.
[0105] The splice point indication 94, 94' may further comprise a flag 94 present at each of the picture parts or at each of the set of picture parts (eg without RAP) indicating, for the respective picture part, the time difference t between its final CPB arrival and its CPB removal rm #-t af #Whether it exceeds the predetermined threshold.
[0106] Alternatively or additionally, the splice point indication 94, 94' may further include a flag 94', which is present at each of the image parts or each of the set of image parts (for example in those image parts without RAP), and the flag indicates whether, for each image part from the corresponding image part onwards until the next first type image part, the time difference trm#-taf# between its final CPB arrival and its CPB removal exceeds a predetermined threshold.
[0107] The sequence of image portions may further include images of the video 12a that are encoded in some manner using temporal inter prediction 90, such as motion compensated prediction, so that the image portions are included in some drawings (such as Fig.25 ) shows the non-discardable image parts that are not underlined, and the discardable image parts that are underlined in the figure.
[0108] These non-discardable picture portions are decodable in an independent manner, while the discardable picture portions are not required for decoding the non-discardable picture portions, but may require the non-discardable picture portions to make themselves decodable. It should be noted that without decoding the discardable picture portions, the video can be decoded in the future without any problems.
[0109] That is, discardable picture parts are not used, for example, as reference pictures for temporal inter prediction, whereas non-discardable pictures may be used as reference pictures for temporal inter prediction. Data streams using layered temporal scalability coding represent examples of these picture parts: picture parts for the lowest (base) temporal layer may be non-discardable, while other parts may be discardable. Another example is an open GOP coded data stream, where RASL pictures may be discardable.
[0110] The video data stream 14c is spliced at the splice point 92, so that the splice point indication 94; 94' indicates the time difference t between its final CPB arrival and its CPB removal. rm #-t af The predetermined image portion 23a' exceeding the predetermined threshold is followed and adjoined by a first predetermined first type image portion 23b' originating from the clipped-in video data stream 14b.
[0111] The video data stream 14c comprises a concatenation flag 96 and third timing information 98 in the first predetermined first type image portion, comparing AuCPBRemovalDelta or au_cp_removal_delay_delta_minus1. The concatenation flag 96 and the third timing information 98 may have been present in the spliced-in data stream 14b before splicing, but the splicing device may have reset the flag 96 from a state indicating unsplicing to a state indicating splicing, and / or the third timing information 98 may have been set. That is, the concatenation flag 96 is set to a first state indicating that the video data stream has been spliced at the first predetermined first type image portion, and the third timing information indicates a third CPB removal time of the first predetermined first type image portion 23b', which is used to determine the CPB removal time in the case of splicing at the first predetermined first type image portion 23b'. The third CPB removal time measures the elapsed time from the last CPB removal of the non-discardable image portion. For example, starting from removing the most recent non-discardable picture portion of the video data stream that has been encoded in the spliced data stream 14c.
[0112] Optionally, each of the predetermined first type image portions shown with cross-hatching in some of the figures may include a cascade flag 96 and third timing information 98, and the cascade flag may be set to a first state and a second state, the second state indicating that the video data stream has not yet been spliced at the corresponding predetermined first type image portion.
[0113] The video encoder according to this embodiment can encode the video into a video data stream in the following manner. For example, it is assumed that this encoder will encode the data stream 14a. It can also be configured to encode the data stream 14b, as discussed above, and the data stream can be considered the same. The encoder will perform encoding so that the video data stream 14a includes a sequence 23a of image portions, each of which is encoded with an image 10a of the video 12a. The video encoder can then classify the image portions into a first type of image portion used as a CPB removal time reference, and a second type of image portion, as discussed above. The encoder can provide a video data stream with first timing information and second timing information, and for each of the predetermined image portions (these can, for example, include a discardable image portion), check whether the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold. If so, the corresponding predetermined image portion can be used as the last image portion before the splicing point 92 with another video data stream 14b.
[0114] The video encoder may further provide the video data stream 14a with splicing point indications 94, 94'.
[0115] For each predetermined picture portion, the video encoder may further use as a predetermined threshold a value determined based on the second CPB removal time of the nearest predetermined picture of the first type before the corresponding predetermined picture portion, such as the pictures before 23a' simply hatched in some figures.
[0116] For each predetermined picture portion, the video encoder may also use a value determined based on the maximum second CPB removal time value as a predetermined threshold and write an indication 99 of the maximum second CPB removal time value into the most recent first type predetermined picture preceding the corresponding predetermined picture portion, such as the picture preceding 23a' which is simply shown as hatched in the accompanying drawing.
[0117] The following is about checking how the spliced data stream as discussed above can be handled inside the decoder, such as by the HRD 46 inside the decoder or in other words by the device 46 for managing the CPB 48 of the video decoder 44, which in turn decodes the video data stream 14c in a manner buffered by the CPB, that is, in a manner in which the image portion is received via the CPB at the removal time, wherein the image portion completely arrives at the CPB at the aforementioned final arrival time. This device can check whether the concatenation flag 96 in the first predetermined first type image portion 23b' indicates that a splicing-in has occurred at the first predetermined first type image portion. The device can then determine the time to remove the first predetermined first type image portion 23b' from the CPB.
[0118] If the concatenation flag in the first predetermined first type image portion indicates that cropping-in has not occurred at the first predetermined first type image portion, the removal time may be determined based on the first timing information 95 of the first predetermined first type image portion 23b'.
[0119] Furthermore, if the cascade flag 96 in the first predetermined first type image portion indicates that clipping occurred at the first predetermined first type image portion, the removal time can be determined based on the second timing information of the first predetermined first type image portion 23b', such as initial_removal_delay, and the third timing information 98 of the first predetermined first type image portion.
[0120] Alternatively, the apparatus may make this determination by determining the maximum between: the most recent CPB removal of the non-discardable image portion plus the third CPB removal time of the first predetermined first type image portion, followed by and adjacent to the first predetermined first type image portion 23a's CPB removal plus the second CPB time minus the time difference t between the final CPB arrival and CPB removal of the predetermined image portion 23a' rm #-t af#. This time is then used to remove the first predetermined first type image portion 23b' from the CPB.
[0121] Furthermore, according to this embodiment, a device is described, namely Fig.25 , for splicing two video data streams (here 14a and 14b) together to obtain 14c. Each data stream is as described above, so that each data stream comprises a sequence 23a, 23b of image portions, each image portion having an image 12a, 12b of a video 12a, 12b encoded therein.
[0122] The image portion is also as described above, including first and second type image portions, and first and second timing information.
[0123] The second video data stream will then include a concatenation flag 96 in the first predetermined first type image portion and third timing information 98. The concatenation flag 96 is set to indicate to the second video data stream that the first predetermined first type image portion continues the second state of the second video data stream, and the third timing information indicates a third CPB removal time of the first predetermined first type image portion, which is used to determine the CPB removal time when the second video data stream is spliced in at the first predetermined first type image portion, and the third CPB removal time measures the elapsed time from the most recent CPB removal of the non-discardable image portion.
[0124] The device can then check, for a predetermined image portion (e.g., no RAP), whether a splicing point indication 94; 94' in the first video data stream indicates that the time difference between its CPB arrival and its CPB removal exceeds a predetermined threshold, so that the predetermined image portion can be used as the last image portion of the first video data stream before being spliced with the second video data stream.
[0125] For example, in one variant, the predetermined threshold has been determined based on the initial_removal_delay of the buffered SEI of the first stream 14a or based on max_val_initial_removal_delay_for_seamless_splicing].
[0126] If the result of this check is affirmative, that is, "yes", the device can set the cascade flag 96 in the first predetermined first type image portion of the second video data stream to a first state indicating that the second video data stream has been spliced in at the first predetermined first type image portion, and cascade the first and second data streams at the predetermined image portion 26a' and the first predetermined first type image portion 26b', respectively, so as to obtain a spliced video data stream.
[0127] Furthermore, if so, the device may rewrite the third timing information 98 so as to measure the elapsed time from the most recent CPB removal of the non-discardable picture portion within the first video data stream within the spliced video data stream, rather than within the second video data stream.
[0128] In addition, if yes, the device may further check whether the elapsed time from the most recent CPB removal of the non-discardable image portion preceding the first predetermined first type image portion has changed when considering the spliced video data stream compared to the second video data stream, and if so, rewrite the third timing information 98 so as to measure the elapsed time from the most recent CPB removal of the non-discardable image portion of the first video data stream within the spliced video data stream.
[0129] Typically, the device may also check the following checks, and if both checks present a yes, perform setup and cascade.
[0130] First, whether the second CPB removal time of the latest first type predetermined image part of the first video data stream 14a before the corresponding predetermined image part or the upper limit determined from the second CPB removal time is greater than the second CPB removal time of the first predetermined first type image part 23b' of the second video data stream.
[0131] The last predetermined image portion of the first type of the first stream 14a is shown simply hatched in some of the figures.
[0132] The second CPB removal time of the most recent first type predetermined image portion may also have a CPB feed delay time for the addition of the most recent first type predetermined image portion. This measures the elapsed time to delay the CPB arrival of the first bit of the most recent first type predetermined image portion.
[0133] The corresponding predetermined image portion is shown simply as hatched in some of the figures, for example, before 23a'.
[0134] The second check is whether the maximum second CPB removal time value 99 indicated in the last predetermined picture portion of the first type is greater than the second CPB removal time of the first predetermined picture portion of the first type 23b' of the second video data stream.
[0135] For example, in one variation, the predetermined threshold may simply be the initial_removal_delay of the buffered SEI of the first stream. Alternatively, the threshold is initial_removal_delay+initial_removal_delay_offset, ie the CPB feed delay time. Further alternatively, as described in the second option, a maximum value max_val_initial_removal_delay_for_seamless_splicing may be signaled.
[0136] For example, AuCPBRemovalDelta needs to be rewritten only if the original value, which is the distance to the non-discardable image in the original second bitstream 14b, is different from the distance at the spliced bitstream 14c.
[0137] For example, it can be assumed that both bit streams have the same frame rate and that the following bit streams shown in decoding order are concatenated, i.e. spliced:
[0138] Second bitstream: RAP1, B0, B1 (non-discardable), B2, RAP2 (splicing_point)
[0139] 1st bitstream: rap1, b0 (non-discardable), b1, b2, b3, b4
[0140] Splice together:
[0141] rap1, b0 (indispensable), b1, b2, b3, RAP2 (splicing_point)
[0142] Originally, the buffering period SEI for RAP2 will include concatenation_flag set to 0 and AuCPBRemovalDelta equal to 2 / frame rate.
[0143] In the spliced bitstream, concatenation_flag shall be set to 1 and AuCPBRemovalDelta shall be equal to 4 / frame rate.
[0144] However, if the spliced bitstream is rap1, b0 (non-discardable), b1, RAP2 (splicing_point), then concatenation_flag is set to 1 and AuCPBRemovalDelta is equal to 2 / frame rate.
[0145] Therefore, the first case requires overriding AuCPBRemovalDelta but the second case does not.
[0146] For a spliced stream comprising an initial removal delay less than max_val_initial_removal_delay_for_seamless_splicing, seamless splicing can be achieved if the flag at the picture timing SEI indicates so. If the initial removal delay of the spliced stream is higher, then it is obviously not known whether the stream can be spliced.
[0147] The above embodiments are about the problem of implementing splicing at a non-discardable point, while the following description is about how to implement the problem of presenting available removal time that can also be used for removable picture parts such as DRAP. In particular, the subsequent embodiments are about providing timing information (removal time) in the case of performing discarding some AUs, such as in the case of RASL pictures of an open GOP structure when performing random access or performing a splicing operation. In other words, whether the removal time is also applicable to dependent random access point DRAP pictures.
[0148] The idea here is to modify the initial_removal time of the buffering period SEI of the RAP. In the following different cases are discussed and examples of removal time of AUs are given.
[0149] Figure 1 1 shows the inventive concept according to the present application. Therein is depicted a video data stream in which a video 12 is encoded and a first CPB removal time t is to be applied at the decoder side in case of buffering a complete version 15 of the video data stream 14. rm (1) t 16, and about a second CPB removal time t to be applied at the decoder side for buffering the reduced version 20 of the video data stream 14. rm (2) The reduced version of the video data stream comprises a second timing information 18, the reduced version differing from the video data stream by removing 22 a portion 24 of the video data stream from a complete version of the video data stream.
[0150] Optionally, the second timing information 18 is provided by means of information on how to modify the first CPB removal time t rm (1) To obtain the second CPB removal time t rm (2) The timing modification information defines the second CPB removal time t rm (2) An example of this timing modification information is, for example, init_removal_delay_correction_offset.
[0151] Furthermore, the first timing information 16 may signal the first CPB removal time t t on a picture-by-picture basis and by means of an increment relative to a preceding marked picture in the decoding order 26.rm (1) .
[0152] The increment may exemplarily be signaled by AuCPBRemovalDelay or au_cpb_removal_delay_minus1. A marked picture may be, for example, at a clean random access point CRA and is a picture containing a buffering period SEI. It is typically a RAP picture but may be other base pictures, such as a temporal layer 0 picture.
[0153] In other words, each picture portion 23 of the video data stream comprises a delta measuring the delay of its removal from the CPB 48 relative to the RAP picture that precedes the picture portion 23 along the coding order 26. A picture portion 23 is also called an access unit AU, such as in the case of HEVC.
[0154] Optionally, the marked images are marked by means of a buffer period message within the image portion of the video data stream to which the marked images relate.
[0155] Furthermore, means 46 for managing a CPB 48 of a video decoder 44 decoding a video data stream may manage the CPB according to the first timing information 16 if the video data stream includes the portion and may manage the CPB according to the second timing information 18 if the portion 24 has been removed.
[0156] Figure 5 The complete bitstream is shown. The initial removal time of the clean random access point CRA is 8.
[0157] exist Figure 7 In , the RASL image is removed. The initial removal time of CRA is 7. The removal times of subsequent images have an incremental shift of 4 compared to the previous case.
[0158] Figure 6 Shown are the removed RASL pictures and the TRAIL (normal post) pictures up to (but excluding) TRAIL4 (which is a DRAP). The initial removal of CRA at this point is 11, making the CPB level the same as at the start of the bitstream.
[0159] One option would be to add alternative timings for different possibilities of dropping frames. That is, one timing for removing RASL images, one timing for removing until the first DRAP, one timing for removing until the second DRAP, and so on.
[0160] However, this will increase the size of the buffering period SEI, and will need to indicate to the receiver what case it applies to, i.e. RASL removed, all AUs up to the first DRAP, all AUs up to the second DRAP, etc. This can be done via SEI.
[0161] However, it would also require encoding of all DRAPs to be done so that the buffered SEI can be written appropriately, which would result in additional delay at the encoder / transmitter side.
[0162] Instead, a single value is indicated at the buffer SEI message, and this value can be modified by subsequent SEI, such as the picture timing SEI message of the next non-removed AU. Then when the removal of AU is performed, only the flag aus_since_rap_removed_flag needs to be set. Its corresponding example can be found in Figure 8 Seen in.
[0163] In combination with the above-mentioned figures, it can be seen that the second timing information 18 can limit the second CPB removal time t by means of the first timing modification information 19 or by means of the first timing replacement information. rm (2) The first timing modification information is about how to modify the first CPB removal time t indicated by the first timing information relative to a first picture portion 28' of the video data stream that is associated with the first predetermined picture CRA and precedes the portion 24' in the video data stream. rm (1) , to obtain a second CPB removal time t relative to the first image portion 28' rm (2) The first timing replacement information is used to partially replace the first CPB removal time relative to the first image to obtain a second CPB removal time.
[0164] The first timing modification information 19 may be signaled, for example, by init_removal_delay_correction_offset.
[0165] Furthermore, first timing modification information 19 or first timing substitution information is signaled in the video data stream in a second picture portion 30' of the video data stream after the portion 24' in the video data stream and associated with the second predetermined picture TRAIL1.
[0166] It should be noted that SEI messages may be handled as also buffered in the CPB, and therefore, may be taken into account when determining the first and second timing information. The first and second timing information may be calculated and delivered once for the SEI message that delivers the included first and second timing information, and once for the discarded SEI message.
[0167] Furthermore, the second image portion may optionally include a signalling 32 indicating whether a portion has been removed.
[0168] Furthermore, the first timing modification information 19 or the first timing replacement information may be signaled in the video data stream regardless of whether the signaling 32 indicates that the portion has been removed or the signaling 32 indicates that the portion has not been removed.
[0169] The first predetermined picture may be a marked picture, such as an intra random access point IRAP or a dependency random access point DRAP picture, ie a picture used as a temporal reference, as described in detail above.
[0170] Furthermore, the second predetermined image may be a TRAIL or DRAP image.
[0171] Furthermore, the second timing information 18 may additionally define a second CPB removal time t rm (2) The second timing modification information is about how to modify the first CPB removal time t indicated by the first timing information relative to the set of one or more third image parts 31' of the video data stream. rm (1) , to obtain a second CPB removal time t relative to a set of one or more third image parts 30' rm (2) , said third picture portion being associated with a third predetermined picture, following the portion 24' in the video data stream and comprising the second picture portion 28', and the second timing modification information 21 may be signaled within the second picture portion 30' in the video data stream.
[0172] The second timing modification information 21 may be signaled, for example, by cpb_removal_delay_offset.
[0173] The second image portion may also comprise a signalling 32 indicating whether the portion has been removed, and the second timing modification information 21 may be signalled in the video data stream whether the signalling 32 indicates that the portion has been removed or the signalling 32 indicates that the portion has not been removed.
[0174] One option is that the set of one or more third image portions 31' of the video data stream extends up to a fourth image portion associated with a marked image, wherein the marked image can be used as a starting point for a timing reference.
[0175] Finally, the marked images may be marked by means of a buffer period message within the image portion of the video data stream to which the marked images relate.
[0176] It should be noted that in the case shown for DRAP, the initial removal of CRA is 11 instead of 8, because the full bit stream is considered and therefore the CPB fill level is higher than the original case. This may bring potential problems and buffer overflow.
[0177] Another option would be to allow (in case of DRAP) that the removal time from CPB is not equal for all frames. More specifically, for RAP and DRAP, there can be unequal distances, while for all other AUs, the distances are equal.
[0178] In this case, the buffering period SEI message for RAP can have two timings. One timing for the entire bitstream or when RASL is removed, and one timing for the DRAP case. Since no RAP is output in the second case, the removal time of the RAP can be such that the final arrival time of the RAP is equal to its removal time. Then, the DRAP AU will have a second removal time, which will build the buffer to the desired level. In this case, the final arrival time of the RAP is the earliest arrival time of the DRAP in both CBR and VBR cases. The described solution is shown in Fig. 9 middle.
[0179] An exemplary signaling syntax is shown in Fig.10 as well as Fig.11 middle.
[0180] In one embodiment, the second timing information 18 may define the second CPB removal time t by means of the third 33 and / or fourth 35 timing modification information on how to modify or by means of the third 33 or fourth 35 timing replacement information. rm (2) .
[0181] The third timing modification / replacement information 33 is about how to modify or replace the first CPB removal time t indicated by the first timing information relative to the first picture portion 28' of the video data stream associated with the first predetermined picture CRA and preceding the portion 24" in the video data stream. rm (1) , to obtain a second CPB removal time t relative to the first image portion 28' rm (2) .
[0182] The third timing modification / substitution information 33 may be signaled, for example, by drap_operation_initial_removal_delay. It should also be noted that cpb_drap_operation_removal_offset is about the arrival time, i.e. the time when a certain picture portion, i.e. a marked picture marked by means of a buffering SEI message enters or is fed into the CPB. The feeding may be the responsibility of a device different from the device for managing the CPB as described in this application. Compliance with the arrival time may depend on the transmission intermediate network nodes.
[0183] The fourth timing modification / replacement information 35 is about how to modify or replace the first CPB removal time t indicated by the first timing information with respect to a set of one or more third picture portions 31" of the video data stream associated with the third predetermined picture and following the portion 24" in the video data stream. rm (1) , to obtain a second CPB removal time t relative to a set of one or more third image parts 30″ rm (2) .
[0184] The fourth timing modification / substitution information 35 may be signaled, for example, by cpb_drap_operation_removal_delay_offset.
[0185] In general, the means 46 for managing the CPB 48 of the video decoder may also perform the modifications or alternatives detailed above.
[0186] Optionally, third timing modification / substitution information 33 can be signaled in the video data stream within a first image portion 28' of the video data stream, and fourth timing modification / substitution information 35 can be signaled in the video data stream within a second image portion 30" of the video data stream, which second image portion is preceding in the set of one or more third image portions in terms of the encoding order 26.
[0187] In addition, the first image portion 28' may include a signal notification 36 indicating whether the distance between the second CPB removal times of the first and second image portions is allowed to deviate from the distance between the second CPB removal times of the consecutive second and third image portions, wherein conditionally when the signal notification 36 indicates that the distance between the second CPB removal times of the first and second image portions is allowed to deviate from the distance between the second CPB removal times of the consecutive second and third image portions, third timing modification information 33 or third timing replacement information 33 is signaled within the first image portion 28' of the video data stream.
[0188] The video encoder may check whether the second predetermined picture is a DRAP that references a first predetermined picture (which is a RAP), set a signal notification 36 to indicate whether the second predetermined picture is a DRAP that references the first predetermined picture (which is a RAP), and if the second predetermined picture is a DRAP that references the first predetermined picture (which is a RAP), encode third timing modification / substitution information 33 in the video data stream within a first picture portion 28' of the video data stream.
[0189] It should be noted that au_cpb_removal_delay_minus1 in the pic_timing SEI message usually increases between consecutive image parts, that is, it increases at the same distance. However, in the case of abandoning or not presenting or outputting the images encoded in the first image part 28', it is acceptable if the regular time distance is interrupted until the DRAP, and then valid from then on, because these images are actually output images. In other words, in the example described, au_cpb_removal_delay_minus1 in the pic_timing SEI message increases at the same distance. This is usually the case, but it is not prohibited to be otherwise. The example shows the initial full bitstream situation when nothing is removed. However, when all the content between the RAP and the DRAP is removed, this is no longer the case, but it is still possible, because when the DRAP functionality is used, the RAP is not output or shown at all.
[0190] Furthermore, the second image portion 30" may include a signalling 38 indicating whether the portion 24' has been removed.
[0191] The device 46 according to the invention for managing the CPB 48 of the video decoder 44 can postpone the removal of the first picture portion of the video data stream after checking the signalling 32 ; 38 .
[0192] Alternatively, the network node 42 for forwarding a video data stream according to the invention may set a signaling 32; 36 in order to indicate that the portion has been removed from the video data stream.
[0193] The network node 42 may also remove, ie discard, the portion 24 of the video data stream from the video data stream.
[0194] like Figure 5As can be seen in, optionally, the second timing information can define the second CPB removal time, so that when buffer removal of the second image portion is performed according to the first CPB removal time, the first CPB filling level 39a'; 39a" at the decoder side that displays itself at the decoder side after the first image portion of the video data stream before the portion in the buffered video data stream is equal to the second CPB filling level 39b'; 39b" at the decoder side, the first portion of the video data stream and the second image portion are after the portion in the video data stream, and the second CPB filling level is when the buffer removal of the second image portion is performed according to the second CPB buffer time, when the portions have not been buffered yet, and the first and second image portions of the buffered video data stream are displayed at the decoder side after the first and second image portions.
[0195] The video encoder may set the first and second timing information so that the video data stream complies with the above content.
[0196] More generally, the second timing information may define a second CPB removal time for a portion associated with one or more RASL image sequences, or define a second CPB removal time for a portion associated with an image sequence before a certain DRAP.
[0197] Furthermore, the video data stream may comprise signalling 32, 35 indicating whether the portion has been removed.
[0198] The DRAP offsets used for random access will be used to calculate the removal time of the AU later. Obviously, instead of indicating the offsets in the picture timing SEI message, the DRAP can be specified to include a buffering period SEI message and indicate these offsets in the buffering period SEI message.
[0199] The embodiments described subsequently are about the problem of temporal scalability and how to indicate information about picture buffer timing in picture timing SEI and buffering period SEI messages in this coding environment. Therefore, embodiments for alternative information in temporal scalability reference pictures in picture timing SEI messages and buffering period SEI messages are described. First, embodiments for alternative information in temporal scalability reference pictures in picture timing SEI messages and buffering period SEI messages, i.e. embodiments utilizing timing offsets in picture timing SEI messages, are presented.
[0200] Figure 12 to Figure 14 Some examples of removing time values when providing different frame rates with the same bitstream (ie, time scalability) are shown.
[0201] Fig.17The table in shows that the deltaTime for "blue" frames is 0, 1 / 60, 3 / 120, 2 / 60, and 6 / 120. The "red frames" are 3 / 120 and 6 / 120. The blue frames are indicated by these rows in the table, including values for 30, 60, and 120fps. The red frames are indicated by these rows in the table, which only include values for 60 and 120fps, but not 30fps.
[0202] for Fig.16 The open GOP case with the highest frame rate is shown in Figure 1. The following applies, also in Fig.15 In Fig.15 In the table, it can be seen that the deltaTime for the "blue" frames is 0, 1 / 60, and 3 / 120. The deltaTime for the "red frames" is 1 / 120 and 2 / 120. Likewise, the blue frames are indicated by these rows in the table, including values for 30, 60, and 120fps. The red frames are referenced by these rows in the table, which only include values for 60 and 120fps, but not 30fps.
[0203] In general, due to the GOP structure of the encoded bitstream, there are some repeating patterns in the deltaTime of the removal times at different frame rates.
[0204] Another aspect to consider is scene cuts, which are interruptions in the described pattern. Different locations of scene cuts will produce different values. For simplicity, Fig.18 Only one position is shown.
[0205] As from Fig.19 As can be seen from the table in , it is clear that the GOP affected by scene cuts does not follow the pattern.
[0206] Fig. 20 An exemplary signaling structure (ie, TargetTemporalId) referencing a picture timing SEI message including different values for different frame rates is shown.
[0207] exist Fig.14 In the embodiment shown in FIG. 1 , a video data stream is encoded in units of groups of pictures 50 using temporal inter-prediction in a temporally scalable manner, so that the pictures 12 of the video are grouped into different hierarchical levels 50. i of images, wherein for each hierarchical level, all images of the corresponding hierarchical level depend only on images of the corresponding level or of the hierarchical level below. In the accompanying drawings, only the beginning of one GOP is shown.
[0208] Furthermore, the portion 24 is composed of images belonging to the highest hierarchical level 503 , or to the highest hierarchical level 503 and one or more hierarchical levels 502 other than the lowest hierarchical level 501 immediately below the highest hierarchical level 503 in the hierarchical order 52 .
[0209] Next, the first timing information 16 signals the first CPB removal time t ... rm (1) . This order is indicated by the ordinal order of the score values used to label the images.
[0210] Likewise, the increment may be signaled exemplarily by AuCPBRemovalDelay or au_cpb_removal_delay_minus1.The current GOP is a picture relative to the preceding picture of the GOP to which the respective picture belongs.
[0211] Furthermore, the marked images may be marked by means of a buffer period message within the image portion of the video data stream to which the marked images relate.
[0212] In one embodiment, the second timing information 18 may define the second CPB removal time t by means of the fifth timing modification information 19 or the fifth timing replacement information 19 used for replacement. rm (2) , said information is signalled in the video data stream within each of the fifth image portions 23 of the video data stream, said fifth image portions 23 being related to a fifth image 54, said fifth image 54 being associated with one or more hierarchical levels other than the highest hierarchical level.
[0213] The fifth timing modification information 19 may be signaled, for example, by au_cpb_removal_pattern_offset, and the fifth timing replacement information 19 may be signaled, for example, by au_cpb_removal_delay_minus1[i], where i is not equal to the current hierarchical level to which the picture sending pic_timing belongs.
[0214] The fifth timing modification / replacement information 19 is about how to modify or replace the first CPB removal time t indicated by the first timing information with respect to the corresponding fifth image portion 23. rm (1) , to obtain a second CPB removal time t relative to the fifth image portion 23 rm (2) .
[0215] The first timing information may be signaled, for example, by au_cpb_removal_delay_minus1[i], where i is equal to the current hierarchy level to which the picture for which pic_timing is sent belongs, or au_cpb_removal_delay_minus1.
[0216] In case the portion 24 comprises the highest hierarchical level down to the respective hierarchical level, above the respective hierarchical level, the fifth timing modification / substitution information 19 is signaled for each hierarchical level between the highest and lowest hierarchical levels (i.e., 501 and 502 in the example shown), and for each hierarchical level above the respective hierarchical level between the highest and lowest hierarchical levels (i.e., 503 for 502, and 503 and 502 for 501 in the example shown). As an example, the number is indicated by num_sub_layer_cpb_removal_delays_minus1 in the case where buffering_period_pattern_flag is one, and by num_sub_layer_cpb_removal_delays_minus1+1 in the case where buffering_period_pattern_flag is zero.
[0217] Fig.21 as well as Fig. 22 An exemplary signaling structure is shown, where alternatively, patterns may be defined within each buffering period SEI and these patterns may be indicated in the picture timing SEI message. When, as described above, for example as a result of a scene cut, the pattern may not be applied, the value may be explicitly signaled (or in the form of some offset).
[0218] According to this embodiment, the second timing information 18 defines the second CPB removal time t rm (2) The fifth timing modification information 19 indicates how to modify the first CPB removal time by means of an offset value, and the video data stream comprises data 60 indicating a list of representative offset values taken by the offset value indicated by the fifth modification information for modifying the first CPB removal time in one or more picture groups. Next, the fifth timing modification information 19 signals the offset value by means of a pointer to the list of representative offset values.
[0219] More generally, in the above embodiment, the video data stream is reduced by a portion 24, i.e. the result is a reduced version 20, and the video data stream comprises the first and second timing information 18. In other words, in case certain hierarchical levels are discarded at a certain intermediate network device, the second timing information is not nested so as not to be nested and replace the corresponding first timing information. In fact, the first and second timing information remain unchanged and at the decoder side, depending on the signaling sent in the video data stream, the correct timing information is selected for CPB handling and has been arranged to explain to the intermediate network device which portion has been removed, or more precisely, which previous highest hierarchical level has been removed / discarded.
[0220] A further example of an embodiment for alternative information in temporal scalable reference pictures in picture timing SEI messages and buffering period SEI messages is an increment at the buffering period SEI message.
[0221] like Fig.23 As can be seen in , different values of vcl_initial_cpb_removal_delay[i] and vcl_initial_cpb_removal_offset[i] should also be explicitly indicated in the buffering period SEI message, because a single value may not be sufficient for all possible frame rates.
[0222] Another embodiment involves multiple bit rate considerations.As discussed above, the current way to support different transmission rates or CPB feed rates is to indicate several values of bit rate, CPBsize and corresponding initial removal time and offset.
[0223] The problem with this current solution is that all potential CPB feed bit rates need to be known in advance, and the rate control at the encoder needs to take all bit rates into account to ensure that the information provided is valid.
[0224] Although this allows a high flexibility to a certain extent, where the mentioned parameters can be changed in many ways, in some practical situations, it may be desirable to consider many potential CPB feed rates from the sending side, but this will be difficult to implement at the encoder side. In these situations, a "very" restrictive rate control that will do some parameterization of the HRD model may be useful, which means that 2 extreme values will be considered - the maximum bit rate (maxBitrate) and the minimum bit rate (minBitrate), and anything in between can be linearly interpolated. This solution will be more restrictive than the flexibility solution in terms of HRD modeling, but will allow any rate between the maximum bit rate and the minimum bit rate to be used at the sending side, and still ensure that there will be no underflow or overflow of the CPBsize and the initial removal time that can be calculated based on the signaled values of the maximum and minimum values.
[0225] hrd_parameters is divided into 3 parts.
[0226] cbr_parameters_present_flag
[0227] ·vbr_min_parameter_present_flag
[0228] ·vbr_max_parameter_present_flag
[0229] Any other bit rate between the two provided for vbr_min_parameter_present_flag and vbr_max_parameter_present_flag may be calculated. Example signaling is shown in Fig.24a , Fig.29 as well as Fig.30 middle.
[0230] Similarly, if CBR, minimum and maximum values are also present in hrd_parameters, the buffering period SEI message shall include the values, such as Figure 24b as well as Fig.31 as shown in .
[0231] According to this, the HRD timing information data, such as the first HRD timing information data, the second HRD timing information data or any one of the plurality of HRD timing information data, and each HRD timing information data includes one or more of the following:
[0232] - an indication 104 of the CPB feed bit rate associated with the corresponding HRD timing information,
[0233] - an indication 106 of the CPB size required for the corresponding CPB feed bit rate associated with the corresponding HRD timing information,
[0234] an indication 108 of a CPB removal time for each of the marked picture portions of the video data stream, measuring the elapsed time from the arrival of the CPB of the first bit of the respective marked picture portion for the respective CPB feed bit rate associated with the respective HRD timing information - the marked picture portions here are respectively picture portions with a buffering period SEI such as a RAP, which are shown cross-hatched in the other relevant figures, but also other pictures such as picture portions shown simply hatched in the other relevant figures,
[0235] - an indication 110 of the CPB feed delay time for each of the marked picture portions of the video data stream, measuring the elapsed time that would delay the arrival of the CPB of the first bit of the respective marked picture portion for the respective CPB feed bit rate associated with the respective HRD timing information.
[0236] If there is a minimum and maximum value, any value therebetween will result in, for example, a linear interpolation of the minimum and maximum values.
[0237] Moreover, since further flexibility is needed in some cases, it may be desirable not to limit the HRD model to a linear interpolation model (e.g., to nonlinearly reduce initial_removal_delay and increase bit rate). Therefore, depending on the scenario, the encoder can choose which mode to use: a mode with linear interpolation or a mode with multiple values. Exemplary signaling is shown in Fig.32 , Fig.33 as well as Fig.34 middle.
[0238] As shown in Figure 24, Fig.29 , Fig.31 , Fig.32 as well as Fig.34 As shown in , the video stream may include first HRD timing information data 70a, 70b associated with a minimum CPB feed bit rate, and second HRD timing information data 72a, 72b associated with a maximum CPB feed bit rate. The first HRD timing information data 70a, 70b and the second HRD timing information data 72a, 72b then allow the third HRD timing information to be determined for the actual CPB feed bit rate by linear interpolation between the first and second HRD timing information data. The actual CPB feed bit rate is, for example, the bit rate at which the video data stream is actually fed into the CPB.
[0239] The video data stream may further include a flag 74, such as hrd_interpolation-mode_flag, indicating whether there is a first HRD timing information data 70a, 70b and a second HRD timing information data 72a, 72b, or a plurality of HRD timing information data 76 associated with each of a set of desired CPB feed bit rates.
[0240] The video encoder may then decide to provide the video data stream with the first HRD timing information 70 and the second HRD timing information 72, or to provide the video data stream with a plurality of HRD timing information data 76 associated with each of a set of desired CPB feed bit rates.
[0241] For all CPB feed bit rates between the minimum and maximum CPB feed bit rates, the video encoder may optionally perform the decision based on a feasibility check of whether linear interpolation between the first and second HRD timing information results in no CPB overflow and underflow conditions.
[0242] The means 46 for managing the CPB 48 of the video decoder 44 may then manage the CPB using the third HRD timing information.
[0243] Similarly, the device can check a flag 74 in the video data stream, such as hrd_interpolation-mode_flag, and, depending on the flag, perform deriving first HRD timing information data 70a, 70b and second HRD timing information data 72a, 72b from the video data stream, or deriving multiple HRD timing information data 76 associated with each of a set of desired CPB feed bit rates from the video data stream.
[0244] Fig.38 An example of three operating points is shown.
[0245] The interpolation case uses the following linear interpolation formula:
[0246] -BR interpolated =BR min +(BR max -BR min )*alpha
[0247] -IND interpolated =IND min +(IND max -IND min )*alpha(where IND=InitialRemovalDelay)
[0248] It should be noted that IND min (for minimum bit rate) greater than IND max (for highest bit rate)
[0249] Fig.37 Involving IND min , where the following abbreviations are used:
[0250] RmD = Removal Delay
[0251] Afinal=final arrival time
[0252] ArrivalEar = the earliest possible arrival time of the first bit of the image due to the frame rate
[0253] ArrivalInit = actual arrival time of the first bit of the image
[0254] It can be seen that the maximum CPB size is 15.
[0255] Fig.36 Involving IND max And, the CPB size required for this second scenario is 12,1125.
[0256] For the calculated initial removal delay and a bit rate linearly with an α of 0.3, Fig.35 The required CPB size to show this is 13,87912. If linear interpolation were to be performed, the calculated CPB size would correspond to 12,97875. As can be seen, the values are different.
[0257] Therefore, if the encoder wishes to obey the constraint of deriving the HRD parameters involved linearly with full flexibility within maximum and minimum values, the rate control at the encoder should take this constraint into account when determining the size of the picture so that the CPB size limit is met during interpolation.
[0258] Since this may not always be the case, one option would be to signal only if the encoder is aware that interpolation is possible, and if not, to signal whatever actual value has been used in the discrete number of operating points.
[0259] As mentioned before, the concepts described above may be employed by a video stream, a video encoder, a video decoder, a network node, an apparatus for managing a coded picture buffer CPB of a video decoder, and an apparatus for splicing video data streams together.
[0260] Likewise, the concepts described can be implemented by corresponding methods according to embodiments of the present invention. These methods are based on considerations identical to those of the video stream, video encoder, video decoder, network node, device for managing the coded picture buffer CPB of the video decoder, and / or device for splicing video data streams together described above. However, it should be noted that the methods can be supplemented by any of the features, functionality, and details described herein that are also relative to the video stream, video encoder, video decoder, network node, device for managing the coded picture buffer CPB of the video decoder, and / or device for splicing video data streams together. In addition, the methods can be supplemented individually and in combination by features, functionality, and details of the video stream, video encoder, video decoder, network node, device for managing the coded picture buffer CPB of the video decoder, and / or device for splicing video data streams together.
[0261] Finally, the concept can also be used to generate a coded data stream according to an embodiment of the present invention. The data stream can also be supplemented by the features, functionalities and details of the method, video encoder, video decoder, network node, device for managing a coded picture buffer CPB of a video decoder, and / or device for splicing video data streams together, individually and in combination.
[0262] Finally, the embodiments described herein may optionally be supplemented by any of the important points or aspects described herein. However, it should be noted that the important points and aspects described herein may be used individually or in combination, and may be introduced into any of the embodiments described herein individually and in combination.
[0263] Although some aspects have been described in the context of an apparatus, it is apparent that these aspects also represent descriptions of corresponding methods, wherein the apparatus or a portion thereof corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent descriptions of a corresponding apparatus or an apparatus portion or an object or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device such as (for example) a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by this apparatus.
[0264] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to cause each method to be executed. Therefore, the digital storage medium may be computer readable.
[0265] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
[0266] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is operative for performing one of the methods. The program code may, for example, be stored on a machine readable carrier.
[0267] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
[0268] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
[0269] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.
[0270] Therefore, a further embodiment of the inventive method is a data stream or a sequence of signals representing a computer program for executing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection (for example, via the Internet).
[0271] A further embodiment comprises processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.
[0272] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
[0273] A further embodiment according to the invention comprises an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
[0274] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, the field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by any hardware device.
[0275] The apparatus described herein may be implemented using a hardware device or using a computer or using a combination of a hardware device and a computer.
[0276] The means described herein or any element of the means described herein may be implemented at least partially in hardware and / or in software.
[0277] The methods described herein may be performed using a hardware device or using a computer or using a combination of a hardware device and a computer.
[0278] The methods described herein, or any part of the methods described herein, may be performed at least in part by hardware and / or by software.
[0279] The embodiments described above merely illustrate the principles of the present invention. It should be understood that modifications and variations to the configurations and details described herein will be apparent to those skilled in the art. Therefore, it is intended to be limited only by the scope of the following claims, rather than by the specific details presented by way of the description and explanation of the embodiments herein.
Claims
1. A video data stream (14a; 14c), comprising a sequence (23a) of picture parts, in each picture part a picture (10a) of a video (12a) is encoded, the picture parts comprising a picture part of a first type used as a temporal reference for CPB removal and a picture part of a second type, first timing information (95) in each image portion relating to a first CPB removal time of the respective image portion, said first CPB removal time measuring the elapsed time since the removal of a previous image portion of the first type, and second timing information in each of the predetermined first type picture portions, relating to a second CPB removal time of the corresponding predetermined first type picture portion in case of resuming decoding of the video data stream onwards from the corresponding predetermined first type picture portion, the second CPB removal time measuring the elapsed time from the arrival of the CPB of the first bit of the corresponding predetermined first type picture portion, The splice point indicator (94, 94') indicates the time difference between the final CPB arrival and its CPB removal (t rm #-t af #) exceeds a predetermined threshold value so that the image portion can be used as the last image portion before a splicing point (92) with another video data stream (14b).
2. The video data stream according to claim 1, include: an indication (99) of a maximum second CPB removal time value, which indicates that as long as the second CPB removal time of the starting first type picture portion of the further video data stream (14b) at which the further video data stream is concatenated with the video data stream at the splicing point (92) is lower than the maximum second CPB removal time value, the further video data stream is concatenated to the time difference (t2) between its final CPB arrival and its CPB removal rm #-t af #) Any one of the image portions indicated by the splicing point indication as exceeding the predetermined threshold will result in seamless splicing.
3. The video data stream according to claim 1, wherein the splice point indication (94, 94') comprises A flag (94) present at each of said image parts or at each of a set of image parts, indicating, for the respective image part, said time difference (t rm #-t af #) exceeds the predetermined threshold, or a flag (94') present at each of said image parts or at each of a set of image parts indicating, for each image part from said respective image part onwards up to the next image part of the first type, said time difference (t rm #-t af #) exceeds the predetermined threshold.
4. The video data stream according to claim 1, wherein The sequence of picture portions has pictures of the video (12a) encoded therein using inter-temporal prediction (90) such that the picture portions include non-discardable picture portions which are decodable in an independently manner and discardable picture portions which are not required for decoding the non-discardable picture portions, wherein the video data stream (14c) is spliced at a splice point (92) such that the splice point indication (94; 94') indicates the time difference (t rm #-t af #) the predetermined image portion (23a') exceeding the predetermined threshold value is followed by and adjacent to a first predetermined first type image portion (23b') originating from the video data stream (14b) cut into, wherein said video data stream (14c) comprises in said first predetermined first type image portion, A cascade flag (96) and third timing information (98), wherein the cascade flag (96) is set to a first state indicating that the video data stream has been spliced at the first predetermined first type image portion, and the third timing information indicates a third CPB removal time of the first predetermined first type image portion, which is used to determine the CPB removal time when splicing is performed at the first predetermined first type image portion, and the third CPB removal time measures the elapsed time from the most recent CPB removal of the non-discardable image portion.
5. The video data stream according to claim 4, wherein each predetermined first type image portion comprises a cascade flag (96) and third timing information (98), wherein the cascade flag can be set to the first state and a second state, wherein the second state indicates that the video data stream has not been spliced at the corresponding predetermined first type image portion.
6. A video encoder for encoding a video into a video data stream such that the video data stream comprises a sequence (23a) of picture parts, each picture part having a picture (10a) of the video (12a) encoded therein, wherein the video encoder is configured to classifying the picture parts into a first type of picture parts used as a CPB removal temporal reference and a second type of picture parts, Provide the video data stream first timing information in each image portion relating to a first CPB removal time of the respective image portion, said first CPB removal time measuring the elapsed time since removal of a previous image portion of the first type, and second timing information in each of the predetermined first type picture portions, relating to a second CPB removal time of the corresponding predetermined first type picture portion in case of resuming decoding of the video data stream onwards from the corresponding predetermined first type picture portion, the second CPB removal time measuring the elapsed time from the arrival of the CPB of the first bit of the corresponding predetermined first type picture portion, For each of the predetermined image parts, the time difference (t rm #-t af #) exceeds a predetermined threshold, so that the predetermined image portion can be used as the last image portion before a splicing point (92) with another video data stream (14b), and The video data stream (14a) is provided with a splice point indication to indicate the time difference (t rm #-t af #) those predetermined image portions that exceed said predetermined threshold.
7. The video encoder according to claim 6, configured to provide the following when providing a splice point indication to the video data stream (14a): A flag (94) is provided to each of the image parts or each of the set of image parts, which indicates, for the respective image part, the time difference (t rm #-t af #) exceeds the predetermined threshold, or A flag (94') is provided to each of the image parts or each of the set of image parts, which indicates, for each image part from the respective image part onwards up to the next image part of the first type, the time difference (t rm #-t af #) exceeds the predetermined threshold.
8. The video encoder according to claim 6, configured to encoding images of the video (12a) as a sequence of image portions using inter-temporal prediction (90) in such a way that the image portions include discardable image portions that are not used as reference images for the inter-temporal prediction and non-discardable images that are used as reference images for the inter-temporal prediction, A cascade flag (96) and third timing information (98) are provided to each of the predetermined first type image parts, wherein the cascade flag can be set to a second state indicating that the video data stream has not been spliced at the corresponding predetermined first type image part, and the third timing information indicates a third CPB removal time of the corresponding first predetermined first type image part, which is used to determine the CPB removal time when splicing is performed at the corresponding first predetermined first type image part, and the third CPB removal time measures the elapsed time from the most recent CPB removal of the non-discardable image part.
9. The video encoder according to claim 6, configured to, for each predetermined image portion, using as the predetermined threshold a value determined based on a second CPB removal time of a most recent predetermined image of the first type preceding the corresponding predetermined image portion, A value determined based on the maximum second CPB removal time value is used as the predetermined threshold value, and an indication (99) of the maximum second CPB removal time value is written into a most recent predetermined picture of the first type preceding the corresponding predetermined picture portion.
10. A device (46) for managing a CPB (48) of a video decoder (44) for decoding a video data stream (14c) according to claim 36 in a manner buffered by the CPB, the device being configured to check whether a concatenation flag in a first predetermined first type picture portion indicates that a clipping has occurred at the first predetermined first type picture portion, The time for removing a first predetermined first type image portion (23b') from the CPB is determined as follows If the concatenation flag in the first predetermined first type picture portion indicates that no clipping has occurred at the first predetermined first type picture portion, is determined based on the first timing information of the first predetermined first type image portion, and If the concatenation flag in the first predetermined first type picture portion indicates that a clipping has occurred at the first predetermined first type picture portion, The determination is then made based on the second timing information and the third timing information of the first predetermined first type image portion.
Citation Information
Patent Citations
Video coding with improved random access point picture behaviors
CN104641637A
Syntax and semantics for buffering information to simplify video splicing
CN105027570A
Low-delay video buffering in video coding
EP2834975A1
Reference picture set signaling and restriction on an electronic device
US20140211849A1
Signaling change in output layer sets
US20140301476A1
