Virtual reference decoder

Abstract

To provide a video encoder, a video decoder, a network node, an apparatus for managing a coded picture buffer (CPB) of a video decoder, and an apparatus for splicing video data streams together.SOLUTION: A video encoded video data stream 14 includes first timing information relating to a first coded picture buffer (CPB) removal time to be applied at the decoder side when buffering a full version 15 of the video data stream, and second timing information relating to a second CPB removal time to be applied at the decoder side for buffering a reduced version of the video data stream, which differs from the video data stream resulting from removal of a portion of the video data stream from the full version of the video data stream.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The virtual reference decoder (HRD), and its use for checking the compatibility of bitstreams and decoders, is typically specified in video coding standards such as AVC (Advanced Video Codec) and HEVC (High Efficiency Video Coding).

[0002] Embodiments of the present invention relate to a video stream, a video encoder, a video decoder, a network node, a device for managing an encoded image buffer (CPB) of a video decoder, and a device for splicing video data streams together. [Background technology]

[0003] The following is an introductory chapter outlining the background of the present invention. To accomplish this, an HRD buffer model is specified, as shown in Figure 2, consisting of a virtual stream scheduler (HSS), an encoded image buffer (CPB), a decoding process (considered instantaneous), a decoded image buffer (DPB), and an output cropping process.

[0004] The model defines the timing and bitrate at which the bitstream is supplied to the encoded image buffer, the time it takes for its decoding unit (in low-latency operation mode, either an AU, access unit, or VCL NAL (VCL: Video Encoding Layer, NAL: Network Abstraction Layer) unit) to be removed from the CPB and decoded instantaneously, and the output time at which the image is output from the DPB.

[0005] By doing so, it is also possible to define the necessary CPB size to avoid underruns, where the decoder experiences a buffer overflow (more data is sent to the decoder than can be held in the CPB) or less data is sent to the decoder (at a lower bitrate than required), and the required decoding unit does not have the appropriate time in the decoder for decoding.

[0006] State-of-the-art encoding standards specify different parameters to describe bitstream and HRD requirements as well as buffer models.

[0007] For example, the hrd_parameter defined within HEVC for each sublayer consists of one or more pairs of Bitrate(i) and CPBsize(i), which indicate that no overflow or underflow will occur if the stream scheduler supplies a bitrate of Bitrate(i) to a CPB of size CPBsize(i).

[0008] In addition to the syntax elements of hrd_parameter, there is additional timing information within the bitstream that specifies the time for removing the image from the CPB, indicating when the VCL NAL units belonging to that image are sent for decoding. For the sake of ease of understanding, sub-image processing will be omitted from the explanation.

[0009] Related information can be found in buffering period SEI (supplementary enhancement information) messages containing InitialCPBRemovalDelay(i), InitialCPBRemovalOffset(i), and AuCPBRemovalDelay, and in image timing SEI messages containing AuCPBRemovalDelay. In its most basic operation, only InitialCPBRemovalDelay(i) and AuCPBRemovalDelay are used.

[0010] In this case, the first access unit to be decoded is the random access point RAP, and its corresponding buffering period SEI message and time 0 are defined as the time when the first bit of the random access point enters the CPB. Next, at time InitialCPBRemovalDelay(i), the image corresponding to the random access point is removed from the CPB, and further removal of non-RAP images from the CPB occurs in InitialCPBRevovalDelay(i)+AuCPBRemovalDelay (HEVC defines several parameters, namely ClockTick, for converting delays to time, but these are ignored here for simplicity).

[0011] When the next RAP comes, the removal time for the non-RAP image is calculated in the same way as before, i.e. InitialCPBRemovalDelay(i)+AuCPBRemovalDelay And this new value is used as an anchor for further deltas up to another RAP, i.e. anchorTime=InitialCPBRemovalDelay(i)+AuCPBRemovalDelay

[0012] Subsequently, image removal is performed using anchorTime + AuCPBRemovalDelay, and anchorTime is updated in the next RAP with a buffered SEI message. For example, anchorTime = anchorTime + AuCPBRemovalDelay. The operating modes described are the simplest. Further cases need to be considered. In the current situation, there are several drawbacks that the present invention would otherwise overcome. Details are as follows. [Overview of the project]

[0013] One embodiment of the present invention relates to a video data stream comprising: first timing information relating to a first encoded image buffer (CPB) removal time applied to the decoder when video is encoded and the full version of the video data stream is buffered; and second timing information relating to a second CPB removal time applied to the decoder to buffer a reduced version of the video data stream, which is different from the video data stream obtained by removing a portion of the video data stream from the full version of the video data stream.

[0014] Another embodiment of the present invention relates to a video data stream in which video is encoded and comprises first HRD timing information data relating to a minimum CPB supply bitrate and second HRD timing information data relating to a maximum CPB supply bitrate, wherein the first HRD timing information data and the second HRD timing information data enable the determination of a third HRD timing information for the actual CPB supply bitrate by linear interpolation between the first and second HRD timing information data.

[0015] Another embodiment of the present invention relates to a video data stream comprising: a sequence of image portions, each image portion having an image of an encoded video, and each image portion comprising a first type of image portion and a second type of image portion that function as a CPB removal time criterion; first timing information for each image portion relating to a first CPB removal time for each image portion, wherein the first CPB removal time measures the elapsed time since the removal of a preceding first type of image portion; second timing information for each of a predetermined first type of image portion relating to a second CPB removal time for each predetermined first type of image portion when decoding of the video data stream from each predetermined first type of image portion onward, wherein the second CPB removal time measures the elapsed time since the arrival of the CPB of the first bit of each predetermined first type of image portion; and a splice point display indicating an image portion whose time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold, so that it can function as the last image portion before a splice point having another video data stream.

[0016] Another embodiment of the present invention relates to a video encoder for encoding video into a video data stream and providing to the video data stream first timing information relating to a first CPB removal time applied to the decoder when buffering a complete version of the video data stream, and second timing information relating to a second CPB removal time applied to the decoder when buffering a reduced version of the video data stream, which is different from the video data stream due to the removal of a portion of the video data stream from the complete version of the video data stream.

[0017] Another embodiment of the present invention relates to a video encoder for encoding video into a video data stream and providing the video data stream with first HRD timing information relating to a minimum CPB supply bitrate and second HRD timing information relating to a maximum CPB supply bitrate, wherein the first and second HRD timing information enable the determination of a third HRD timing information of the actual CPB supply bitrate by linear interpolation between the first and second HRD timing information.

[0018] Another embodiment of the present invention is a video encoder for encoding video into a video data stream such that the video data stream comprises a sequence of image portions, each image portion having an image of the encoded video, the video encoder classifies the image portions into first type image portions and second type image portions which function as CPB removal time criteria, and provides to the video data stream first timing information for each image portion with respect to a first CPB removal time for each image portion, the first CPB removal time being a measure of the elapsed time since the removal of a preceding first type image portion, and each predetermined first timing information for resuming decoding of the video data stream from each predetermined first type image portion onward. The present invention relates to a video encoder that provides a second timing information for each of a predetermined first type of image portion relating to a second CPB removal time for the image portion of a predetermined first type, wherein the second CPB removal time measures the elapsed time from the arrival of the CPB of the first bit of each predetermined first type of image portion; checks each of the predetermined image portions to see whether the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold so that it functions as the last image portion before a splice point having another video data stream; and provides a splice point display to the video data stream to indicate these predetermined image portions where the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold.

[0019] Another embodiment of the present invention relates to a network node for transferring a video data stream in order to remove a portion of the video data stream from the video data stream.

[0020] Another embodiment of the present invention relates to an apparatus for managing a CPB of a video decoder that decodes a video data stream in a manner buffered by the CPB, wherein the apparatus manages the CPB according to first timing information when a portion consists of a video data stream, and second timing information when a portion is removed.

[0021] Another embodiment of the present invention relates to a device for managing the CPB of a video decoder that decodes a video encoded video data stream, the device derives first HRD timing information relating to a minimum CPB supply bitrate from the video data stream, second HRD timing information relating to a maximum CPB supply bitrate, determines third HRD timing information for the actual CPB supply bitrate by linear interpolation between the first and second HRD timing information, and manages the CPB using the third HRD timing information.

[0022] Another embodiment of the present invention relates to a device for managing a CPB of a video decoder that decodes a video data stream in a manner buffered by the CPB, the device checking whether a concatenation flag of the first predetermined first type of image portion indicates that a splice-in occurred in the first predetermined first type of image portion, and determining a time to remove the first predetermined first type of image portion from the CPB, wherein the determination is based on first timing information of the first predetermined first type of image portion if the concatenation flag of the first predetermined first type of image portion indicates that a splice-in did not occur in the first predetermined first type of image portion, and based on second and third timing information of the first predetermined first type of image portion if the concatenation flag of the first predetermined first type of image portion indicates that a splice-in occurred in the first predetermined first type of image portion.

[0023] Another embodiment of the present invention is an apparatus for splicing a first video data stream and a second video data stream together, each of which is a sequence of image portions, each image portion having an image of an encoded video, and the image portions comprising a first type of image portion and a second type of image portion that function as a CPB removal time criterion; a first timing information for each image portion relating to a first CPB removal time for each image portion, wherein the first CPB removal time measures the elapsed time since the removal of a preceding first type of image portion; and a predetermined first type of image information relating to a second CPB removal time for each predetermined first type of image portion when resuming decoding of the video data stream from each predetermined first type of image portion onward. The apparatus comprises a second timing information for each image portion, wherein the second CPB removal time measures the elapsed time from the arrival of the CPB of the first bit of each predetermined first type image portion, and the second video data stream comprises a concatenation flag and a third timing information for the first predetermined first type image portion, wherein the concatenation flag is set to a second state indicating the second video data stream continuing the second video data stream for the first predetermined first type image portion, and the third timing information indicates a third CPB removal time for the first predetermined first type image portion that helps determine the CPB removal time when the second video data stream is spliced ​​in with the first predetermined first type image portion, wherein the third CPB removal time measures the elapsed time from the most recent CPB removal of an image portion that cannot be discarded.The apparatus of the present invention checks whether the splice point indicator of the first video data stream indicates that the time difference between the arrival of a CPB and its removal exceeds a predetermined threshold for a given image portion, so that the first video data stream can function as the last image portion before being spliced ​​with the second video data stream. If so, it sets the concatenation flag of the first predetermined first type image portion of the second video data stream to a first state indicating that the second video data stream has been spliced ​​in at the first predetermined first type image portion, and concatenates the first and second data streams at the predetermined image portion and the first predetermined first type image portion, respectively, to obtain a spliced ​​video data stream.

[0024] The above concepts can be implemented by methods according to embodiments of the present invention. These methods are based on the same considerations as the decoders, encoders, devices, and data streams described above. However, it should be noted that the methods can be supplemented by any of the features, functions, and details described herein, and the same applies to the decoders, encoders, devices, and data streams. Furthermore, the methods can be supplemented by the features, functions, and details of the decoders, encoders, devices, and data streams, either individually or in combination.

[0025] Finally, the concept can also be used to generate encoded data streams according to embodiments of the present invention. The data streams can be supplemented individually or in combination with features, functions, and details of decoders, encoders, devices, and methods. Embodiments of the present invention will be described further with reference to the following accompanying drawings. [Brief explanation of the drawing]

[0026] [Figure 1] This figure shows an encoded data stream according to one embodiment of the present application. [Figure 2]This is a flowchart of a virtual reference decoder according to one embodiment of this application. [Figure 3] This figure shows the state of the encoded image buffer, including the removal time, according to one embodiment of this application. [Figure 4] This figure shows the state of the encoded image buffer, including the removal time, according to one embodiment of this application. [Figure 5] This figure shows the state of the encoded image buffer, including the removal time, according to one embodiment of this application. [Figure 6] This figure shows the state of the encoded image buffer, including the removal time, according to one embodiment of this application. [Figure 7] This figure shows the state of the encoded image buffer, including the removal time, according to one embodiment of this application. [Figure 9] This figure shows the state of the encoded image buffer, including the removal time, according to one embodiment of this application. [Figure 8] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 10] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 11] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 20] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 21] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 22] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 23] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 24a] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 24b] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 26] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 27] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 28] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 29] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 30] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 31] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 32] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 33] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 34] This figure shows an exemplary signaling syntax according to an embodiment of the present application. [Figure 12] This figure shows the data stream structure for different frame rates according to the embodiments of this application. [Figure 13] This figure shows the data stream structure for different frame rates according to the embodiments of this application. [Figure 14] This figure shows the data stream structure for different frame rates according to the embodiments of this application. [Figure 16] This figure shows the data stream structure for different frame rates according to the embodiments of this application. [Figure 18] This figure shows the data stream structure for different frame rates according to the embodiments of this application. [Figure 15] This figure shows exemplary removal time values ​​when different frame rates are provided in the same bitstream, according to embodiments of the present application. [Figure 17] This figure shows exemplary removal time values ​​when different frame rates are provided in the same bitstream, according to embodiments of the present application. [Figure 19]This figure shows exemplary removal time values ​​when different frame rates are provided in the same bitstream, according to embodiments of the present application. [Figure 25] This figure shows encoded data streams stitched together according to one embodiment of the present application. [Figure 35] This figure shows a table of values ​​for calculating the CPB size according to the embodiment of this application. [Figure 36] This figure shows a table of values ​​for calculating the CPB size according to the embodiment of this application. [Figure 37] This figure shows a table of values ​​for calculating the CPB size according to the embodiment of this application. [Figure 38] This is an exemplary graph of CPB fill levels according to an embodiment of the present application. [Modes for carrying out the invention]

[0027] In the figures, similar reference numerals indicate similar elements and features.

[0028] The following discussion addresses the underlying considerations of the present invention and describes several solutions. In particular, several details are disclosed, which can be optionally introduced into any of the embodiments disclosed herein. Specifically, the following description begins with a brief presentation of the problems that arise and are faced when dealing with splicing, stream extraction, stripping of scalable bitstreams, and bitstream supply under various transmission conditions. Next, specific modifications of the problems are presented, followed by embodiments that utilize the corresponding modifications. The following issues will be examined and addressed.

[0029] Splicing: To add an AuCPBRemovalDelay so that the removal time of the image's CPB is used, the removal time of the previous RAP with buffered SEI is used as an anchor.

[0030] However, when splicing occurs, it is not readily apparent what the previous value of anchorTime was at the splicing point. Therefore, in such cases, the derivation of the CPB removal time for a RAP with a spliced ​​buffered SEI is performed in a different way.

[0031] To solve this problem, HEVC includes two additional parameters: concatenation_flag, which indicates whether splicing occurred in the RAP containing the buffered-period SEI message, and AuCPBRemovalDelta, which refers to the delta in time to the previous non-discardable image instead of referring to the previous RAP with the buffered-period SEI message. This eliminates the need to calculate an arbitrary value and rewrite the buffered-period SEI message at the splicing point. The CPB removal time from the RAP where splicing occurs is then as follows: Removal(previousNonDiscardable)+AuCPBRemovalDelta

[0032] This assumes a seamless splicing scenario. However, it cannot be guaranteed that the CPB status will be the same at the splicing point of the original bitstream (when the RAP with SEI message results during the splicing buffering period is not spliced ​​and remains in its original bitstream) and at the time of splicing. More specifically, Figure 3 illustrates the problem described, where the upper bitstream in frame #3 is spliced ​​with the lower bitstream from frame #2 onwards.

[0033] Therefore, (the frame is t af (This will be earlier than when we arrived at point 3) Removal(previousNonDiscardable)+AuCPBRemovalDelta Instead of having a removal time, Removal(previousNonDiscardable)+InitialCPBRevovalDelay(i)+ taf 2 - t rm 2 The removal time of is used instead.

[0034] That is, the removal time of the spliced RAP is Removal(previousNonDiscardable)+SplicingDelta where SplicingDelta is AuCPBRemovalDeta and InitialCPBRevovalDelay(i)+t af 2 - t rm is set to the maximum value of 2. For example, as shown in Figure 3, if by chance the second value is greater than the first value, non - seamless splicing occurs.

[0035] The removal times of AuFinalArrivalTime and another image following the NonDiscardable image are unknown, and thus, if the difference between both values taf X and t rm is not the same as in the case of the NonDiscardable image, the derived removal time is incorrect (InitialCPBRemovalDelay(i)+t af X - t rm X). Therefore, this splicing operation only functions when the first bitstream ends with the corresponding NonDiscardable image.

[0036] CPB fullness and delay: Another problem that occurs in the HRD model is that a time delay for the removal of the first access unit is required to utilize the CPB buffer.

[0037] When the first RAP after splicing enters the CPB, the removal of its AU is delayed until InitialCPBRemovalDelay(i) time, as previously discussed. By supplying the CPB with Bitrate(i) during InitialCPBRemovalDelay(i), a given CPB fullness, e.g., CPB A This is achieved. Subsequently, in a further RAP with buffering period SEI messages, CPB A Larger CPB B It is impossible to achieve this. The reason can be seen by mathematically checking the earliest time that AU can enter CPB.

[0038] initArrivalTime[n]=Max(AuFinalArrivalTime[n-1],initArrivalEarliestTime[n]), where initArrivalEarliestTime[n]=RemovalTime[n]-InitCpbRemovalDelay(i)

[0039] This means that if the AU following the buffering period SEI message cannot enter the CPB earlier than its removal time InitialCPBRemovalDelay(i), then supplying Bitrate(i) to the CPB during InitialCPBRemovalDelay(i) will result in the CPB A Since CPB fullness is the only thing to achieve, CPB A Larger CPB B It means that it cannot be achieved.

[0040] To solve this problem, as shown in Figure 4, the virtual sender (or HSS in the figure) may delay the scheduling of the first RAP that has a buffered SEI message at a given time offset InitialCPBRemovalOffset(i). It is important to note that this only works with VBR and not with CBR, as shown below. initArrivalTime[n]=AuFinalArrivalTime[n-1] As a result, the scheduling will be changed as follows:

[0041] initArrivalEarliestTime[n]=RemovalTime[n]InitCpbRemovalDelay(i)-InitialCPBRemovalOffset(i)

[0042] This is CPB B This means that the CPB size can correspond to the size achieved by supplying Bitrate(i) to the CPB for InitCpbRemovalDelay(i) + InitialCPBRemovalOffset(i).

[0043] AU Dropping: The HEVC HRD model also supports dropping some NAL units. More specifically, "discardable" images associated with RAP images can be dropped. A "discardable" image is a RASL (Random Access Skip Reading) image, which in an open GOP (Group of Images) structure precedes a RAP image in presentation order but follows it in decoding order, and also references images that precede the RAP in decoding order. Therefore, when random access is performed on the corresponding RAP image, the RASL image cannot be decoded. For this reason, when splicing such a bitstream with another bitstream, or when starting a session at that RAP location, sending the RASL image is a waste of resources, and in some cases, these images are dropped before transmission. Clearly, when the bitstream is modified, the timing information, namely the initial arrival time and removal time of the image, changes.

[0044] Therefore, the buffering period SEI message includes alternative timing for when the RASL image associated with the RAP having the buffering period SEI message is dropped. ·AltInitCpbRemovalDelay(i) ·AltInitialCPBRemovalOffset(i)

[0045] Time scalability: When sublayers are dropped, timing information also changes. For example, if the original bitstream has 60Hz, the nominal removal time distance between two consecutive images in decoding order is expected to be 1 / 60th of a second. If every other image is dropped, the distance becomes 1 / 30th of a second. This means that the image timing SEI messages need to be replaced. Furthermore, InitCpbRemovalDelay(i) and InitialCPBRemovalOffset(i) also need to be changed. Therefore, the buffering period SEI messages need to be replaced. Due to this behavior, additional image timing SEI messages and buffering period SEI messages are typically included in so-called nesting SEI messages. When middleware performs sublayer bitstream extraction (e.g., dropping every other frame to obtain a 30Hz bitstream from a 60Hz bitstream), the original image timing SEI messages and buffering period SEI messages are replaced with the corresponding image timing SEI messages and buffering period SEI messages from the nesting SEI messages.

[0046] Bitrate Variations: HRD parameters such as initial_removal_delay and initial_removal_delay_offset, as well as bitrate and CPB_size, are typically provided for several values. The bitrate at which a given bitstream is fed into the CPB can vary, and therefore several values ​​are provided to yield a valid HRD model.

[0047] However, there are several aspects that have not been properly considered. • Only bitrates that result in CBR may exist. • If the appropriate bitrate is not known a priori, the bitstream cannot provide a valid HRD model (and its parameters).

[0048] As outlined above, splicing nondiscardable points is a difficult task at present, and therefore the embodiments described below aim to precisely achieve the possibility of splicing even nondiscardable points. The first simple modification is to correct the formula so that it does not always use a nondiscardable image as an anchor, but rather uses the last image received after splicing. That is, Removal(previousNonDiscardable)+SplicingDelta

[0049] Here, SplicingDelta is AuCPBRemovalDelta and InitialCPBRevovalDelay(i)+t af 2-t rm Set to the maximum value of 2. from, -Removal(previousNonDiscardable)+AuCPBRemovalDelta

[0050] -Removal(lastPictureBeforeSplicing)+InitialCPBRevovalDelay(i)+t af 2-t rm 2. Here, Removal(lastPictureBeforeSplicing) is actually t rm 2 Change to the maximum value among them. (In the example above)

[0051] Still, as seen in the formula, the splicer takes the previous image as the last image before splicing, for example t rm If it is decided to consider it as 1, the maximum value of both cases will always be Removal(previousNonDiscardable)+AuCPBRemovalDelta. This is because the splicer changes the value of AuCPBRemovalDelta accordingly, (t af 2-t rm Unless the difference in (2) is ensured to allow for seamless switching, it will prevent seamless splicing in any AU.

[0052] It is possible to change the value of AuCPBRemovalDelta to the buffering period SEI. However, (t af X and t rm Tracking X) is not that easy, so the splicer is (t af X and t rm Tracking the value of X) is not easy.

[0053] Since seamless splicing at different points is desirable, some kind of signaling to indicate to the splicer whether a given image allows this would be desirable. This would be done on the encoder side (t af X and t rm Check the X) value and the difference (t rm Xt af This can be achieved by ensuring that X) does not become smaller than a given value. Signaling can be performed, for example, by image timing SEI associated with the image that can be used as the last one before the splicing point. An example is shown in Figure 26. Alternatively, even if an image is nondiscardable by convention, subsequent discardable images will satisfy the requirements. An example can be seen in Figure 27.

[0054] (t rm Xt afThe requirement for the minimum value that X) must satisfy is related to the initial_removal_delay of the spliced ​​bitstream. The buffering period SEI should have some indication of what value the image can be used for seamless splicing. An example can be seen in Figure 28. Therefore, as a result of considering the possible modifications outlined above, we evaluated the following embodiments.

[0055] In the signaling examples in Figure 25, and in Figures 26, 27, and 28, it can be seen that the video data stream may comprise a sequence of image portions 23a, such as access units, and each image portion may have an image 10a of the encoded video 12a. Figure 25 depicts two data streams 14a and 14b, showing the splicing of streams 14a and 14b at splicing point 92 such that the trailing end of the portion of stream 14a that precedes splicing point 92, i.e., its left, abuts the leading end of the portion of stream 14b that follows splicing point 92, i.e., its right. The result of splicing performed by a splicing device such as a server, video conferencing system, or client-server streaming system, i.e., the spliced ​​data stream, is shown in 14c.

[0056] Image portions may comprise a first type of image portion, which are shown by cross-hatching, or simply hatching in some figures, and are referred to above as mark images. An example of this is an AU that has or comprises a buffering period SEI. These first type of image portions function as CPB removal time references, i.e., these CPB removals are used as reference anchorTimes for other timings given as time offsets such as AuCPBRemovalDelay.

[0057] The image portion may further include a second type of image portion, which is shown without hatching in some figures, such as images without a buffering period SEI.

[0058] Data streams 14a and 14b are further provided with first timing information 95, which compares AuCPBRemovalDelay or au_cpb_removal_delay_minus1 in each image portion and notifies the first CPB removal time for each image portion. The first CPB removal time measures the elapsed time since the removal of the preceding first type of image portion, i.e., the one preceding in the encoding or bitstream order as depicted in Figure 25, specifically the immediately preceding one.

[0059] Data streams 14a and 14b further include second timing information which compares initial_removal_delay which may exist within the image timing syntax of the buffering period SEI for each of the predetermined first type image portions, for example, before the splice point display discussed below. These predetermined first type image portions are simply shown hatched and are shown cross-hatched in Figure 25 to distinguish them from other first type image portions which are specific first type image portions, such as AUs with a buffering period SEI that are shown as RAP, for example, or in other words, AUs with a buffering period SEI that are shown as RAP. The second timing information notifies of the second CPB removal time for each predetermined first type image portion when decoding of the video data stream from each predetermined first type image portion onward is resumed. The second CPB removal time measures the elapsed time from the arrival of the CPB of the first bit of each predetermined first type image portion, i.e., from the arrival of the decoder's CPB.

[0060] At least one of the data streams, in Figure 25, is 14a, i.e., added to its splicing point 92, which should be split or separated to obtain another stream, here 14b, can further be provided with splice point indicators 94, 94', for example spliceable_flag or following_pic_spliceable_flag, which can function as the last image portion before splice point 92 having another video data stream 14b, i.e., suitable for enabling splicing, between its final CPB arrival and CPB removal as shown above. rm #-t af This indicates areas of the image where the time difference, such as #, exceeds a predetermined threshold, for example, areas without RAP. It should be noted that thresholds can be calculated, for example, as follows:

[0061] The removal time is assumed to be equidistant and equal to 1 / frame rate. The desired removal time for newly spliced ​​AUs, such as AU23b' in Figure 25, is t rm It should be #+1 / framerate, t rm # represents the removal time of the preceding AU or image portion, which is indicated by 23'a.

[0062] t rm #+1 / frame rate is t af #+initial_removal_delay must be greater than or equal to t af # represents the final arrival time of the CPB for each AU23b'. Therefore, the "predetermined threshold" is initial_removal_delay-1 / frame rate.

[0063] And in a variation of max_val_initial_removal_delay_for_seamless_splicing, namely in a variation where this threshold is explicitly shown in at least one of the data streams, such as another stream 14b being added to data stream 14a, t rm#+1 / frame rate is t af #+max_val_initial_removal_delay_for_seamless_splicing is greater than or equal to max_val_initial_removal_delay_for_seamless_splicing. Then the "specified threshold" is max_val_initial_removal_delay_for_seamless_splicing-1 / frame rate.

[0064] In other words, the video data stream 14a may also have a display 99 of the maximum second CPB removal time value, which is the time difference t between its final CPB arrival and its CPB removal, as long as the second CPB removal time of the first type image portion 23b' at the beginning of another video data stream 14b, which is joined with the video data stream 14a at the splice point 92, is less than the maximum second CPB removal time value. rm #-t af This demonstrates that concatenating any portion of the image indicated by the splice point display that exceeds a predetermined threshold with another video data stream 14b results in seamless splicing.

[0065] Note that the example buffering period SEI shown above includes display 99, concatenation flag 95, and third CPB removal time 98. This shows an example where both streams 14a and 14b carry the same type of informational data. In another example, this may be different.

[0066] Splice point indicators 94 and 94' show the time difference t between the arrival of the final CPB and the removal of the CPB for each image portion. rm #-t af The system may further include a flag 94 present in each image portion or each set of image portions, for example, without RAP, indicating whether # exceeds a predetermined threshold.

[0067] Alternatively or additionally, the splice point indicators 94, 94' may further provide each image portion or each set of image portions with a flag 94' present in those image portions that are not RAPs, for example, the time difference t between the arrival of its last CPB and the removal of its CPB from each image portion to the next first type of image portion. rm #-t af # indicates whether or not it exceeds a predetermined threshold.

[0068] The sequence of image segments may further have images of video 12a encoded using time interpretation 90 such as motion compensation prediction, such that the image segments comprise non-discardable image segments shown without underlining in some figures such as Figure 25, and discardable image segments shown with underlining.

[0069] These non-discardable image portions can be decrypted using a standalone method, while discardable image portions, although not required to decrypt the non-discardable portions, may require them because they are decryptable in themselves. Note that if the discardable image portions are not decrypted, the video can be decrypted without problems in the future.

[0070] In other words, discardable image portions do not function as reference images for time interval prediction, for example, while non-discardable images can function as reference images for time interval prediction. A data stream encoded using hierarchical time scalability represents an example of such image portions, where the image portion of the lowest (base) time layer may be non-discardable, while others may be discardable. Another example is an open GOP encoded data stream where the RASL image may be non-discardable.

[0071] The video data stream 14c has a time difference t between its final CPB arrival and its CPB removal. rm #-t afA predetermined image portion 23a', indicated by the splice point indicator 94;94' that # exceeds a predetermined threshold, is followed by a first predetermined first type image portion 23b' originating from the splice-in video data stream 14b, which is spliced ​​at splice point 92 so as to be in contact with it.

[0072] The video data stream 14c has a concatenation flag 96 and a third timing information 98 in the first predetermined first type image portion and compares AuCPBRemovalDelta or au_cp_removal_delay_delta_minus1. The concatenation flag 96 and the third timing information 98 may have already been present in the splice-in data stream 14b before splicing, but the splicing device may reset flag 96 from a state indicating non-splicing to a state indicating splicing, and / or set the third timing information 98. That is, the concatenation flag 96 is set to a first state indicating that the video data stream is spliced ​​in the first predetermined first type image portion, and the third timing information indicates a third CPB removal time for the first predetermined first type image portion 23b', which helps determine the CPB removal time when splicing in the first predetermined first type image portion 23b'. The third CPB removal time measures the elapsed time since the most recent CPB removal of an image portion that cannot be discarded. For example, this is since the removal of the most recent non-discardable image portion of a video data stream encoded in a spliced ​​data stream 14c.

[0073] Optionally, each of a predetermined first type of image portion, shown in hatching in some figures, comprises a concatenation flag 96 and a third timing information 98, the concatenation flag being set to a first or second state, the second state indicating that the video data stream is not spliced ​​in each of the predetermined first type of image portions.

[0074] The video encoder according to this embodiment can encode video into a video data stream in the following manner. For example, suppose such an encoder is an encoder data stream 14a. It is also possible to configure it to encode a data stream 14b which can be interpreted similarly, as described above. The encoder performs encoding such that the video data stream 14a comprises a sequence of image portions 23a, each image portion having an image 10a of the encoded video 12a. The video encoder can then classify the image portions into a first type of image portion that functions as a CPB removal time criterion and a second type of image portion as described above. The encoder provides the video data stream with first and second timing information and can check for each of a given image portion (for example, these may include discardable image portions) whether the time difference between its final CPB arrival and its CPB removal exceeds a predetermined threshold. If so, each given image portion may function as the last image portion before a splice point 92 with another video data stream 14b. The video encoder can further provide splice point indicators 94, 94' to the video data stream 14a.

[0075] The video encoder may, for each predetermined image portion, further use a predetermined threshold value determined based on the second CPB removal time of the most recent first type of predetermined image preceding each predetermined image portion, for example, one simply hatched in some of the figures preceding 23a'.

[0076] The video encoder may also use a value determined based on the maximum second CPB removal time value as a predetermined threshold for each predetermined image portion, and write an indication 99 of the maximum second CPB removal time value to each predetermined image portion, for example, the most recent predetermined image of the first type preceding the one simply hatched in the figure preceding 23a'.

[0077] The following describes how the aforementioned spliced ​​data stream may be processed within the decoder, by the HRD 46, or in other words, by the device 46 for managing the CPB 48 of the video decoder 44, the video decoder 44 decodes the video data stream 14c so that it is buffered by the CPB, i.e., receives the image portion via the CPB at the removal time, and the image portion reaches the CPB completely at the aforementioned final arrival time. The device can check whether the concatenation flag 96 of the first predetermined first type image portion 23b' indicates that splicing has been performed on the first predetermined first type image portion. Next, the device can determine the time to remove the first predetermined first type image portion 23b' from the CPB.

[0078] The time to be removed can be determined based on the first timing information 95 of the first predetermined first type image portion 23b', if the concatenation flag of the first predetermined first type image portion indicates that no splice-in was performed in the first predetermined first type image portion.

[0079] The time to be removed can be determined based on a second timing information of the first predetermined image portion 23b', such as initial_removal_delay, and a third timing information 98 of the first predetermined image portion, if the concatenation flag 96 of the first predetermined image portion indicates that a splice-in has been performed in the first predetermined image portion.

[0080] Optionally, the device performs CPB removal on the most recent non-discardable image portion plus a third CPB removal time for the first predetermined first type image portion, and CPB removal on the predetermined image portion 23a' that subsequently contacts the first predetermined first type image portion plus a second CPB time minus the time difference t between the final CPB arrival and CPB removal of the predetermined image portion 23a'. rm #-t afThis can be done by determining the maximum value between #. This time is then used to remove the first predetermined first type image portion 23b' from the CPB.

[0081] In this embodiment, a device for splicing together two video data streams, 14a and 14b, to generate 14c, is described, as shown in Figure 25. Each of these streams comprises a sequence of image portions 23a and 23b, as described above, with each image portion having images 12a and 23b of the encoded video 12a and 23b. The image portion also includes the first and second types of image portions, as well as the first and second types of timing information, as described above.

[0082] Next, the second video data stream includes a concatenation flag 96 and third timing information 98 for the first predetermined first type image portion. The concatenation flag 96 is set to a second state indicating the second video data stream, where the first predetermined first type image portion continues the second video data stream, and the third timing information indicates a third CPB removal time for the first predetermined first type image portion, which helps determine the CPB removal time when splicing the second video data stream into the first predetermined first type image portion, where the third CPB removal time measures the elapsed time since the most recent CPB removal of the non-discardable image portion.

[0083] The device can then check whether the splice point indicator 94;94' of the first video data stream indicates, for example, that for a given image portion without a RAP, the time difference between the arrival of the CPB and the removal of the CPB exceeds a predetermined threshold, so that the first video data stream can function as the last image portion before it is spliced ​​with the second video data stream.

[0084] For example, in one modified example, a predetermined threshold is determined based on the initial_removal_delay of the buffering SEI of the first stream 14a, or based on max_val_initial_removal_delay_for_seamless_splicing.

[0085] If the result of this check is affirmative, i.e., "yes", the device can set the concatenation flag 96 of the first predetermined first type image portion of the second video data stream to a first state indicating that the second video data stream has been spliced ​​in with the first predetermined first type image portion, and can concatenate the first and second data streams of the predetermined image portion 26a' and the first predetermined first type image portion 26b', respectively, to obtain a spliced ​​video data stream.

[0086] Furthermore, if so, the device can rewrite the third timing information 98 to measure the elapsed time since the most recent CPB removal of the non-discardable image portion of the first video data stream in the spliced ​​video data stream instead of the second video data stream.

[0087] Furthermore, if so, the device can further check whether the elapsed time since the most recent CPB removal of the non-discardable image portion preceding the first predetermined first type of image portion has changed when considering the spliced ​​video data stream in comparison to the second video data stream, and if so, it can rewrite the third timing information 98 to measure the elapsed time since the most recent CPB removal of the non-discardable image portion of the first video data stream in the spliced ​​video data stream. Generally, the device may also perform the following checks and, if both checks are positive, then perform the setup and coupling.

[0088] First, is the second CPB removal time equal to, or determined from, the second CPB removal time of the first predetermined image portion of the first type in the first video data stream 14a immediately preceding each predetermined image portion greater than the second CPB removal time of the first predetermined image portion of the first type 23b' in the second video data stream? In some figures, a predetermined image portion of the first type immediately preceding the first stream 14a is simply shown by hatching.

[0089] The second CPB removal time for the most recent first type of predetermined image portion may also include an additional CPB supply delay time for the most recent first type of predetermined image portion. This measures the elapsed time during which the arrival of the CPB for the first bit of the most recent first type of predetermined image portion is delayed. Each designated image portion is, for example, simply hatched in some of the figures, preceding 23a'.

[0090] The second check is whether the maximum second CPB removal time value 99 shown in the most recent predetermined image portion of the first type is greater than the second CPB removal time of the first predetermined image portion 23b' of the second video data stream.

[0091] For example, in one variation, the given threshold could be only the initial_removal_delay of the buffering SEI of the first stream. Alternatively, the threshold could be initial_removal_delay + initial_removal_delay_offset, i.e., the CPB supply delay time. Furthermore, as described in the second option, the maximum value max_val_initial_removal_delay_for_seamless_splicing can be sent.

[0092] For example, AuCPBRemovalDelta only needs to be rewritten if its original value, which is the distance to the non-discardable image in the original second bitstream 14b, is different from the distance in the spliced ​​bitstream 14c.

[0093] For example, both bitstreams have the same frame rate, and the following bitstreams, shown in decoding order, can be assumed to be concatenated, i.e., spliced. Second bitstream: RAP1, B0, B1 (non-discardable), B2, RAP2 (splicing_point) First bitstream: rap1, b0 (non-discardable), b1, b2, b3, b4 Splice them together: rap1, b0 (cannot be discarded), b1, b2, b3, RAP2 (splicing_point)

[0094] Originally, the RAP2 buffering period SEI includes concatenation_flag set to 0 and AuCPBRemovalDelta equal to 2 / framerate.

[0095] In a spliced ​​bitstream, concatenation_flag is set to 1, and AuCPBRemovalDelta is equal to 4 / framerate.

[0096] However, if the spliced ​​bitstream is rap1, b0 (non-discardable), b1, RAP2 (splicing_point), concatenation_flag is set to 1 and AuCPBRemovalDelta is equal to 2 / framerate. Therefore, in the first case, it is necessary to rewrite AuCPBRemovalDelta, but not in the second case.

[0097] For spliced ​​streams with an initial removal delay smaller than max_val_initial_removal_delay_for_seamless_splicing, seamless splicing can be achieved if the flag in the image timing SEI indicates so. If the initial removal delay of the spliced ​​stream is higher, it is obviously not feasible to know whether it is splicable or not.

[0098] While the above embodiments concerned the issue of enabling splicing at points that cannot be discarded, the following description addresses the issue of how available removal time can be made available for removable image portions such as DRAP. In particular, subsequent embodiments relate to providing timing information (removal time) when several AUs are dropped, such as in the case of RASL images with an open GOP structure when random access is performed or when a splicing scan is performed. In other words, whether the removal time is applicable to dependent random access points, DRAP, and images.

[0099] The idea here is to modify the initial_removal time of the RAP buffering period SEI. Different cases are discussed below, and examples of AU removal times are shown.

[0100] Figure 1 illustrates the concept of the invention described herein. A first CPB removal time t is applied to the decoder side when video 12 is encoded and the full version 15 of the video data stream 14 is buffered. rm (1) The first timing information 16 relating to the first CPB removal time t is applied to the decoder side to buffer a reduced version 20 of the video data stream 14, which is different from the video data stream due to the removal 22 of a portion 24 of the video data stream from the full version of the video data stream. rm (2) A video data stream having second timing information 18 relating to is depicted.

[0101] Optionally, the second timing information 18 is the second CPB removal time t rm (2) To generate the first CPB removal time t rm (1) The timing correction information regarding the method of correcting the second CPB removal time t rm (2) This defines the timing correction information. An example of such timing correction information is, for example, init_removal_delay_correction_offset.

[0102] Furthermore, the first timing information 16, for each image, is incrementally calculated relative to the preceding mark image in the decoding order 26, determining the first CPB removal time t for the image. rm (1) It can signal this.

[0103] The increment can be symbolically signaled by AuCPBRemovalDelay or au_cpb_removal_delay_minus1. Mark images may be, for example, clean random access point (CRA) images and include buffering period SEI. These are typically RAP images, but may be other required images, such as time layer 0 images.

[0104] In other words, each image portion 23 of the video data stream comprises an increment that measures the delay of its removal from the CPB48 to the preceding RAP image along the encoding order 26. The image portion 23 is also called an access unit AU, as in the case of HEVC. Optionally, a marked image is marked by a buffering period message in the video data stream within the image portion associated with the marked image.

[0105] Furthermore, the device 46 for managing the CPB 48 of the video decoder 44 that decodes the video data stream can manage the CPB according to first timing information 16 when that part is composed of the video data stream, and according to second timing information 18 when part 24 is removed. Figure 5 shows the complete bitstream. The initial removal time for clean random access point CRA is 8.

[0106] In Figure 7, the RASL image has been removed. The initial removal time for CRA is 7. The removal time for subsequent images has a delta offset of 4 compared to the previous case.

[0107] Figure 6 shows the removed RASL image and the TRAIL (normal trailing) image up to TRAIL4, which is DRAP (excluding this part). The initial removal of CRA is 11 at this point so that the CPB level is the same as if the bitstream had been present from the beginning.

[0108] One option would be to add alternative timings for various possible frame drops, such as one for removing RASL images, one for removing the first DRAP, one for removing the second DRAP, and so on.

[0109] However, this requires increasing the size of the buffering period SEI and indicating to the receiver which case applies, i.e., whether RASL is removed, all AUs up to the first DRAP, or all AUs up to the second DRAP. This can be done in the SEI.

[0110] However, all DRAP encoding must also be performed so that the buffered SEI can be written properly, which introduces additional delay on the encoder / transmitter side.

[0111] Alternatively, a single value may be shown in the buffered SEI message, which can be corrected by a subsequent SEI, such as the next unremoved AU image timing SEI message. Then, when the AU removal is performed, it is only necessary to set the flag aus_since_rap_removed_flag. An example of this can be seen in Figure 8.

[0112] Here, in combination with the figure above, the second timing information 18 is the second CPB removal time t for the first image portion 28'. rm (2) To generate a first CPB removal time t, which is indicated by first timing information for the first image portion 28' of the video data stream relating to a first predetermined image CRA and precedes the portion 24' of the video data stream. rm (1) By the first timing correction information 19 regarding the method of correcting, or by this first CPB removal time t rm (1) The first timing substitution information for replacing the second CPB removal time t rm (2) It can be seen that it is possible to define it.

[0113] The first timing correction information 19 can be signaled, for example, by init_removal_delay_correction_offset.

[0114] Furthermore, the first timing correction information 19 or the first timing substitution information is signaled in the video data stream within a second image portion 30' of the video data stream, following a portion 24' of the video data stream, and relating to a second predetermined image TRAIL 1.

[0115] It should be noted that SEI messages can be processed as buffered in the CPB as well, and therefore may need to be taken into consideration when determining the first and second timing information. The first and second timing information may be calculated and transmitted once for SEI messages that contain the first and second timing information, and once for dropped SEI messages. Furthermore, optionally, the second image portion may include a signal 32 indicating whether or not that portion has been removed.

[0116] The first timing correction information 19 or the first timing replacement information can be signaled in the video data stream regardless of whether the signaling 32 indicates that the portion has been removed or not.

[0117] The first predetermined image may be a mark image, such as an intra-random access point (IRAP) or dependent random access point (DRAP) image, i.e., one that functions as a timing reference, as detailed above. Furthermore, the second predetermined image may be a TRAIL or DRAP image.

[0118] Furthermore, the second timing information 18 determines the second CPB removal time t for one or more sets of third image portions 30'. rm (2) To generate a third predetermined image, a first CPB removal time t is determined by first timing information for a set of one or more third image portions 31' of a video data stream comprising a second image portion 28', in relation to a third predetermined image, according to portion 24' of the video data stream. rm (1) The second timing correction information 21 regarding the method of correcting the second CPB removal time t rm (2)The second timing correction information 21 can be defined and signaled in the video data stream within the second image portion 30'. The second timing correction information 21 can be signaled, for example, by cpb_removal_delay_offset.

[0119] The second image portion may also include a signal 32 indicating whether or not that portion has been removed, and the second timing correction information 21 can be signaled in the video data stream regardless of the signal 32 indicating whether or not that portion has been removed.

[0120] One option is to extend one or more sets of third image segments 31' of the video data stream up to a fourth image segment relating to a mark image. Here, the mark image can serve as the starting point for the timing criterion. Finally, a marked image can be marked by a buffering period message in the video data stream within the image portion associated with the marked image.

[0121] In the case shown for DRAP, note that the initial removal of the CRA is 11 instead of 8, as is the case when the full bitstream is considered and therefore the CPB fill level is higher than in the original case. This can lead to potential problems and buffer overflows.

[0122] Another option would be to allow the removal times from the CPB (in the case of DRAP) to not be equidistant for all frames. More specifically, RAP and DRAP can have non-equal distances, and so can all other AUs.

[0123] In that case, the buffering period SEI message for the RAP can have two timings: one for the entire bitstream, or when the RAP is removed, and another for the DRAP. In the second case, since the RAP is not output, the removal time for the RAP can be such that the final arrival time of the RAP is equal to that removal time. Next, the DRAP AU has a second removal time to accumulate the buffer to the desired level. In such a case, the final arrival time of the RAP is the earliest arrival time of the DRAP in both the CBR and VBR cases. The described solution is shown in Figure 9. Exemplary signaling syntax is shown in Figures 10 and 11.

[0124] In one embodiment, the second timing information 18 is modified by the third timing modification information 33 and / or the fourth timing modification information 35, or by the third timing replacement information 33 or the fourth timing replacement information 35, with respect to the modification method, for the second CPB removal time t rm (2) It is possible to define this.

[0125] The third timing correction / replacement information 33 is the second CPB removal time t for the first image portion 28'. rm (2) To generate the first CPB removal time t is determined by the first timing information for the first image portion 28' of the video data stream that precedes portion 24'' of the video data stream in relation to the first predetermined image CRA. rm (1) Regarding how to modify or replace it.

[0126] The third timing correction / replacement information 33 can be signaled, for example, by drap_operation_initial_removal_delay. It should also be noted that cpb_drap_operation_removal_offset relates to the arrival time, i.e., the time when a particular image portion, i.e., a marked image marked by a buffering SEI message, enters or is supplied to the CPB. Supply may be the responsibility of an apparatus other than the apparatus for managing the CPB as described in this application. Adherence to the arrival time may depend on the transmitting intermediate network node.

[0127] The fourth timing correction / replacement information 35 is the second CPB removal time t for one or more sets of third image portions 30''. rm (2) To generate a first CPB removal time t, which is indicated by first timing information for a set of one or more third image portions 31'' of the video data stream following a portion 24'' of the video data stream in relation to a third predetermined image. rm (1) Regarding how to modify or replace it.

[0128] The fourth timing correction / replacement information 35 can be signaled, for example, by cpb_drap_operation_removal_delay_offset. In general, the device 46 for managing the CPB48 of the video decoder can also perform the modifications or replacements detailed above.

[0129] Optionally, the third timing correction / substitution information 33 can be signaled in the video data stream within the first image portion 28' of the video data stream, and the fourth timing correction / substitution information 35 can be signaled in the video data stream within the second image portion 30'' of the video data stream that precedes the encoding order 26 of one or more sets of third image portions.

[0130] Furthermore, the first image portion 28' may include a signaling 36 indicating whether the distance between the second CPB removal times of the first and second image portions can deviate from the distance between the second CPB removal times of a consecutive pair of the second and third image portions, and the third timing correction information 33 or third timing substitution information 33 may be signaled in the video data stream within the first image portion 28' of the video data stream, subject to the signaling 36 indicating whether the distance between the second CPB removal times of the first and second image portions can deviate from the distance between the second CPB removal times of a consecutive pair of the second and third image portions.

[0131] The video encoder checks whether the second predetermined image is a DRAP that references the first predetermined image which is a RAP, sets the signaling 36 to indicate whether the second predetermined image is a DRAP that references the first predetermined image which is a RAP, and if the second predetermined image is a DRAP that references the first predetermined image which is a RAP, it can encode the third timing correction / substitution information 33 in the video data stream within the first image portion 28' of the video data stream.

[0132] Note that normally, au_cpb_removal_delay_minus1 in the pic_timing SEI message increments consistently between consecutive image segments, i.e., increments consistently over the same distance. However, if the first image segment 28' is either removed from the encoded image or not presented or output, there is an image that is actually output, so it is acceptable if the normal temporal distance is interrupted up to DRAP and is valid thereafter. In other words, in the example, au_cpb_removal_delay_minus1 in the pic_timing SEI message increments consistently over the same distance. This is usually the case, but it is not prohibited if it is not. The example shows this is the case of a full bitstream where nothing is originally removed. However, when removing everything between RAP and DRAP, this is no longer the case, but this is not a problem as RAP is not output or displayed at all when using the DRAP function. Furthermore, the second image portion 30'' may include a signaling 38 indicating whether or not portion 24' has been removed.

[0133] The apparatus 46 for managing the CPB 48 of the video decoder 44 according to the present invention can postpone the removal of the first image portion of the video data stream after inspection of the signaling 32;38.

[0134] Alternatively, a network node 42 for transferring a video data stream according to the present invention may configure signaling 32;36 to indicate that the portion is to be removed from the video data stream. The network node 42 can also remove, or drop, portions 24 of the video data stream from the video data stream.

[0135] As shown in Figure 5, optionally, the second timing information is a first CPB fill level 39a';39a'' on the decoder side that appears on the decoder side after buffering the first image portion of the video data stream preceding the portion of the video data stream, and the second CPB removal time can be defined such that the first CPB fill level 39a';39a'' following the portion of the video data stream, when the second image portion is debuffered according to the first CPB removal time, is equal to the second CPB fill level 39b';39b'' on the decoder side that appears on the decoder side after buffering the first and second image portions of the video data stream when the second image portion is debuffered according to the second CPB buffer time having an unbuffered portion. The video encoder can set first and second timing information so that the video data stream conforms to the above.

[0136] More generally, the second timing information can define a second CPB removal time for either a portion relating to a sequence of one or more RASL images, or a portion relating to a sequence of images preceding a particular DRAP. Furthermore, the video data stream may include signals 32 and 35 indicating whether or not that portion has been removed.

[0137] The DRAP offset used for random access is used to calculate the removal time for subsequent AUs. Clearly, instead of displaying the offset in the image timing SEI message, it is required that the DRAP include a buffering period SEI message, and that these offsets be displayed in the buffering period SEI message.

[0138] The embodiments described below address the problem of how to represent temporal scalability and information regarding image buffer timing in image timing SEI and buffering period SEI messages in such an encoding environment. Therefore, embodiments are described in which temporal scalability in image timing SEI and buffering period SEI messages is referenced by alternative information within the image. First, embodiments are presented in which temporal scalability in image timing SEI and buffering period SEI messages is referenced by alternative information within the image, i.e., embodiments that utilize offsets for timing in image timing SEI messages.

[0139] Figures 12 to 14 show some examples of removal time values ​​(i.e., time scalability) when different frame rates are provided on the same bitstream.

[0140] The table in Figure 17 shows that the deltaTime for frames labeled "blue" is 0, 1 / 60, 3 / 120, 2 / 60, and 6 / 120. For frames labeled "red," it is 3 / 120 and 6 / 120. The blue frames are referred to by these rows in the table, which include values ​​for 30, 60, and 120 fps. The red frames are referred to by these rows in the table, which include values ​​for 60 and 120 fps only, and not for 30 fps.

[0141] In the case of the open GOP, where the highest frame rate is shown in Figure 16, the following applies, also shown in Figure 15. In the table in Figure 15, we can see that the deltaTime for frames labeled "blue" is 0, 1 / 60, and 3 / 120. For "red frames," it is 1 / 120 and 2 / 120. Here again, the blue frames are referred to by these rows in the table, which include the values ​​for 30, 60, and 120 fps. The red frames are referred to by these rows in the table, which include only the values ​​for 60 and 120 fps, and not the value for 30 fps.

[0142] To summarize, at the deltaTime of the removal time of different frame rates, there are some patterns that are repeated due to the GOP structure of the encoded bitstream.

[0143] Another aspect to consider is the scene cut because the described pattern is interrupted. Different positions for scene cuts result in different values. For simplicity, only one position is shown in FIG. 18. As can be seen from the table in FIG. 19, clearly, the GOP affected by the scene cut does not follow the pattern.

[0144] FIG. 20 shows an exemplary signaling structure that refers to an image timing SEI message containing different values for different frame rates (i.e., the target TemporalId).

[0145] In the embodiment shown in FIG. 14, the video data stream is encoded in a temporally scalable manner, using temporal inter prediction in units of groups of images 50, such that the video image 12 is grouped into images of different hierarchical levels 50, where for each hierarchical level, all images of that hierarchical level simply depend on the images of that level or lower hierarchical levels. Only the start part of one GOP is shown in the figure. i The part 24 is also composed of images belonging to the top - most hierarchical level 503 or one or more hierarchical levels 502 excluding the bottom - most hierarchical level 501 immediately below the top - most hierarchical level 503 in the hierarchical order 52 and the top - most hierarchical level 503.

[0146] Next, the first timing information 16 signals, for each image, the first CPB removal time t for the image by the increment with respect to the preceding marked image 53 of the current group in the decoding order. This order is indicated by the order of the fractional values used when the images are labeled.

[0147] rm (1)

[0148] ​​Here too, the increment can be signaled, for example, by AuCPBRemovalDelay or au_cpb_removal_delay_minus1. The current GOP is an image with respect to the preceding image of the GOP of which each image is a part. Furthermore, the mark image can be marked by a buffering period message of the video data stream within the image portion related to the mark image.

[0149] In one embodiment, the second timing information 18 is signaled in the video data stream in each of the fifth image portions 23 of the video data stream for the fifth image 54 associated with one or more hierarchy levels other than the top hierarchy level, by the fifth timing correction information 19 or the fifth timing replacement information 19 for replacement, for defining the second CPB removal time t rm (2) can be defined.

[0150] The fifth timing correction information 19 can be signaled, for example, by au_cpb_removal_pattern_offset, and the fifth timing replacement information 19 can be signaled, for example, by au_cpb_removal_delay_minus1[i], where i is not equal to the current hierarchy level to which the image for which pic_timing is transmitted belongs.

[0151] The fifth timing correction / replacement information 19 relates to a method of modifying or replacing the first CPB removal time t rm (2) indicated by the first timing information for each of the fifth image portions 23, for generating the second CPB removal time t rm (1) for the fifth image portion 23.

[0152] The first timing information can be signaled, for example, by au_cpb_removal_delay_minus1[i] or u_cpb_removal_delay_minus1, where i is equal to the current hierarchy level to which the image being sent belongs.

[0153] If part 24 has a top-level hierarchical level up to each hierarchical level above each hierarchical level, the fifth timing correction / replacement information 19 is signaled for each hierarchical level between the top-level and bottom-level hierarchical levels, i.e., between 501 and 502 in the illustrated example, and for each hierarchical level between the top-level and bottom-level hierarchical levels above each hierarchical level, i.e., between 503 of 502 and 503 and 502 of 501 in the illustrated example. As an example, the numerical value is indicated by num_sub_layer_cpb_removal_delays_minus1 when buffering_period_pattern_flag is 1, and by num_sub_layer_cpb_removal_delays_minus1+1 when buffering_period_pattern_flag is zero.

[0154] Figures 21 and 22 show exemplary signaling structures in which, alternatively, patterns can be defined within each buffering period SEI and indicated within image timing SEI messages. As mentioned above, when patterns are not applied, for example, for scene cuts, values ​​can be signaled explicitly (or in the form of some offset).

[0155] According to this embodiment, the second timing information 18 is corrected by the fifth timing correction information 19, for example, au_cpb_removal_pattern_offset, to the second CPB removal time t rm (2)The fifth timing correction information 19 indicates a method for correcting the first CPB removal time by an offset value, and the video data stream includes data 60 indicating a list of representative offset values ​​assumed by the offset values ​​indicated by the fifth correction information for correcting the first CPB removal time in one or more groups of images. The fifth timing correction information 19 then signals the offset value by a pointer that points to within the list of representative offset values.

[0156] More generally, in the embodiments described above, the video data stream is reduced by only portion 24, i.e., resulting in a reduced version 20, and the video data stream comprises first and second timing information 18. In other words, the second timing information is not nested so as to replace the corresponding first timing information without nesting if a particular hierarchy level is dropped in some intermediate network device. Rather, the first and second timing information remains unchanged on the decoder side, and the precise timing information is selected for use by the CPB, which processes in response to signaling transmitted in the video data stream, and is configured to tell the intermediate network device which portion was removed, or more precisely, which previous top hierarchy level was removed / dropped.

[0157] Another example of an embodiment that references alternative information within an image for temporal scalability in image timing SEI messages and buffering period SEI messages is the delta in the buffering period SEI message.

[0158] As can be seen in Figure 23, the different values ​​of vcl_initial_cpb_removal_delay[i] and vcl_initial_cpb_removal_offset[i] are also clearly insufficient for all possible frame rates and should therefore be indicated within the buffering period SEI message.

[0159] Another embodiment refers to considerations of multiple bitrates. As mentioned above, the current method for supporting different transmission rates or CPB supply rates is to specify several values ​​for the bitrate, CPBsize, and their respective initial rejection time and offset.

[0160] The problem with this current solution is that all potential CPB supply bitrates must be known in advance, and rate control in the encoder must take them all into account to ensure that the provided information is valid.

[0161] While this allows for a degree of flexibility, where the mentioned parameters can be modified in many ways, in some real-world scenarios, taking into account many potential CPB supply rates, which may be desirable from the transmitter side but difficult to implement on the encoder side, can be useful. In such circumstances, a "fairly" limited rate control where some parameterization of the HRD model is performed may be helpful, meaning that two extremes, the maximum and minimum bitrates, are considered, and everything in between can be linearly interpolated. Such a solution is more limited than a flexible solution with respect to the HRD modeling, but it allows the transmitter to use any rate between the maximum and minimum bitrates, and still ensures there is no underflow or overflow, with CPBsize and initial removal time that can be calculated based on the transmitted values ​​for the maximum and minimum. hrd_parameters is divided into three parts. · cbr_parameters_present_flag ·vbr_min_parameter_present_flag ·vbr_max_parameter_present_flag

[0162] It is possible to calculate any other bitrate between the two given values ​​for vbr_min_parameter_present_flag and vbr_max_parameter_present_flag. Exemplary signaling is shown in Figures 24a, 29, and 30.

[0163] Similarly, the buffering period SEI message, if present in hrd_parameters as shown in Figures 24b and 31, also includes the CBR, minimum, and maximum values.

[0164] According to this, HRD timing information data, for example, a first HRD timing information data, a second HRD timing information data, or any of multiple HRD timing information data, and each HRD timing information data comprises one or more of the following: - Display of the CPB supply bitrate associated with each HRD timing information 104, - Displaying the required CPB size for each CPB supply bitrate associated with each HRD timing information 106,

[0165] -Display 108 of the CPB removal time for each marked image portion of the video data stream, measuring the elapsed time from the arrival of the CPB for the first bit of each marked image portion for each CPB supply bitrate associated with each HRD timing information - where the marked image portions are those having a buffering period SEI such as RAP, and these are shown by cross-hatching in another appropriate figure, while other images are shown by simple hatching in another appropriate figure, etc.

[0166] - Display of each CPB supply delay time for each marked image portion of the video data stream, measuring the elapsed time at which the arrival of the first bit of the CPB for each marked image portion is delayed for each CPB supply bitrate associated with each HRD timing information. If there are minimum and maximum values, any value between them will result in, for example, a linear interpolation between the minimum and maximum values.

[0167] Nevertheless, in some cases, greater flexibility in the HRD model, rather than limiting it to a linear interpolation model, would be desirable (for example, reducing initial_removal_delay instead of linearly and increasing the bitrate). Therefore, depending on the scenario, the encoder can choose to use either a mode with linear interpolation or one with multiple values. Exemplary signaling is shown in Figures 32, 33, and 34.

[0168] As shown in Figures 24, 29, 31, 32, and 34, a video stream may comprise first HRD timing information data 70a,b relating to the minimum CPB supply bitrate and second HRD timing information data 72a,b relating to the maximum CPB supply bitrate. The first HRD timing information data 70a,b and the second HRD timing information data 72a,b are then linearly interpolated between the first and second HRD timing information data to enable the determination of a third HRD timing information for the actual CPB supply bitrate. The actual CPB supply bitrate is, for example, the bitrate at which the video data stream is actually supplied to the CPB.

[0169] The video data stream may further include a flag 74, such as hrd_interpolation-mode_flag, indicating whether a first set of HRD timing information data 70a,b and a second set of HRD timing information data 72a,b exist, or whether multiple sets of HRD timing information data 76 exist for each of the required sets of CPB supply bitrates.

[0170] Next, the video encoder may decide to provide the video data stream with a first HRD timing information 70 and a second HRD timing information 72, or to provide the video data stream with a plurality of HRD timing information data 76 for each of the required sets of CPB supply bitrates.

[0171] The video encoder may optionally perform a feasibility check to determine whether linear interpolation between the first and second HRD timing information results in a situation where there is no CPB overflow or underflow at all CPB supply bitrates between the minimum and maximum CPB supply bitrates. The device 46 for managing the CPB 48 of the video decoder 44 can then manage the CPB using third HRD timing information.

[0172] Similarly, the device may examine a flag 74 in the video data stream, for example hrd_interpolation-mode_flag, and, depending on the flag, perform the derivation of first HRD timing information data 70a,b and second HRD timing information data 72a,b from the video data stream, or derive a plurality of HRD timing information data 76 from the video data stream for each of the required sets of CPB supply bitrates. Figure 38 shows an example of three operating points. The following linear interpolation formula is used for cases where interpolation is performed. -BR interpolated =BR min +(BR max -BR min )*alpha -IND interpolated =IND min +(IND max -IND min )*alpha(where IND=InitialRemovalDelay) Note that IND min (For the minimum bitrate) is IND maxPlease note that this is larger than (at the highest bitrate). Figure 37 is IND min Refer to the following abbreviations. RmD = Removal Delay Afinal = Final arrival time ArrivalEar = The earliest possible arrival time of the first bit of an image, due to the frame rate. HappyInit = Actual arrival time of the first bit of the image It can be seen that the maximum CPB size is 15. Figure 36 is IND max See [reference]. The required CPB size for this second case is 12,1125.

[0173] Figure 35 shows that for linearly calculated initial rejection delay and bitrate with alpha = 0.3, the required CPB size in this case is 13,87912. If linear interpolation is performed, the calculated CPB size corresponds to 12,97875. As you can see, these values ​​are different.

[0174] Therefore, if the encoder is to adhere to the constraint that the HRD parameters involved are derived linearly with full flexibility within the range of maximum and minimum values, the rate control in the encoder should take this into account when determining the image size so that the CPB size constraint during interpolation is satisfied.

[0175] Since this is not always possible, one option would be to signal that interpolation is possible only if the encoder is handling it, and otherwise signal all the actual values ​​used at the discrete operating point.

[0176] As mentioned above, the above concept can be employed by a video stream, a video encoder, a video decoder, a network node, a device for managing the encoded image buffer (CPB) of the video decoder, and a device for splicing video data streams together.

[0177] Similarly, the concept can be implemented by corresponding methods according to embodiments of the present invention. These methods are based on the same considerations as described above for the devices for managing video streams, video encoders, video decoders, network nodes, encoded image buffers (CPBs) of video decoders, and / or devices for splicing video data streams together. However, it should be noted that the methods can also be supplemented by any of the features, functions, and details described herein with respect to video streams, video encoders, video decoders, network nodes, devices for managing encoded image buffers (CPBs) of video decoders, and / or devices for splicing video data streams together. Furthermore, the methods can be supplemented individually or in combination by the features, functions, and details of the devices for managing video streams, video encoders, video decoders, network nodes, devices for managing encoded image buffers (CPBs) of video decoders, and / or devices for splicing video data streams together.

[0178] Finally, the concept can also be used to generate encoded data streams according to embodiments of the present invention. The data streams, individually or in combination, can also be supplemented by the features, functions, and details of the method, video encoder, video decoder, network node, apparatus for managing the encoded image buffer (CPB) of the video decoder, and / or apparatus for splicing video data streams together.

[0179] In conclusion, the embodiments described herein can be optionally supplemented by any of the important features or aspects described herein. However, it should be noted that the important features and aspects described herein can be used individually or in combination, and may be introduced into any of the embodiments described herein, individually or in combination.

[0180] While some embodiments have been described in the context of apparatus, it is clear that these embodiments also represent descriptions of corresponding methods, where the device or a part of it corresponds to a method step or a feature of a method step. Similarly, embodiments described in the context of a method step also represent descriptions of corresponding apparatus or a part of it, or items or features of the corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0181] Depending on the specific implementation requirements, the embodiments of the present invention may be implemented in hardware or software. These embodiments may be implemented using a digital storage medium such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, which stores electronically readable control signals and cooperates (or can cooperate) with a computer system programmable to perform each method. Thus, the digital storage medium may be computer-readable.

[0182] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein can be performed.

[0183] Generally, embodiments of the present invention can be implemented as a computer program product having program code, which operates to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier. Another embodiment includes a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0184] Therefore, in other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0185] Accordingly, further embodiments of the methods of the present invention include a computer program for performing one of the methods described herein, on which it is recorded, a data carrier (or digital storage medium or computer-readable medium). The data carrier, digital storage medium, or recording medium is typically tangible and / or non-temporary.

[0186] Accordingly, a further embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0187] Further embodiments include processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein. Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0188] Further embodiments of the present invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0189] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0190] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0191] The apparatus described herein, or any component of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0192] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0193] The methods described herein, or any part thereof, can be performed at least partially by hardware and / or software.

[0194] The embodiments described above are merely illustrative of the principles of the present invention. Modifications and variations of the arrangements and details described herein will be obvious to those skilled in the art. Therefore, it is intended that the invention is limited only by the scope of the immediate claims and not by the specific details presented herein.

Claims

1. A video encoder for encoding video into a video data stream such that the video data stream comprises a sequence of image portions, wherein each image portion has an image of the encoded video therein. In the first Buffering Period (BP) Supplemental Enhancement Information (SEI) message, the maximum value of the initial rejection delay is indicated, To indicate a first flag in an image timing SEI message corresponding to a first image portion of the video data stream, wherein the first flag equal to 1 indicates that if the first image portion is followed by a second image portion having a second BP SEI message in which the concatenation flag is equal to 1 and the initial removal delay is less than the maximum value of the initial removal delay, then a nominal coded image buffer (CPB) removal time based on the elapsed time since the removal of a previous non-discardable picture portion is applied. A video encoder configured to perform the following actions.

2. The video encoder according to claim 1, wherein the elapsed time since the removal of the previously non-discardable image portion is indicated in the second BP SEI message.

3. The video encoder according to claim 1, wherein the first image portion, the second image portion, and the previous non-discardable image portion are access units.

4. A method for encoding video into a video data stream such that the video data stream comprises a sequence of image portions, wherein each image portion has an image of the encoded video therein. In the first Buffering Period (BP) Supplemental Enhancement Information (SEI) message, the maximum value of the initial rejection delay is indicated, To indicate a first flag in an image timing SEI message corresponding to a first image portion of the video data stream, wherein the first flag equal to 1 indicates that if the first image portion is followed by a second image portion having a second BP SEI message in which the concatenation flag is equal to 1 and the initial removal delay is less than the maximum value of the initial removal delay, then a nominal coded image buffer (CPB) removal time based on the elapsed time since the removal of a previous non-discardable picture portion is applied. A method for providing this.

5. The method according to claim 4, wherein the elapsed time since the removal of the previously non-discardable image portion is indicated in the second BP SEI message.

6. The method according to claim 4, wherein the first image portion, the second image portion, and the previous non-discardable image portion are access units.

7. When executed by at least one processor, In the first Buffering Period (BP) Supplemental Enhancement Information (SEI) message, the maximum value of the initial rejection delay is indicated, To indicate a first flag in an image timing SEI message corresponding to a first image portion of a video data stream, wherein the first flag equal to 1 indicates that if the first image portion is followed by a second image portion having a second BP SEI message in which the concatenation flag is equal to 1 and the initial removal delay is less than the maximum value of the initial removal delay, then a nominal coded image buffer (CPB) removal time based on the elapsed time since the removal of a previous non-discardable picture portion will be applied. By doing so, the system provides an instruction to at least one processor to encode the video into the video data stream such that the video data stream comprises a sequence of image portions, Each image portion contains an image of the encoded video. Non-temporary computer-readable media.

8. The non-temporary computer-readable medium according to claim 7, wherein the elapsed time since the removal of the previously non-discardable image portion is indicated in the second BP SEI message.

9. The non-temporary computer-readable medium according to claim 7, wherein the first image portion, the second image portion, and the previous non-discardable image portion are access units.

10. A program for encoding a video into a video data stream such that the video data stream comprises a sequence of image portions, wherein each image portion has an image of the encoded video within it. The aforementioned program, on a computer system, In the first Buffering Period (BP) Supplemental Enhancement Information (SEI) message, the maximum value of the initial rejection delay is indicated, To indicate a first flag in an image timing SEI message corresponding to a first image portion of the video data stream, wherein the first flag equal to 1 indicates that if the first image portion is followed by a second image portion having a second BP SEI message in which the concatenation flag is equal to 1 and the initial removal delay is less than the maximum value of the initial removal delay, then a nominal coded image buffer (CPB) removal time based on the elapsed time since the removal of a previous non-discardable picture portion is applied. To execute program.

Images