Network device and method of error handling

Abstract

To provide a network device with reduced packets, and a method for error handling.SOLUTION: In order to ease transmission of a data stream 20, an encoder 10 encodes a video 12 into the data stream 20. Slices 26a, 26b, which are portions of the data stream 20, are restricted to either completely lying within one tile 22, i.e., to not cross any tile boundary, or to be composed of two or more tiles in tile order completely. In encoding the video 12 in units of the slices 26a and 26b, the encoder 10 uses entropy coding and in particular context-adaptive entropy coding with continuous adaptation of the contexts' entropy probabilities so as to adapt the probabilities used for entropy encoding to the actual symbol statistics and picture content, respectively. The contexts' probabilities are reset at the beginning of each slice.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a network device and an error handling method related to a transport stream of a series of packets in which a video data stream is transmitted.

[0002] Depending on the application, video data stream transmission based on transport packets is damaged by packet loss. This type of packet loss can occur, for example, from transmission errors that exceed the error correction capabilities of any forward error correction optionally used in the transport stream, the lack of any uplink connection for sending an acknowledgment of the received signal, or a combination of both. Regardless of the effectiveness of the receipt uplink acknowledgment, it is desirable to minimize the affected portion of the video data stream that cannot be decoded due to the non-receipt of lost packets.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0004] However, the packets in the transport stream can contain the necessary information for decoding the content carried by subsequent packets in the transport stream. In the HEVC standard [HEVC (High Efficiency Video Coding) / International Standard for High Efficiency Video Coding], for example, a video data stream includes independent slices and dependent slices. A dependent slice is, for example, contained in the immediately preceding independent slice and depends on the independent slice insofar as slice header data inherited to decode the dependent slice is involved.

[0005] Therefore, it is advantageous to have a provisional concept that can reduce the amount of undecodeable portion of the video data stream affected by packet loss.

[0006] Therefore, it is an object of this application to provide this kind of concept for dealing with errors occurring in the transport stream of a series of packets through which a video data stream is being carried. That is, the concept can affect packets that are lost as little as possible, even if they are received but cannot be decoded. [Means for solving the problem]

[0007] This objective is achieved by the subject matter of the attached independent claim. The findings of this application are that the number of packets (despite being accurately received) due to packet loss provides and analyzes error tolerance in packets of a sequence of packets, and for each run of one or more lost packets in a sequence of packets, the first packet in the sequence of packets after each run of one or more lost packets that carries the start of any tile of the video data stream, and simultaneously carries a slice, the slice header of which is contained in one of the packets in the sequence and is not lost. In particular, the side information overhead for transmitting error-tolerant data is relatively low compared to the reduction of packets adversely affected by packet loss.

[0008] Advantageous embodiments are the subject matter of the dependent claims, and preferred embodiments of this application relate to any of the drawings described below. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows a schematic diagram of an encoder by which video is encoded and a video data stream is generated, and embodiments of the present invention can be supported by the encoder of Figure 1. [Figure 2A] Figure 2A is a schematic diagram showing a decoder, the video that is reconstructed based on the video data stream, the video data stream, and the transport via a series of packets, and an embodiment of the present application may be applied to the decoder of Figure 2. [Figure 2B] Figure 2B is a schematic diagram showing a decoder, the video that is reconstructed based on the video data stream, the video data stream, and its transport through a series of packets, and an embodiment of the present invention may be applied to the decoder of Figure 2. [Figure 3] Figure 3 schematically shows Image 14 divided into tile and slice portions according to the first option. [Figure 4] Figure 4 schematically shows an overview of Image 14 using other division options. [Figure 5] Figure 5 illustrates an example of two packet streams on a lossy channel, illustrating the problems addressed by the embodiment of this application. [Figure 6] Figure 6 shows a schematic block diagram of a network device according to an embodiment. The network device may consist of only a part of it, or it may be connected before the decoder in Figure 2. [Figure 7] Figure 7 shows, in more detail, the possible operating modes of the error handler in Figure 6, using the structure of a schematic flowchart. [Modes for carrying out the invention]

[0010] The following description of embodiments of the present application begins with a description of a typical video codec or typical encoder / decoder structure. The problems arising from packet loss are described later. Embodiments of the present application are described later. These embodiments are applicable, in particular, to the encoder / decoder structures described above.

[0011] Figure 1 shows an encoder 10 configured to encode a video 12 consisting of a series of images 14 arriving at the encoder 10's input into a data stream at the encoder 10's output. The encoder 10 may be configured to encode the series of images 14 using an encoding order that may, but does not necessarily, follow the chronological order 16 of the images 14. More precisely, the encoder 10 may be a composite video encoder configured to select different available prediction modes for blocks 18 in which the images 14 are divided. Such prediction modes may include, for example, a spatial prediction from a previously encoded portion of a previously encoded image and a temporal prediction from a previously encoded portion of the same image. However, additionally or alternatively, other prediction modes may also be supported by encoder 10, for example, as an inter-layer prediction mode from a previously encoded layer of lower quality. Alternatively, inter-viewpoint prediction from a previously encoded viewpoint showing the same scene in time aligns the images 14 of video 12. The encoder 10 sends signals of prediction parameters associated with the selected prediction mode, along with the encoding of residual predictions in the data stream 20 at its output. For example, spatial prediction may involve copying already encoded samples to the current block 18 and including an extrapolation direction indicating direction / extrapolation along the neighborhood, and temporal prediction modes may be implemented, such as prediction compensation with motion vectors, like motion prediction parameters. The prediction mode between viewpoints can be implemented in a motion-compensated manner, thereby obtaining the disparity vector as a prediction parameter. When making a prediction, the "previously encoded part" of video 12 is defined by the encoding order described above. And it sequentially passes through image 14. Within each image 14, the encoding order also passes through blocks 18 in a predetermined order. And it reads, for example, from the upper left corner of image 14 to the lower right corner at the bottom in a raster scan method.

[0012] To enable parallel encoding and parallel decoding of images 14 with selective / partial decoding of video 12 and / or images 14 of video 12, encoder 10 of FIG. 1 supports so-called tile splitting. According to the tile splitting, each image 14 is partitioned, for example, into an array of a number of tiles 22. In FIG. 1, it is illustratively shown that one image 14 is divided into a 2×2 array of tiles 22, but it is also possible to use any m×n division (if m + n>1, the tile splitting is effective). The division into tiles 22 can be restricted so as not to cross block 18. That is, it is restricted to be aligned so as to block the boundary. The tiles may be, for example, a p×q array of blocks 18. As a result, the tiles in a row of tiles have equal q, and the tiles in a column of tiles have equal p.

[0013] Encoder 10 sends a signal of the tile splitting of image 14 in data stream 20 and, in particular, encodes each tile 22. That is, interdependencies resulting from, for example, spatial prediction, context selection of entropy-encoded data stream 20, which are individually decodable from data stream 20, for example, as prediction and entropy decoding up to each tile 22, are restricted at tile boundaries so as not to cross the latter. The above encoding is suitable for tiles that are split as follows. Within the extent of each image 14 adapted to be split, the encoding order passes through the image 14 first within the first one extent of the tile 22. And it traverses to the next tile in the tile order. The tile order may be a raster scan order leading from the top leftmost tile to the bottom rightmost tile of the image 14.

[0014] For the sake of convenience of explanation, FIG. 1 shows the encoding order for one exemplary image 14 having the reference numeral 24.

[0015] To facilitate the transmission of the data stream 20, the encoder 10 encodes the video 12 into the data stream 20 in the above-described method in terms of so-called slices. A slice is a part of the data stream 20 that follows the above-described encoding order. The slice is restricted to either be completely within the extent of one tile 22, i.e., not to cross any tile boundary, or to be composed of two or more tiles for the tile. That is, so that they entirely cover two or more tiles, thereby coinciding with the contour of the covering tiles and the slice boundary.

[0016] FIG. 1 shows the image 14 of FIG. 1 split into two slices 26a, 26b. The first slice 26a of the encoding order 24 is composed of the first two tiles 22 in tile order, and the second slice 26b covers the lower half of the image 14, i.e., the third and fourth tiles 22 in tile order. When the encoder 10 encodes the video 12 in terms of the slices 26a and 26b, in order to adapt the probabilities used for entropy encoding to the actual symbol statistics and picture content respectively, context-adaptive entropy encoding is used, particularly context-adaptive entropy encoding using successive adaptation of context entropy probabilities. The probabilities of the context are reset at the start of each slice 26a and 26b, or reset or initialized at each tile boundary within each slice.

[0017] Figure 1 illustrates slice 26 of the data stream 20. The slice contains data for the first two tiles 22 of the image 14. Furthermore, slice 26 includes a slice header 30 that provides some high-level information about the corresponding portion of the image 14 and the encoding type chosen to encode slice 26, namely, information about whether the first two tiles 22, for example, tile 26, relate to an intra-encoded portion, a p-encoded portion, or a b-encoded portion. Without the information in the slice header 30, the tiles of slice 26a cannot be correctly decoded.

[0018] Another mechanism is to further subdivide the slice so that the transmission of the further coded data stream 20 can be subdivided. According to this principle, each slice 26a and 26b constitutes exactly one independent slice segment, or rather, a series of independent slice portions followed by slice 26a or a dependent slice portion. Slice 26a cannot be divided further. Thus, the encoder 10 can simply output slice 26a completely. With respect to slice 26b, things are different. In the coding order having the tile boundary of tile 22 in slice 26b which coincides with the boundary of slice portions 28a and 28b, with the dependent slice portion 28b, slice 26b consists of the independent slice portion 28a that is followed. The segments 28a and 28b of the slice therefore have similar properties, as the slice does, i.e., they are independently decodeable except for the slice header. The dependent slice portion 28b inherits the same slice header 30 from the independent slice portion 28a that precedes, i.e., leads, slice 26b.

[0019] Before discussing the challenges arising from possible packet loss during transmission, the decoder 50, which is adapted to the encoder 10 in Figure 1, is described with respect to Figure 2. The decoder 50 then illustrates an embodiment for a network device for processing a data stream. The decoder 50 receives the data stream 20 and reconstructs the video 14 from it. The decoder 50 receives, for example, slice 26a followed by slice 26b. For example, decoder 50 may be of a type that decodes composite video; that is, it may be a composite video decoder. It then uses the confirmed prediction node described above to reconstruct the image 14 portion of video 12, corresponding to slices 26a and 26b. In decoding slice 26a, for example, the decoder 50 determines the slice type of slice 26a and uses the slice header 30 to reconstruct the first and second tiles 22 of image 14 from slice 26a in a slice type-dependent manner. For example, for I slices, the effectiveness of P and B slices is provided, while the temporal prediction mode is not available. Therefore, the analysis of the payload data of slice 26a can rely on the slice header 30. In particular, the decoder 50 can encode slice 26a in the context-adaptive manner outlined above by, for example, initializing the initial context probability of slice 26a, and then using the prediction mode and prediction parameters signaled within slice 26a to predict the first and second tiles 22 within slice 26a. The resulting prediction signal is then combined with the prediction residuals that fall within the range of the payload data of slice 26a. In decoding tile 22, the decoder 50 follows the coding order outlined above. However, the decoder 50 can perform several decoding tasks in parallel, as long as they involve tile 22. This is true for the prediction, for example, so that the prediction is constructed so as not to cross tile boundaries. Thus, interdependence between the decoding of tiles in the same image 14 is avoided, and entropy decoding can be performed in parallel as far as tile 22 is concerned.

[0020] When decoding slice 26b, the decoder 50 is capable of decoding slice 26b, which is independent of slice 26a. In particular, the independent slice portion 28a that provides the data for the third tile 22 in image 14 includes a slice header on its own, and the decoder 50 can reconstruct this third tile without needing any other data. However, as long as the dependent slice portion 28b is involved, the decoder 50 immediately does the same thing, namely inheriting slice header data from the slice header 30 contained in the independent slice portion preceding the independent slice portion 28a of the same slice 26b, and therefore decoding the fourth tile requires knowledge of the slice header of slice portion 28a in addition to the existence of slice portion 28b.

[0021] With respect to the transmission of data stream 20, in the Network Abstraction Layer (NAL) unit, it takes the form of slice segments 26a, 28a, and 28b, or is framed. In the following description, the slice portion and the NAL unit of the slice portion are not specifically distinguished. This is because the slice portion is substantially the same as the NAL unit that carries it. The header of the small NAL unit simply has an NAL unit type that indicates the contents of the NAL unit, even though it is a slice portion.

[0022] However, it should be noted that during transmission, the slice portion may be even more fragmented so that it fits into the payload portion of the converted packet. Preferably, this is done in such a way that the start or beginning of a new tile in a particular slice 26a is inserted into a new transport packet. As will be further described below, if slice header data is available, and for example if the previous packet is lost, this means that the same means is decodeable with respect to the dependent slice portion 28b whose beginning indicates the beginning of encoding a new tile. Figure 2 illustrates the splitting of slice portions 26a, 28a, and 28b within the payload data section 32 of a transport packet 34, which includes a transport packet header 36 in addition to the payload portion 32. There, Figure 2 also shows that the trailing portion of the payload portion of the transport packet, which includes the ends of the slice portions, may be filled with padding bits 38 that are distinguished from size portion data by simply being hatched within the scope of packet 34 and showing slice segment data with shaded heading bits 38.

[0023] A problem arises whenever packets are lost during transmission. In particular, imagine that slice portion 28a is not received at all by the decoder 50 due to the loss and fragmentation of slice portion 28a in the second and third packets. However, the first transport packet 34 carries the slice header 30. Therefore, the decoder can resume decoding image 14 with the dependent slice segment 28b, and its decoder 50 can certainly provide that the slice header 30 of the independent slice segment 28a, which was received before the lost packet, is the slice header belonging to the dependent slice segment 28b. However, this is not guaranteed for decoder 50 in any case.

[0024] For example, looking at Figure 3, which shows a typical image 14 subdivided / divided into six tiles, that is, three columns of two rows of tiles 22, each tile has one slice segment. In particular, in Figure 3, the first tile is incorporated into an independent slice portion 28a, while the following five slice portions are subordinate slice portions 28b. Figure 4 shows the same tile being divided. However, it shows that the first three tiles of the first column of tile 22 make up one slice 26a consisting of the first independent slice portion 28a that covers the first tile. The second and third tiles 22 in Figure 4 are followed by two dependent slice portions 28b. Similarly, the second slice 26b includes a sequence of independent slice portions 28a covering the fourth tile in Image 14. This is followed by two dependent slice portions 28b relating to the fifth and sixth tiles in Image 14. If all data related to image 14 is received, decoding image 14 is not a problem, regardless of whether the encoding side selected the option in Figure 3 or the option in Figure 4. However, problems arise, for example, when the fourth slice portion is lost. In the case of Figure 3, there is actually no problem with the subsequent slice segments related to the fifth and sixth tiles of the same image 14, which inherit the slice header data from the first slice segment. However, in the case of Figure 4, as they require the slice header data of the lost fourth packet, the slice portions relating to the fifth and sixth tiles are no longer of value. And in the case of Figure 4, it is an independent slice portion.

[0025] In the case of Figure 3, the decoder 50 suggests the concept outlined below in order to enable it to resume encoding of image 14 with respect to the fifth and sixth tiles 22. A data stream with fault-tolerant data provides the ability to identify packets carrying slice headers of independent slice segments for dependent slice segments.

[0026] Note that in a slice decoding the entropy contained in a subslice, such as slice 26a in Figure 2, an encoder and decoder 50 are formed whenever decoding is in progress, which resets the continuous context probability, i.e., the context probability, so that the first syntactic element of the second or subsequent tile of each slice portion occurs up to a default value. For this reason, as shown in Figure 26a, if there is, for example, one slice portion, then if the first transport packet 34 carrying the slice header 30 is received correctly, then as a result, the second tile of the decryptable slice portion 26a is still transported, even though the second one is carrying slice 26a. The decoder 50 can perform entropy decoding, which uses a default initialization value for the context probability of decoding, to decode the data of slice 26a relating to the second tile 22. In this decoding, it uses slice header data 30 consisting of the first transport packet 34 of six packets in which slice 26a is fragmented into tiles. That is, it opens a new packet 34 with respect to the second tile by the first syntactic element and fills it with the preceding packet containing the end of the data of slice 26a relating to the first tile with padding data. Thus, the case described in Figure 3 is very similar to the case where a single, independent slice covers the entire image 14. In the entropy coding / decoding of this independent slice portion, the context probability is reinitialized each time a tile boundary occurs between consecutive tiles in the tile order. Accordingly, the decoder can, for example, resume decoding the fifth tile despite packet loss, for the fourth tile, provided that the header of the slice header is correctly received at the beginning of an independent slice segment.

[0027] The issues outlined in Figures 3 and 4 are, in other words, outlined again below. In video transmission scenarios, loss is often expected. This type of loss may result in data that, despite being received correctly, cannot be decoded into data that is lost due to dependencies. For example, in RTP, and as illustrated in Figure 2, slices 26a and 26b can be transported through several RTP packets 34 (usually called split units). If one of those packets 34 in a slice is lost, many decoders, such as decoder 50, must either discard all the data in the corresponding slice or decode some of the lost data and discard the rest of the received data in that slice. However, slices 26a,b may each contain independent parts that can be decoded. This is the case for multiple tiles contained in a single slice for HEVC[1]. When multiple tiles 22 are contained within a single slice (for example, two or more as shown in Figures 3 and 4), it may be desirable to transport data that considers tile boundaries (by the first tile of the remaining tiles within the range of the slices contained in the independent slice portion and the slices contained in the dependent slice portion) within a single independent slice segment (cp.26a in Figure 2) or one slice segment per tile (cp.26b in Figure 26b). In other words, RTP packets 34 can be aligned to tile boundaries. Or, to put it another way, each RTP packet may contain data for only one tile, rather than for several tiles, and smart data fragmentation may occur. For example, an RTP packet can be aligned to a tile boundary if it is a single independent slice portion carrying several tiles. By doing so, if some data from certain tiles is lost, or if some selective decoding of partial data is performed, it is still possible to decode other tiles because they do not depend on the non-received tiles for decoding.

[0028] Where described, if both or any dependent slices of slice portions for multiple tiles (cp. 26a) are used (cp. 26b in Figures 2, 3, and 4), then all tile slices 22 require the correct reception of the slice segment header 30 of the independent slice 28a. However, in scenarios where not all data is received and some packets 34 are missing, it is impossible to know whether the slice segment header of the last independent slice segment is the same as the slice segment header of a given slice segment after the loss (and, for example, independent tile), without the concept outlined further below. Or, the essential slice segment header of an independent slice was not received due to the loss. An example is shown in Figure 5.

[0029] In the top-level embodiment of Figure 5, if the required slice segment header information includes an independent portion of the data from the time of reception (e.g., a tile), as described above, the fifth packet, which exemplifies packaging each slice portion of Figure 3 into separate packets, can be decoded. On the other hand, in the bottom-level embodiment of Figure 5, which exemplifies packaging each slice portion of Figure 4 into separate packets, the fifth packet cannot be decoded from the time the header information contained in the previous packets is lost.

[0030] The concepts outlined below utilize the fact that there is some data that is independent of others and can be used to provide several kinds of error tolerance in lossy environments. The problem is that without the concepts outlined below, it is impossible to determine whether this crucial information, contained in the previous slice segment header of an independent slice, has been received or lost.

[0031] Therefore, according to the concepts outlined below, some signaling is added to allow the receiver to detect whether previously received slice header data from an independent slice segment is applied to the currently received data, or whether some of the required data is applied to data that is missing.

[0032] An example of this type of signaling could be some supplementary information in the NAL unit specific to the RTP payload. For example, the PACSI of the RTP payload, which formats for SVC (RFC6190), has been extended to include identifiers in the slice segment header necessary for decoding the data in HEVC or RTP packets.

[0033] This signaling may involve a flag (e.g., a T flag) indicating the presence or absence of this type of error-tolerant information, for example, in the form of an identifier. Supplementary information is used to assign this identifier to a specific slice segment header or to indicate which slice segment header of an independent slice segment having a given identifier is necessary for certain data to be encodeable. In other words, if this information directly precedes data that has a slice segment header for an independent slice segment, the identifier is assigned to the slice segment header of that independent slice segment. Otherwise, it indicates which is the identifier for the slice segment header necessary to correctly decode the following data.

[0034] In the embodiment, the initial data includes independent decryptable data that requires specific header information, which is sent only once for all the data. This header information can identify essential header information if received correctly, and additional supplementary information can decrypt other received independent decryptable data, even if some data is lost, if it matches previously received header information.

[0035] The video data stream provides error tolerance data for the transmitted packets, and the analysis is performed based solely on this outlined concept, which is explained in more detail below in relation to the following diagram.

[0036] In particular, Figure 6 shows a network device 200 that could be positioned in front of the decoder 50 or could form part of it. The network device comprises a receiver 202 and an error handler 204.

[0037] The transport stream received by receiver 202 is indicated by 206. This is formed by a series of packets 208, corresponding to the elements 34 in Figures 1 and 2. The video data stream 210 is then transported through these packets. As mentioned above, the packets may be, for example, RTP packets, but in alternative embodiments, IP packets or the like may be used. The video data stream 210 includes tiles 212 corresponding to elements 20 in Figures 1 and 2, corresponding to elements 12 in Figures 1 and 2, corresponding to elements 14 in Figures 1 to 4 of video 216, corresponding to elements 22 in Figures 1 to 4 of image 214, encoded therein according to some encoding order 218, corresponding to elements 24 in Figures 1 and 2, and then leads to the next image 214 in the image encoding order, for example, through the image tiles in the raster scan order, however this does not necessarily coincide with the presentation time order between images 214. In particular, tile 212 is encoded into a data stream 210 using entropy coding and spatial prediction. In this process, tile 212 is encoded into a data stream 210 with context-induced entropy coding and a spatial prediction that is restricted to not cross the boundaries of tile 212, as illustrated by the dotted lines in the figure. The relationships between the contiguous portions covered by tiles 212 of a typical image 214 of the data stream 210 are illustrated on the one hand using the same wavy lines as in image 212, and on the other hand using the data stream 210. Using the regulations, tile 212 is codecable and decodeable in parallel, as far as entropy coding and spatial prediction are concerned.

[0038] The video data stream has tiles 212 encoded therein according to an encoding order 218, with slices 220 as the unit. Each slice 220 also contains data for one tile 212, only in the case where it is modeled for two slices on the right side, as illustrated in full in Figure 6 for the one on the left containing data for three main tiles of the image according to the encoding order 218, or in the case where it is modeled for two slices on the right side, as illustrated in Figure 6 or as illustrated with more tiles. Each slice corresponds to element 30 in Figures 1 and 2 and begins with slice header 222. As discussed above as an example, a set of specific higher-level syntactic elements that are globally valid for the entire slice, such as quantization step size, default encoding mode, slice type code or other things of that kind. This means that, in the case of slice 220, which covers multiple tiles, all tiles contained in that slice, except for the first one, are necessary for successful decoding, however, the data in the slice header of the slice located at the beginning of the slice is required.

[0039] Although not mentioned previously, it may also mean that the slice is further subdivided into so-called independent slice portions and dependent slice portions, according to the coding order. The initial independent slice portion, clearly consisting of slice headers, is followed by one or more dependent slice portions that inherit at least some of the slice headers of the independent slice portions and thus require this portion to be available for decoding the dependent slice portions. Each may begin a tile, coinciding with the beginning of any dependent or independent slice segment within the tile.

[0040] The video data stream 210 is packetized into a sequence of packets 208 along an encoding order 218, such that each packet yields data for only one tile. This is again illustrated in Figure 6, which uses four different dashed line types associated with four different tiles of a typical image.

[0041] While the receiver 210 is receiving the transport stream 206, the error handler 204 is configured to identify lost packets in the packet 208, i.e., packets that have not been received, were not received in time, contain errors, or have been received in a state where forward errors cannot be corrected, for example, due to too many bit errors occurring during transmission. Furthermore, the error handler 204 analyzes the error tolerance data within packet 208 of the sequence of packets to identify the first packet in the sequence of packets of each of the one or more lost packets, carrying the beginning of any tile and whose slice header is contained in a slice that is contained in any of the packets of the sequence that are not lost. For example, imagine that packet 208, identified using arrow 224, is lost. Typically, the following two packets, which form the same slice, i.e., fragments 226 and 228, would be discarded by the transport layer. Here, error handler 204 identifies packet 226 as a packet that satisfies all of the requirements mentioned above. (1) This is after one or more lost packets, i.e., after each run of packet 224. (2) The slice header 222 of the slice 220 to which the data contained in packet 226 belongs is not lost. (3) Packet 226 carries the beginning of tile 212, (4) This packet 226 is the first packet that satisfies conditions (1) to (3) above. Packet 228, which lies between the lost packet 224 and the positively mentioned packet 226, does not satisfy requirement (2) above. Therefore, the error handler 204 can decode the content, i.e., its tile 212, into packet 226 instead of discarding it. Naturally, the error handler 204 continues to process packets sequentially from that packet until it encounters the next run of one or more lost packets.

[0042] Correctly mentioned error-tolerant data may be included in the packet header of the transport packet 208, or, for example, in supplemental enhanced NAL units of the data stream 210 scattered between payload slices 220.

[0043] The operating modes of the error handler 204 are described in more detail below with reference to Figure 7. In particular, the error handler sequentially examines the sequence of packet inputs in one method 300 to identify lost packets. Identification 302 may include, for example, checking the packet header 36 (compare with Figures 1 and 2) such as the round-robin packet number for each packet 208. Method 300 detects runs of one or more lost packets in a sequence of packets, as illustrated in 304, and shows the sequence of packets 208 in that order, with the lost packets placed above sequence errors 306 and displayed below errors 306. As illustrated, one exemplary run can be seen in 308. Another method 310, which is performed sequentially by the error handler 204, concerns analyzing error tolerance data in packets of a sequence of packets. Within this method 310, for each run 308, the error handler-204 carries the head of any tile and identifies the first packet in the sequence of packets following each run 308 that carries the slice, and that its slice header is contained in one of the packets in the sequence of packets and is not lost. Method 310 cycles through the received packets following run 308. The first received packet 308 after run 308 is shown in Figure 7 using A. Within method 310, the error handler 204 checks whether each packet A carries the start of any tile. Within this check 312, the error handler 204 submits, for example, the payload data section 32 of packet A to the parsing process to determine whether this payload portion 32 starts at the beginning of encoding an arbitrary tile, or at least is parsable up to the point of reaching such encoding. For example, if the start of the payload data section 32 coincides with the start of a slice segment NAL unit, the error handler 204 reads an edge field from the slice segment to evaluate whether the slice segment starts encoding a new tile.

[0044] Alternatively, the error handler 204 checks whether packet A is the first fragment of a NAL unit, for example, based on the fragmentation parameters in the transport packet header 36, in which case it is inferred that packet A has the starting point of a new tile in the payload data 32. If step 312 determines that the packet does not coincide with the start of a new tile, method 310 proceeds to the next received packet, in this case B. However, if the result of check 312 is positive, i.e., a tile start is found, then in method 310, the error handler 204 checks whether the current packet A itself contains a slice header. If the answer is YES, then everything is normal and can resume from packet A in the decoding procedure after the run of lost packet 308, as shown in step 316. However, if the check in step 314 reveals that the current packet does not contain a slice header, the error handler 204 examines the error-tolerant data in the current packet A to identify the slice header of the slice carried by each packet, i.e., the slice header inherited by the dependent slice segment relating to the new tile, or the slice header of the slice, i.e., to include the independent slice segment to which the new tile identified in step 312 belongs. Identifier 318 can operate as follows: For example, the transport packet header 36 of packet A itself may contain error-tolerant data, which may be pointers to some of the preceding packets. If this packet belongs to the received packet 208 and a check is performed at 320, decoding is resumed at 316. However, if the required slice header belongs to the lost packet, i.e., not to the received slice header, method 310 searches for a packet with the new slice header in step 322. This step corresponds to the concatenation of steps 312 and 314, and since each “dependent tile” requires a slice header belonging to one of the lost packets, if the current packet does not contain the new slice header, the process returns to step 312.

[0045] Instead, it should be noted that specific NAL units may be scattered among the actual slice segment NAL units described above to carry fault-tolerant data.

[0046] When resuming decoding in step 316, decoding of the video data stream is resumed after each run of one or more lost packets from the identified packet by applying the slice header contained in any of the received packets, and is used to decode the tile whose start was identified in step 312, as identified by the fault-tolerant data.

[0047] Therefore, the above explanation clarifies the transfer of error recovery in partial slices.

[0048] The error-tolerant data outlined above can, for example, point to the required slice header by indicating the packet number of the packet in which the required slice header is located. This can be done using an absolute or relative method, i.e., an offset value from the current packet containing the error-tolerant data to the packet containing the required slice header. Alternatively, the required packet can be indexed by the slice address of a separate slice segment containing the required slice header. As mentioned above, every slice segment contains a slice address indicating the location of the first block encoded in this slice segment within image 14.

[0049] To maintain backward compatibility with other devices that cannot process / analyze fault-tolerant data, an extension mechanism including respective flags can be used to enable older decoders to ignore / skip fault-tolerant data and, accordingly, discard it.

[0050] Needless to say, the above concept of using fault-tolerant data manifests itself in a corresponding network device on the transmitting side. This type of network device may be included within the scope of the encoder in Figure 1 or connected to its output. This transmitting network device is configured to transmit a video data stream via a transport stream of a series of packets. The video data has tiles of video images where the image is divided, and these are encoded according to entropy coding and spatial prediction using a coding order. The tiles are then encoded into a video data stream with context derivation of entropy coding, and spatial prediction is restricted so as not to cross the boundaries of the tiles. There, the video data stream has tiles, each slice having an image of the video encoded according to a coding order that is in units of slices, each slice containing either only one tile's data or two or more tiles. Each slice begins with a slice header. The network device is configured to package a video data stream into a sequence of packets in an encoding order such that each packet brings only one tile of data, and to insert error-tolerant data into each packet in the sequence of packets so as to identify the preceding packet in the sequence, for each sequence of packets that does not include a slice header for the slice that each packet partially brings. This includes a slice header for each packet.

[0051] Although some embodiments are described in the context of the apparatus, it is clear that these embodiments also mean descriptions of the corresponding methods. Thus, a block or apparatus corresponds to a method step or a feature of a method step. Similarly, described in the context of a method step, an embodiment means a description of a corresponding block or item or a feature of a corresponding apparatus. Some or all of the ways in which the steps may be such are that a hardware device, such as a microprocessor (or using one), can run a programmable computer or electronic circuit. In some embodiments, one or more of the most important method steps can be performed by this type of apparatus.

[0052] Depending on the specific implementation requirements, embodiments of the present invention can be implemented in hardware or in software. The implementation can be carried out using digital storage media such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs, or FLASH memory, with electronically readable control signals stored on them. And, in order for each method to be executed, it works (or can work) with a programmable computer system. Thus, a digital storage medium can be a readable computer.

[0053] Some embodiments of the present invention include a data storage medium having electronically readable control signals, so that one of the methods described herein can be carried out, and which can cooperate with a programmable computer system.

[0054] Typically, embodiments of the present invention can be executed as a computer program product with program code. When the computer program product runs on a computer, the program code is implemented to carry out one of the methods. The program code can be stored, for example, in a machine-readable carrier.

[0055] Other embodiments include computer programs for performing one of the methods described in this specification, which are stored in a machine-readable carrier.

[0056] In other words, an embodiment of the method of the present invention is, therefore, a computer program having program code. To perform one of the methods described in this specification, the computer program then runs on a computer.

[0057] Further embodiments of the method of the invention are, therefore, data storage media (or digital storage media or computer-readable media) on which computer programs are recorded to perform one of the methods described herein. Data storage media, digital storage media or recorded media are typically specific and / or non-transferable.

[0058] A further embodiment of the method of the invention is a data stream or a sequence of signals, which therefore means a computer program for performing one of the methods described herein. For example, a data stream or sequence of signals may be configured to be transmitted through a data communication connection, for example, by the Internet.

[0059] Further embodiments include processing means, such as a computer or a programmable logic device, or are configured to perform one of the methods described in this specification.

[0060] Further embodiments include a computer on which a computer program is installed in order to perform one of the methods described in this specification.

[0061] Further embodiments of the present invention include an apparatus or system configured to transfer (for example, electronically or optically) a computer program to a receiver in order to perform one of the methods described in this specification. The receiver may be, for example, a computer, a mobile device, a memory device, or something of that kind. The device or system may include, for example, a file server for transferring computer programs to the receiver.

[0062] In some embodiments, programmable logic devices (e.g., field-programmable gate arrays) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by any hardware device.

[0063] The embodiments described above are merely examples to illustrate the principles of the present invention. It is understood that the modifications and changes to the preparations and the details described herein will be obvious to those skilled in the art. Therefore, it is not intended to be limited solely by the scope of the imminent patent claims and not solely by the description of the embodiments and the specific details shown herein as descriptions.

[0064] reference [1]B. Bross, W.-J. Han, J.-R. Ohm, GJ Sullivan, T. Wiegand (Eds.), “High Efficiency Video Coding (HEVC) text specification draft 10“, JCTVC-L1003, Geneva, CH, Jan. 2013 [2] G. Tech, K. Wegner, Y. Chen, M. Hannuksela, J.Boyce (Eds.), “MV-HEVC Draft Text 3 (ISO / IEC 23008-2 PDAM2)“, JCT3V-C1004, Geneva, CH, Jan. 2013 [3] G. Tech, K. Wegner, Y. Chen, S. Yea (Eds.), “3D-HEVC Test Model Description, draft specification“, JCT3V-C1005, Geneva, CH, Jan. 2013 [4]WILBURN, Bennett, et al. High performance imaging using large camera arrays. ACM Transactions on Graphics, 2005, 24. Jg., Nr. 3, S. 765-776. [5]WILBURN, Bennett S., et al. Light field video camera. In: Electronic Imaging 2002. International Society for Optics and Photonics, 2001. S. 29-36. [6]HORIMAI, Hideyoshi, et al. Full-color 3D display system with 360 degree horizontal viewing angle. In: Proc. Int. Symposium of 3D and Contents. 2010. S. 7-10.

Claims

1. An error handler for a video decoder, the error handler being configured to receive a transport stream of a sequence of packets through which a video data stream is transferred, where in the video data stream, tiles of the image into which the video image is divided are encoded using entropy encoding and spatial prediction in an encoding order, in the video data stream, the tiles of the image of the video are encoded in slice units in the encoding order, each slice containing data of only one tile or completely containing two or more tiles, and each slice starting from a slice header, the video data stream is packetized into a sequence of the packets in the encoding order, and each packet is adapted to carry data of only one tile, the error handler further identifies lost packets in a sequence of packets, analyzes error resilience data in the packets of the sequence of packets, and for each run of one or more lost packets in the sequence of packets, identifies a first packet in the sequence of packets after the run of one or more lost packets, the first packet carrying a start portion of one of the tiles and being involved in carrying a slice, and at which the decoding of the video data stream will resume, and based on the error resilience data of the first packet, identifies a packet in the sequence of packets that is not a lost packet and contains the slice header of the slice carried by the first packet, and the decoding resumes using the slice header of the slice carried by the first packet, error handler.

2. The error handler checks whether each subsequent packet following each run of the one or more lost packets carries a start portion of any of the tiles, and for each packet found by the check to match a start portion of any of the tiles, Identifying a slice header of a slice carried by each of the packets based on the error tolerance data of each of the packets, and Checking whether the slice header of the slice carried by each of the packets is included in any of the packets in the packet sequence that are not lost packets and precede each run of one or more lost packets, Thereby being configured to sequentially inspect each packet following each run of the one or more lost packets, The error handler according to claim 1.

3. The error handler according to claim 1, wherein the error handler is configured to read the error tolerance data from a transport packet header of the packet.

4. The error handler according to claim 1, wherein the error handler is configured to obtain a pointer or identifier to a packet including the slice header of the slice carried by each of the packets from the error tolerance data of each of the packets.

5. The error handler according to claim 1, wherein the error tolerance data of the first packet includes an offset value from the first packet to a packet including the slice header.

6. A method for error handling, comprising: Receiving a transport stream of a sequence of packets in which a video data stream is transferred, where In the video data stream, tiles of the image into which the video image is divided are encoded using entropy encoding and spatial prediction in the encoding order, In the data stream, the tiles of the image of the video are encoded in slice units in the encoding order, and each slice either includes data of only one tile or completely includes two or more tiles, and each slice starts with a slice header, The video data stream is packetized into the sequence of packets in the encoding order, and each packet is configured to carry data of only one tile of the image, The step of receiving, The step of identifying lost packets in the sequence of packets, Analyzing the error resilience data within the packets of the sequence of said packets to identify, for each run of one or more lost packets of the sequence of said packets, the first packet after each run of the one or more lost packets, which packet carries the start portion of one of said tiles and is involved in the conveyance of a slice and which is the first packet within the sequence of said packets at which the decoding of said video data stream will resume, and based on the error resilience data of said first packet, identifying the packets of the sequence of said packets that are not lost packets and that contain the slice header of the slice carried by said first packet, and comprising, a method in which decoding is resumed using the slice header of the slice carried by said first packet. **Claim 7** A computer program having program code for performing the method according to claim 6 when running on a computer.

Images