Receiving device and program
The receiving device addresses decoding interruptions and cost increases by combining and replacing lost data in L0 and L1 bitstreams, ensuring seamless video playback.
Patent Information
- Application Number
- JP2024079426
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-28
AI Technical Summary
When transmitting L0 and L1 bitstreams through different transmission paths, loss in the L1 bitstream due to noise can cause decoding interruptions and require additional resolution conversion processing, leading to increased costs and video interruptions.
A receiving device that combines L0 and L1 bitstreams, determines reception states, and replaces missing parts with replacement data to maintain decoding without initialization, avoiding resolution conversion and cost increases.
Ensures continuous decoding without interruptions and reduces additional processing costs by replacing lost data in the L1 bitstream, maintaining video playback.
Smart Images

Figure 2024166153000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a receiving device and a program. [Background technology]
[0002] In recent years, the devices used to watch broadcasts and streaming services have become more diverse, including large TVs, mobile devices, tablets, etc. Since these devices have different video processing capabilities and display resolutions, it is desirable to transmit video with a resolution and frame rate suitable for each device.
[0003] Versatile Video Coding (VVC), which is the latest video coding method, employs a multi-layer coding function for efficiently coding a plurality of videos as described above (see, for example, Non-Patent Document 1).
[0004] In multi-layer coding, multiple input images to be coded are coded in a layer structure. The lowest layer (L0) is coded first, and the decoded image of L0 is used for prediction in the coding process of the higher layers (L1, L2, ...), which improves the prediction efficiency of the higher layers and has the effect of improving coding efficiency compared to coding each layer independently. In particular, spatial scalable coding, in which low-resolution image is coded as L0 and high-resolution image L1 is coded using the decoded image of the low-resolution image, is a typical use case of multi-layer coding.
[0005] As in the current examples of terrestrial digital broadcasting and one-segment broadcasting, broadcasting services may use different broadcasting formats depending on the receiving capabilities of the target viewing terminal. For example, for viewing terminals that are expected to be viewed while moving, such as mobile phones, a broadcasting format that can be received and decoded even when the C / N ratio is low may be adopted, while for terminals that are expected to be viewed at fixed reception in the home, a broadcasting format that can be received and decoded only in environments with a high C / N ratio may be adopted.
[0006] Of the multi-layer bit streams encoded by VVC multi-layer encoding, the L0 bit stream encoding low-resolution video is transmitted over transmission path A, which has a small transmission capacity but is resistant to noise, and the L1 bit stream encoding high-resolution video is transmitted over another transmission path B, which is not resistant to noise but has a large transmission capacity.This makes it possible to watch low-resolution video on viewing terminals designed for mobile reception such as mobile terminals, and high-resolution video on viewing terminals such as large televisions for fixed reception at home. [Prior art documents] [Non-patent literature]
[0007] [Non-Patent Document 1] ISO / IEC 23090-3, “Information technology - Coded representation of immersive media - Part 3: Versatile video coding” Summary of the Invention [Problem to be solved by the invention]
[0008] When the L0 bit stream and the L1 bit stream are transmitted over different transmission paths, the receiving device must synthesize the L0 bit stream and the L1 bit stream transmitted over the respective transmission paths, restore the multi-layer bit stream output by the encoder on the transmitting side, and input it to the decoder. As mentioned above, when the L1 bit stream is transmitted using a transmission method that has a large transmission capacity but is vulnerable to noise, there may be cases where the L1 bit stream is lost due to rain attenuation in bad weather, even in fixed reception.
[0009] In such a case, since the high-resolution video cannot be decoded for the portion of the L1 bitstream data that is missing, the decoder must decode the L0 bitstream and display the low-resolution video. In this case, the decoding process on the receiving device side needs to change the screen resolution of the video output from the high-resolution video decoded from the L1 bitstream, which is the previous video output, to the low-resolution video decoded from the L0 bitstream, so the decoding process needs to be initialized. This may cause a problem that the video is interrupted once on the receiving side. In addition, a resolution conversion process must be implemented in the receiving device in addition to the decoder to up-convert the low-resolution video obtained by decoding the L0 bitstream to the resolution of the high-resolution video that was previously displayed, which may lead to a problem of increased costs.
[0010] Therefore, the present invention aims to provide a receiving device and a program that can continue decoding processing without initializing the decoding processing even if a loss occurs in a bit stream of a higher layer, and that can suppress an increase in costs. [Means for solving the problem]
[0011] A receiving device according to a first aspect includes a first receiving means for receiving a bit stream of a first layer in multi-layer coding, a second receiving means for receiving a bit stream of a second layer higher than the first layer, a combining means for performing a layer combining process for combining the bit stream of the first layer and the bit stream of the second layer and outputting a multi-layer bit stream, and a determining means for determining a receiving state in the second receiving means. When the determining means determines that a part of the bit stream of the second layer is lost, the combining means replaces the bit stream of the lost part with replacement data and then performs the layer combining process.
[0012] A program according to the second aspect causes a computer to function as the receiving device according to the first aspect. Effect of the Invention
[0013] According to the present invention, a receiving device and a program can be provided that can continue decoding processing without initializing the decoding processing even if a loss occurs in a bit stream of a higher layer, and that can suppress an increase in costs. [Brief description of the drawings]
[0014] [Figure 1] FIG. 1 is a diagram illustrating a transmission system according to an embodiment. [Diagram 2] 1 is a diagram illustrating a configuration of a transmission device according to a first embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a multi-layer bitstream. [Figure 4] 1 is a diagram illustrating a configuration of a receiving device according to a first embodiment. [Diagram 5] FIG. 4 is a diagram showing a normal decoding process in the receiving device according to the first embodiment. [Figure 6] 4 is a diagram showing a decoding process when an L1 transmission error occurs in the receiving device according to the first embodiment. FIG. [Figure 7] 5 is a diagram for explaining the operation of a reception state determination unit according to the first embodiment. FIG. [Figure 8] 4 is a diagram showing an example in which a reception state determining unit according to the first embodiment performs determination on a frame-by-frame basis. FIG. [Figure 9] 4 is a diagram showing an example in which a reception state determination unit according to the first embodiment performs determination on a chunk-by-chunk basis; FIG. [Figure 10] FIG. 11 is a diagram for explaining a first modification of the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating a configuration of a transmitting device according to a second embodiment. [Figure 12] FIG. 11 is a diagram illustrating a configuration of a receiving device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] The embodiments will be described with reference to the drawings. In the description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.
[0016] (1) First embodiment The first embodiment will be described with reference to FIGS.
[0017] (1.1) Transmission System 1 is a diagram showing a transmission system 10 according to the first embodiment. As shown in FIG.
[0018] The transmission system 10 may employ a video coding method that supports multi-layer coding. In multi-layer coding, multiple input videos to be coded are coded in a layer structure. The lowest layer (L0) is coded first, and the decoded video of L0 is used for prediction in coding processing of higher layers (L1, L2, ...), thereby improving prediction efficiency of the higher layers, and improving coding efficiency compared to the case where each layer is coded independently.
[0019] In a video coding method that supports multi-layer coding, spatial scalable coding may be applied, in which a low-resolution video is coded as a base layer (also referred to as "L0"), and an enhancement layer (also referred to as "L1"), which is a high-resolution video, is coded using a decoded image of the low-resolution video. For example, the transmitting device 100 may code a 2K resolution video as L0, perform a prediction process of a 4K resolution video using the 2K resolution video, and code difference information between the high-resolution video and the predicted image together with the prediction information as an enhancement layer (also referred to as "L1").
[0020] As a video coding method that supports multi-layer coding, VVC standardized by MPEG may be used. VVC may be considered to be a method defined in ISO / IEC 23090-3. In VVC, in addition to scalability regarding two-layer resolution of 2K / 4K, scalability regarding three-layer resolution of 2K / 4K / 8K is supported, and it may be supported to use an enhancement layer for superimposing additional information such as subtitles and sub-video instead of up-conversion.
[0021] In the transmission system 10, RTP (Realtime Transport Protocol) or MMT (MPEG Media Transport) may be adopted as a transmission method. RTP may be considered to be a method defined in IETF RFC 3550. MMT may be considered to be a method defined in ISO / IEC 23008-1 or ARIB STD-B60.
[0022] The base layer (L0) and the enhancement layer (L1) may be transmitted through different transmission paths 500. For example, among the multi-layer bit streams encoded by multi-layer encoding, the L0 bit stream encoding low-resolution video is transmitted through a transmission path 500A that has a small transmission capacity but is resistant to noise, and the L1 bit stream encoding high-resolution video is transmitted through another transmission path 500B that is not resistant to noise but has a large transmission capacity. By transmitting in this manner, it becomes possible to watch low-resolution video on a receiving device 200 that is premised on mobile reception such as a mobile terminal, and high-resolution video on a receiving device 200 such as a large TV that receives fixedly at home.
[0023] When the L0 bit stream and the L1 bit stream are transmitted over different transmission paths 500, the receiving device 200 must synthesize the L0 bit stream and the L1 bit stream transmitted over the respective transmission paths, restore the multi-layer bit stream output by the encoder on the transmitting side, and input it to the decoder. When the L1 bit stream is transmitted using a transmission method that has a large transmission capacity but is vulnerable to noise, there may be cases where the L1 bit stream is lost due to rain attenuation in bad weather, even in fixed reception.
[0024] The transmission paths 500A and 500B may be optical transmission paths or wireless transmission paths. Although not particularly limited, the transmission paths 500A and 500B may be the same type of transmission path or different types of transmission paths. For example, the transmission path 500A may be a broadcast transmission path, and the transmission path 500B may be a communication transmission path. The transmission path 500A may be a terrestrial broadcast transmission path, and the transmission path 500B may be a satellite broadcast transmission path.
[0025] However, the error rate of the transmission path 500A may be lower than that of the transmission path 500B. The error rate is defined by various parameters such as wavelength, frequency, modulation method, and error tolerance. For example, in a case where the transmission path 500A and the transmission path 500B are both terrestrial broadcast transmission paths, the transmission path 500A may use 64QAM (Quadrature Amplitude Modulation) as a modulation method, and the transmission path 500B may use 1024QAM as a modulation method.
[0026] (1.2) Configuration according to the first embodiment The configuration according to the first embodiment will be described.
[0027] (1.2.1) Configuration of the transmitting device 2 is a diagram showing a configuration of the transmission device 100 according to the first embodiment. The transmission device 100 according to the first embodiment has a multi-layer encoding unit 110, a layer separation unit 120, and a transmission means .
[0028] The multi-layer coding unit 110 codes each unit constituting a base layer (L0) and generates an L0 bit stream. The multi-layer coding unit 110 codes each unit constituting an enhancement layer (L1) and generates an L1 bit stream. The units may be referred to as NAL (Network Abstraction Layer) units. The multi-layer coding unit 110 codes each unit in a predetermined order and outputs a multi-layer bit stream as shown in FIG. 3 to the layer separation unit 120.
[0029] The layer separation unit 120 determines whether a unit constitutes a base layer or an enhancement layer based on a layer ID (e.g., nuh_layer_id) included in the header of the unit. The layer separation unit 120 outputs an L0 bitstream consisting of units constituting the base layer to the transmission means 130A, and outputs an L1 bitstream consisting of units constituting the enhancement layer to the transmission means 130B.
[0030] The transmitting means 130 transmits the bit stream output by the layer separation unit 120 via the transmission path 500. In the first embodiment, the transmitting means 130A transmits the L0 bit stream output by the layer separation unit 120 via the transmission path 500A. The transmitting means 130B transmits the L1 bit stream output by the layer separation unit 120 via the transmission path 500B.
[0031] The transmitting means 130 includes a packetizing unit 131 and a transmitting unit 132. The packetizing unit 131 packetizes units to generate packets. The packets may be packets conforming to the RTP. For example, the packets may be IP packets. The IP packets may include an IP header, a UDP header, an RTP header, and a payload.
[0032] Specifically, the packetizer 131A generates a packet corresponding to a unit constituting a base layer. The packet includes the unit corresponding to the base layer as a payload. The packetizer 131B generates a packet corresponding to a unit constituting an enhancement layer. The packet includes the unit corresponding to the enhancement layer as a payload.
[0033] The transmitting unit 132 transmits the packets generated by the packetizing unit 131 via the transmission path 500. Specifically, the transmitting unit 132A transmits the packets generated by the packetizing unit 131A via the transmission path 500A. The transmitting unit 132B transmits the packets generated by the packetizing unit 131B via the transmission path 500B. Note that the transmitting units 132A and 132B may include an ONU (Optical Network Unit) for an optical line, or may include a modulation unit for wireless communication.
[0034] (1.2.2) Configuration of the receiving device 4 is a diagram showing a configuration of the receiving device 200 according to the first embodiment. The receiving device 200 according to the first embodiment includes a receiving means 210, a layer combining unit 220, a memory unit 230, a reception state determining unit 240, and a multi-layer decoding unit 250.
[0035] The receiving means 210 receives a bit stream via a transmission path 500. In the first embodiment, the receiving means 210A receives an L0 bit stream transmitted via a transmission path 500A. The receiving means 210B receives an L1 bit stream transmitted via a transmission path 500B. The receiving device 200 includes a receiving unit 211, a depacketizing unit 212, and a buffer unit 213.
[0036] The receiving unit 211 receives each packet constituting a bit stream via the transmission path 500. The receiving unit 211A receives packets corresponding to units constituting a base layer via the transmission path 500A. The receiving means 210B receives packets corresponding to units constituting an enhancement layer via the transmission path 500B.
[0037] The depacketizer 212 depacketizes the packets received by the receiver 211 to extract units. Specifically, the depacketizer 212A extracts units constituting the base layer from the packets received by the receiver 211A. The depacketizer 212B extracts units constituting the enhancement layer from the packets received by the receiver 211B.
[0038] The buffer unit 213 accumulates the units extracted by the depacketizer unit 212. The buffer unit 213 may be a FIFO (First-In First-Out) type buffer unit. The buffer unit 213 is used to absorb errors in transmission delay. Specifically, the buffer unit 213A accumulates units that constitute a base layer. The buffer unit 213B accumulates units that constitute an enhancement layer.
[0039] The layer combining unit 220 performs a layer combining process of combining the L0 bit stream received by the receiving means 210A and the L1 bit stream received by the receiving means 210B, and outputs a multi-layer bit stream to the multi-layer decoding unit 250. Specifically, the layer combining unit 220 analyzes the inside of the units stored in the buffer unit 213, rearranges the units in a predetermined order similar to that of the multi-layer encoding unit 110, and outputs the units (multi-layer bit stream) to the multi-layer decoding unit 250 in the predetermined order similar to that of the multi-layer encoding unit 110.
[0040] Although not particularly limited, the layer combining unit 220 may decode a POC (Picture Order Count) of a picture header included in a unit, and then rearrange the units based on the POC. The decoding of the POC may be considered as an example of analyzing the inside of a unit. Note that an appropriate method may be adopted as a method for analyzing the inside of a unit depending on the encoding / decoding method of the unit.
[0041] The memory unit 230 holds replacement data used for replacement in the L1 bitstream, and outputs the replacement data to the layer combining unit 220. Specific examples of the replacement data will be described later. The memory unit 230 may hold different replacement data for each resolution in the enhancement layer. In this case, the memory unit 230 may output replacement data corresponding to the current resolution in the enhancement layer to the layer combining unit 220.
[0042] The receiving state determining unit 240 determines the receiving state in the receiving means 210B (i.e., the receiving state of the L1 bit stream). The receiving state determining unit 240 may determine whether or not there is a packet loss caused by a transmission error in the L1 bit stream. The receiving state determining unit 240 may determine a frame (also called a "picture") corresponding to a lost packet, or may determine a chunk corresponding to a lost packet.
[0043] A chunk is a unit that groups together a number of frames, and is a unit that is divided into bitstreams of a certain length when encoding video. A chunk is a unit that groups a certain number of frames to form a bitstream, which allows for more efficient transmission and playback of the stream. The chunk may be determined based on the frame size and bitrate. In general, a smaller chunk size reduces the delay of the bitstream, but increases the bitrate.
[0044] The multi-layer decoding unit 250 decodes the units (multi-layer bit stream) output from the layer combining unit 220 in the same predetermined order as the multi-layer encoding unit 110, and outputs the decoded video. The multi-layer decoding unit 250 may be provided in a device external to the receiving device 200, for example, a display device.
[0045] In the receiving device 200 configured in this way, when the reception state determination unit 240 determines that a part of the L1 bitstream is lost, the layer combining unit 220 replaces the L1 bitstream of the lost part with replacement data and then performs layer combining processing. In this way, the layer combining unit 220 replaces the L1 bitstream of the lost part with replacement data and then performs layer combining processing, thereby being able to output to the multi-layer decoding unit 250 a multi-layer bitstream of the same format as when there is no loss in the L1 bitstream.
[0046] Therefore, even if a loss occurs in the L1 bitstream, the multi-layer decoding unit 250 does not need to initialize the decoding process, and it is possible to prevent the video from being interrupted. Also, it is not necessary to implement a resolution conversion process for up-converting the low-resolution video obtained by decoding the L0 bitstream to the resolution of the high-resolution video that has been displayed up until that point, in addition to the decoder, in the receiving device 200, and it is possible to suppress an increase in costs.
[0047] (1.3) Operation of the receiving device FIG. 5 is a diagram showing a normal decoding process in the receiving device 200 according to the first embodiment.
[0048] The layer combining unit 220 combines the L0 bitstream and the L1 bitstream to output a multi-layer bitstream to the multi-layer decoding unit 250.
[0049] In the L0 decoding process, the multi-layer decoding unit 250 obtains L0 prediction mode information and L0 prediction residuals from the L0 bitstream, generates an L0 predicted image based on the L0 prediction mode information, and obtains an L0 decoded image by combining the L0 prediction residuals and the L0 predicted image.
[0050] In the L1 decoding process, the multi-layer decoding unit 250 obtains L1 prediction mode information and an L1 prediction residual from the L1 bitstream, generates an L1 predicted image using an L0 decoded image based on the L1 prediction mode information, and obtains an L1 decoded image by synthesizing the L1 prediction residual and the L1 predicted image. Prediction for generating an L1 predicted image from an L0 decoded image is called inter-layer prediction in VVC, and in inter-layer prediction, a technique called RPR (Reference Picture Resampling) is used for prediction between images of different resolutions. Specifically, it is a process of performing inter prediction while enlarging an L0 decoded image to the resolution of an L1 decoded image.
[0051] FIG. 6 is a diagram showing a decoding process in the receiving device 200 according to the first embodiment when an L1 transmission error occurs.
[0052] When an L1 bitstream is lost, the layer splicing unit 220 outputs to the multi-layer decoding unit 250 a multi-layer bitstream in which the lost L1 bitstream is replaced with replacement data held in the memory unit 230 .
[0053] The replacement data includes prediction mode information for generating a predicted image in which the decoded image in L0 is enlarged to the display resolution in L1, and no-residue information indicating that no prediction residual is included for each image block in L1.
[0054] Specifically, in the replacement data, as prediction mode information, information is stored in which a zero vector is stored in all blocks as a motion vector (MV) used when performing inter-layer prediction of an L0 decoded image, and as prediction residuals, information is stored in which prediction residuals in all blocks are zero.
[0055] As a result, the multi-layer decoding unit 250 generates an enlarged image of the L0 decoded image obtained by performing inter-layer prediction on the L0 decoded image using a motion vector with zero motion information as an L1 decoded image when decoding using replacement data. This makes it possible to prevent the multi-layer decoding unit 250 from being stopped or initialized due to a loss of the L1 bitstream, and also makes a separate resolution enhancement process in the multi-layer decoding unit 250 unnecessary.
[0056] In the above example, the replacement data includes a motion vector (zero vector) with zero motion information and a non-prediction residual indicating that there is no residual signal. Furthermore, in the replacement data, block division of the image to be coded is avoided as much as possible (for example, division of CTU or less is not performed except for the edge of the screen). By using such replacement data, the data size can be reduced, and the memory capacity required for the receiving device 200 can be reduced.
[0057] 7 to 9 are diagrams for explaining the operation of the reception state determination unit 240. In these diagrams, it is assumed that each of the L0 bit stream and the L1 bit stream is composed of a plurality of chunks (Chunk0, Chunk1, Chunk2, etc.). In each chunk, a frame indicated with "IDR" means Instantaneous Decoder Refresh, and a frame indicated with "CRA" means Clean Random Access. Such a frame, a CRA frame, may be a frame that does not require a reference frame such as an I frame (I picture).
[0058] 7, each of the receiving means 210 performs a receiving process of a bit stream on a frame-by-frame and chunk-by-chunk basis. For example, the receiving means 210B performs a receiving process of an L1 bit stream on a frame-by-frame and chunk-by-chunk basis. The receiving state determining unit 240 determines a lost frame including a lost packet and / or a chunk including a lost frame.
[0059] FIG. 8 is a diagram showing an example in which the reception state determining unit 240 performs determination on a frame-by-frame basis.
[0060] The reception status determination unit 240 determines a lost frame corresponding to a lost portion (one or more packets) of the L1 bit stream as a replacement frame. In the illustrated example, the reception status determination unit 240 determines that a loss has occurred in one B picture included in Chunk2 as a replacement frame. In this case, the layer splicing unit 220 performs layer splicing processing after replacing the replacement frame with replacement data.
[0061] Furthermore, the reception status determination unit 240 may determine a lost frame and a frame that uses the lost frame as a reference image as a replacement frame group. In the illustrated example, the reception status determination unit 240 determines a lost B picture in Chunk2 and B pictures before and after the lost B picture that references the lost B picture as a replacement frame group. In this case, the layer combining unit 220 performs layer combining processing after replacing each frame constituting the replacement frame group with replacement data.
[0062] In this way, by determining which frames should be replaced on a frame-by-frame basis, the number of frames to be replaced can be reduced, and the time during which the enlarged image of the L0 decoded image is displayed can be minimized.
[0063] FIG. 9 is a diagram showing an example in which the reception state determining unit 240 performs determination on a chunk-by-chunk basis.
[0064] The reception status determination unit 240 determines a lost chunk including a lost portion (one or more packets) of the L1 bit stream as a replacement chunk. In the illustrated example, the reception status determination unit 240 determines that a loss has occurred in a frame included in Chunk2, and that Chunk2 is a replacement chunk. In this case, the layer combining unit 220 replaces the replacement chunk (Chunk2) with replacement data and then performs layer combining processing.
[0065] Furthermore, the reception state determination unit 240 may determine each frame in the lost chunk and a frame that uses the frame in the lost chunk as a reference image as a replacement frame group. In this case, the layer combining unit 220 performs layer combining processing after replacing each frame constituting the replacement frame group with replacement data.
[0066] In this way, the reception status determination unit 240 determines not only the lost frame but also the chunk to which the frame belongs as a lost chunk. The reception status determination unit 240 may determine the lost chunk and the frames that refer to the lost chunk as a replacement frame group. Since the determination of loss on a chunk-by-chunk basis is only performed in the system layer, this can be easily performed. In addition, the range of replacement can be easily determined, and for example, all frames up to the next CRA·IDR picture may be determined as replacement frames.
[0067] (1.4) Summary of the First Embodiment The receiving device 200 according to the first embodiment includes a receiving means 210A for receiving a first layer (L0) bit stream in multi-layer coding, which is a bit stream transmitted via a transmission path 500A, a receiving means 210B for receiving a second layer (L0) bit stream higher than L0, which is a bit stream transmitted via a transmission path 500B different from the transmission path 500A, a layer combining unit 220 for performing a layer combining process for combining the L0 bit stream and the L1 bit stream and outputting a multi-layer bit stream, and a receiving state determination unit 240 for determining the receiving state in the receiving means 210B. When the receiving state determination unit 240 determines that a part of the L1 bit stream is lost, the layer combining unit 220 replaces the bit stream of the lost part with replacement data and then performs the layer combining process.
[0068] As a result, even if a loss occurs in the L1 bitstream, the layer splicing unit 220 can output to the multi-layer decoding unit 250 a multi-layer bitstream of the same format as when there is no loss in the L1 bitstream. Therefore, even if a loss occurs in the L1 bitstream, the multi-layer decoding unit 250 does not need to initialize the decoding process, and it is possible to prevent the video from being interrupted once. In addition, it is not necessary to implement a resolution conversion process for up-converting the low-resolution video obtained by decoding the L0 bitstream to the resolution of the high-resolution video that has been displayed up until that point, in addition to the decoder, in the receiving device 200, and it is possible to suppress an increase in costs.
[0069] (1.5) First Modification of the First Embodiment In the above-mentioned first embodiment, an example in which L0 and L1 have different resolutions is assumed, but L0 and L1 may have the same resolution. For example, as shown in FIG. 10, the multi-layer encoding unit 110 of the transmitting device 100 may output an L0 bit stream obtained by encoding a first image constituting a program video as L0, and may output an L0 bit stream obtained by encoding a second image in which subcontent (e.g., subtitles) is added to the first image as L1. Here, the first image and the second image have the same resolution. As a result, in the receiving device 200, when only L0 is decoded, the program video can be reproduced, and when L0 and L1 are decoded, the program video with subcontent can be reproduced. In the latter case, even if the L1 bit stream is lost, the decoded video of L0 can be displayed seamlessly.
[0070] (1.6) Second Modification of the First Embodiment In the above-mentioned first embodiment, an example has been described in which, when the reception state determination unit 240 determines that a part of the L1 bit stream is lost, the layer combining unit 220 replaces the bit stream of the lost part with replacement data and then performs the layer combining process, but the present invention is not limited to this. The layer combining unit 220 may also replace the bit stream with replacement data even when the entire L1 bit stream has been received. By performing such an operation, it is possible to use the L1 decoded image (video with sub-content) and the L0 decoded image (video without sub-content) in the sub-content coding as described above for seamless display switching.
[0071] (1.7) Third Modification of the First Embodiment Although not mentioned in the first embodiment, the transmitted bit stream includes an APS (Adaptation Parameter Set). When an APS is transmitted in a missing frame, the missing APS may be referenced in a subsequent B frame (B picture). Therefore, it is preferable to transmit the APS through the transmission path 500A on the L0 side, that is, the transmission path 500 where no packet loss occurs.
[0072] (1.8) Fourth Modification of the First Embodiment Although not mentioned in the first embodiment, when MMT is used as the transmission method, ISOBMFF (ISO Base Media File Format) metadata may be used. In such a case, the packetizer 131 of the transmitting device 100 has a function of generating ISOBMFF metadata, and may multiplex the ISOBMFF metadata as MPU metadata or MFU data into an MMT packet. Here, the ISOBMFF may be CMAF (Common Media Application Format). The CMAF segment (chunk) may include an initialization segment and a media segment. The initialization segment includes global metadata ("moov") required for initializing the decoding function. The media segment includes media data ("mdat") of a video divided into short pieces and local metadata ("moof") corresponding to the media data. In such a case, when replacement data is replaced on the receiving device 200 side, a descriptor indicating the size of the data described in the moof in CMAF may not match the size of the replaced data. Therefore, when replacement data is replaced by the receiving device 200, it is preferable that the receiving device 200 rewrites the data size of the moof.
[0073] (2) Second embodiment The second embodiment will be described with reference to Figs. 11 and 12, focusing mainly on the differences from the first embodiment.
[0074] (2.1) Overview of the second embodiment Since the transmission path 500A is used for transmitting packets corresponding to units constituting the base layer, it is generally assumed that the error tolerance of the transmission path 500A is higher than that of the transmission path 500B. That is, the transmission path 500B (enhancement layer) is more likely to experience a transmission error than the transmission path 500A (base layer).
[0075] The error resilience may vary depending on the transmission characteristics (type of the transmission line, transmission capacity of the transmission line, etc.) of the transmission line 500 itself. The error resilience may vary depending on the strength of error correction applied in the transmission line 500.
[0076] Against this background, the inventors conducted extensive research and noticed that when an error occurs in the enhancement layer, the impact on playback quality varies depending on the location of the error, and discovered that it is possible to improve the playback quality of the enhancement layer even under conditions where the error rate of the transmission path corresponding to the enhancement layer is constant.
[0077] A transmitting device according to a second embodiment includes an encoding unit that encodes a base layer and an enhancement layer that extends the base layer, a first transmitting unit that transmits a first packet via a first transmission path, a second transmitting unit that transmits a second packet via a second transmission path, and a control unit that allocates specific units having a specific importance among the units constituting the enhancement layer to the first transmission path and allocates units other than the specific units to the second transmission path, wherein the first packet includes the units constituting the base layer and the specific units, and the second packet includes the units constituting the enhancement layer other than the specific units.
[0078] A receiving device according to a second embodiment includes a first buffer unit that accumulates a first packet received via a first transmission path, a second buffer unit that accumulates a second packet received via a second transmission path, a decoding unit that decodes a base layer and an enhancement layer that enhances the base layer, and a depacketizing unit that outputs the first packet and the second packet to the decoding unit based on at least one of a sequence number, a packet counter value, an extension counter value, and a timestamp assigned to the first packet and the second packet, wherein at least one of the sequence number, the packet counter value, the extension counter value, and the timestamp are assigned based on the order in which units are output from an encoding unit, and the first packet includes, in addition to units constituting the base layer, a specific unit having a specific importance among units constituting the enhancement layer, and the second packet includes units constituting the enhancement layer other than the specific unit.
[0079] According to the second embodiment, the transmitting device transmits a specific unit having a specific importance among the units constituting the enhancement layer via the first transmission path. With this configuration, even under a condition where the error occurrence rate of the transmission path corresponding to the enhancement layer (i.e., the second transmission path) is constant, the error occurrence rate of the specific unit constituting the enhancement layer can be suppressed to the same level as that of the first transmission path, and the reproduction quality of the enhancement layer can be improved.
[0080] According to the second embodiment, the receiving device outputs the first packet and the second packet to a decoding unit based on at least one of the sequence numbers, the packet counter value, the extension counter value, and the timestamp assigned to the first packet and the second packet. With this configuration, even if a specific unit constituting the enhancement layer is received via the first transmission path, the base layer and the enhancement layer can be appropriately decoded.
[0081] (2.2) Configuration according to the second embodiment The configuration according to the second embodiment will be described.
[0082] (2.2.1) Configuration of the transmitting device 11 is a diagram showing a configuration of a transmission device 100 according to the second embodiment. The transmission device 100 according to the second embodiment includes a multi-layer encoding unit 101, a layer ID separation unit 103, an importance evaluation unit 104, a packetization unit 105, and a transmission unit 132.
[0083] The multi-layer coding unit 101 codes units constituting the base layer and the enhancement layer, similarly to the multi-layer coding unit 110. The units may be referred to as NAL units. The multi-layer coding unit 101 codes the units in a predetermined order.
[0084] The layer ID separation unit 103, like the layer separation unit 120, determines whether a unit constitutes a base layer or an enhancement layer based on a layer ID (e.g., nuh_layer_id) included in the header of the unit. The layer ID separation unit 103 outputs the unit constituting the base layer to the packetizing unit 105, and outputs the unit constituting the enhancement layer to the importance evaluation unit 104.
[0085] The importance evaluation unit 104 evaluates the importance of the units constituting the enhancement layer. Specifically, the importance evaluation unit 104 assigns a specific unit having a specific importance among the units constituting the enhancement layer to the transmission unit 132A (transmission path 500A), and assigns units other than the specific unit to the transmission unit 132B (transmission path 500B).
[0086] The particular importance may be determined according to the options set out below.
[0087] In option 1, the importance assessment unit 104 may assign a unit including a parameter set as a specific unit to the transmitter 132A (transmission path 500A). Specifically, the importance assessment unit 104 may refer to the header of the unit (e.g., NAL unit type (nal_unit_type)) to identify a unit including a parameter set as a specific unit. For example, the importance assessment unit 104 may identify a unit including a Sequence Parameter Set, a Picture Parameter Set, and an Adaptation Parameter Set as a specific unit.
[0088] According to option 1, by suppressing errors in units including parameter sets, it is possible to suppress a situation in which the enhancement layer cannot be reproduced at all even under conditions in which the error rate of transmission path 500B is constant.
[0089] In option 2, the importance evaluation unit 104 may assign a unit constituting the first picture to the transmission unit 132A (transmission path 500A) as a specific unit. Specifically, the importance evaluation unit 104 may specify a unit constituting the first picture as a specific unit in decoding a Structure Of Picture (SOP). The first picture may be considered to be an Intra-coded (I) picture of a Group Of Picture (GOP). Note that, for the first picture of a GOP, the base layer is data of an I picture that can be decoded independently, whereas the enhancement layer (corresponding to a specific unit) that extends this may be data equivalent to a Predictive-coded (P) picture obtained by encoding difference information after inter-layer prediction based on the decoding result of the base layer.
[0090] According to option 2, by suppressing errors in the units constituting the leading picture, it is possible to suppress a situation in which the entire group of pictures referencing the leading picture cannot be played back even under conditions where the error rate of the transmission path 500B is constant.
[0091] In option 3, the importance evaluation unit 104 may assign units constituting a picture that is not referenced by other pictures to the transmission unit 132B (transmission path 500B) as units other than the specific units. Specifically, the importance evaluation unit 104 may identify units constituting a picture that is not referenced by other pictures in decoding the SOP as units other than the specific units. In other words, the importance evaluation unit 104 may identify units constituting a picture that is referenced by other pictures in decoding the SOP as the specific units.
[0092] Option 3 allows for errors in pictures that are not referenced by other pictures, while suppressing degradation of the quality of the enhancement layer.
[0093] In addition, two or more options selected from the above options 1-3 may be combined.
[0094] The packetizer 105, like the packetizer 131, packetizes the units to generate packets. The packets may be packets conforming to the RTP. For example, the packets may be IP packets. The IP packets may include an IP header, a UDP header, an RTP header, and a payload.
[0095] In the second embodiment, the packetizer 105 generates packets corresponding to units constituting the base layer and specific units constituting the enhancement layer. The packetizer 105 outputs the generated packets to the transmitter 132A. On the other hand, the packetizer 105 generates packets corresponding to units other than the specific units constituting the enhancement layer. The packetizer 105 outputs the generated packets to the transmitter 132B.
[0096] In the second embodiment, the packetizer 105 may assign at least one of a sequence number and a timestamp to a packet based on the order in which the units are output from the multi-layer encoder 101.
[0097] For example, the packetizer 105 may have a sequence number counter that counts the sequence number. The sequence number counter may be a counter that is incremented by one each time a packet is generated.
[0098] For example, the packetizer 105 may have a clock counter for adding a time stamp. The clock counter may be a counter that counts up at a constant frequency.
[0099] The packetizer 105 may assign both a sequence number and a time stamp to a packet. In this case, the time stamp may be used to rearrange the packets, and the sequence number may be used to detect a missing packet.
[0100] In the second embodiment, compared to existing transmitting devices, the transmitting device 100 has an importance evaluation unit 104, but since the processing delay of the importance evaluation unit 104 is small, it should be noted that the order of units constituting the packet output from the packetizing unit 105 is the same as the predetermined order in which the units are encoded (output) by the multi-layer encoding unit 101. In other words, it should be noted that the predetermined order in which the units are encoded (output) by the multi-layer encoding unit 101 is not changed by referring to at least one of the sequence number and the timestamp assigned to the packet.
[0101] Although not particularly limited, packetizer 105 may perform a fragmentation process to divide one unit and store it in two or more packets, and packetizer 105 may perform an aggregation process to store two or more units in one packet.
[0102] Similar to the transmitting unit 132, the transmitting unit 132 transmits the packets generated by the packetizing unit 105 via the transmission path 500. Specifically, the transmitting unit 132A transmits the packets generated by the packetizing unit 105A via the transmission path 500A. The transmitting unit 132B transmits the packets generated by the packetizing unit 105B via the transmission path 500B.
[0103] In the second embodiment, the packet transmitted from the transmitting unit 132A may be referred to as a first packet. The first packet includes a unit constituting a base layer and a specific unit. The packet transmitted from the transmitting unit 132B may be referred to as a second packet. The second packet includes a unit other than the specific unit among the units constituting an enhancement layer.
[0104] (2.2.2) Configuration of the receiving device 12 is a diagram showing a configuration of a receiving device 200 according to the second embodiment. The receiving device 200 according to the second embodiment has a receiving unit 201, a buffer unit 203, a comparing unit 205, a depacketizing unit 207, and a multi-layer decoding unit 209.
[0105] The receiving unit 201 receives packets via the transmission path 500. Specifically, the receiving unit 201A receives a first packet via the transmission path 500A. The receiving unit 201B receives a second packet.
[0106] The buffer unit 203 accumulates packets received by the receiving unit 201. The buffer unit 203 may be a FIFO type buffer unit. The buffer unit 203 is used to absorb errors in transmission delay. Specifically, the buffer unit 203A accumulates a first packet received by the receiving unit 201A. The buffer unit 203B accumulates a second packet received by the receiving unit 201B.
[0107] The comparison unit 205 instructs the buffer unit 203 to output the packet stored in the buffer unit 203 based on at least one of the sequence number and the timestamp assigned to the packet stored in the buffer unit 203 .
[0108] For example, comparing unit 205 may instruct buffer unit 203A or buffer unit 203B to output the packet having the smaller sequence number out of the first packet stored in buffer unit 203A and the second packet stored in buffer unit 203B.
[0109] For example, comparison unit 205 may instruct buffer unit 203A or buffer unit 203B to output the packet with the earlier timestamp out of the first packet stored in buffer unit 203A and the second packet stored in buffer unit 203B.
[0110] The depacketizer 207 depacketizes the packets output from the buffer 203 to extract units. Specifically, the depacketizer 207 extracts units constituting the base layer and specific units constituting the enhancement layer from the first packets stored in the buffer 203A. The depacketizer 207 extracts units other than the specific units constituting the enhancement layer from the second packets stored in the buffer 203B.
[0111] Here, since the packets stored in the buffer unit 203 are instructed in the order indicated by the comparison unit 205, the depacketizing unit 207 rearranges the units in a predetermined order similar to that of the multi-layer encoding unit 101, and outputs the units in the same predetermined order as that of the multi-layer encoding unit 101.
[0112] In the second embodiment, since the packets stored in the buffer unit 203 are instructed in the order instructed by the comparison unit 205, the depacketizing unit 207 can output the units in the same predetermined order as the multi-layer encoding unit 101 without analyzing the inside of each unit. Therefore, it should be noted that there is no need to change the circuit or program of the receiving device 200 according to the encoding / decoding method of the units.
[0113] Multi-layer decoding section 209 decodes the units output from depacketizing section 207 in the same predetermined order as multi-layer encoding section 101 .
[0114] In the second embodiment, the packet received by the receiving unit 201A may be referred to as a first packet. The first packet includes a unit constituting a base layer and a specific unit. The packet received by the receiving unit 201B may be referred to as a second packet. The second packet includes a unit other than the specific unit among the units constituting an enhancement layer.
[0115] (2.3) Summary of the second embodiment In the second embodiment, the multi-layer coding unit 101 constitutes a coding unit that codes a base layer and an enhancement layer that enhances the base layer. The transmission unit 132A constitutes a first transmission unit that transmits a first packet via a first transmission path (transmission path 500A). The transmission unit 132B constitutes a second transmission unit that transmits a second packet via a second transmission path (transmission path 500B). The importance evaluation unit 104 constitutes a control unit that distributes a specific unit having a specific importance among units that constitute the enhancement layer to the first transmission path (transmission path 500A) and distributes units other than the specific unit to the second transmission path (transmission path 500B). The packetizing unit 105 constitutes a packetizing unit that generates packets to be transmitted via the first transmission path (transmission path 500A) and the second transmission path (transmission path 500B).
[0116] In the second embodiment, the receiving unit 201A constitutes a first receiving unit that receives a first packet via a first transmission path (transmission path 500A). The receiving unit 201B constitutes a second receiving unit that receives a second packet via a second transmission path (transmission path 500B). The multi-layer decoding unit 209 constitutes a decoding unit that decodes a base layer and an enhancement layer that enhances the base layer. The depacketizing unit 207 constitutes a depacketizing unit that outputs the first packet and the second packet to the decoding unit based on at least one of the sequence numbers and the timestamps assigned to the first packet and the second packet. The depacketizing unit may include a comparing unit 205.
[0117] In the above-mentioned second embodiment, RTP has been mainly described as a transmission method. However, the above disclosure is not limited to this. MMT may be used as a transmission method. The header of an MMT packet may include at least one of a sequence number and a timestamp, similar to RTP.
[0118] In the second embodiment, the transmitting device 100 transmits a specific unit having a specific importance among the units constituting the enhancement layer through the first transmission path. According to this configuration, even under a condition where the error occurrence rate of the transmission path corresponding to the enhancement layer (i.e., the transmission path 500B) is constant, the error occurrence rate of the specific unit constituting the enhancement layer can be suppressed to the same level as that of the transmission path 500A, and the reproduction quality of the enhancement layer can be improved.
[0119] In the second embodiment, the receiving device 200 outputs the first packet and the second packet to the decoding unit based on at least one of the sequence numbers and the timestamps added to the first packet and the second packet. With this configuration, even if a specific unit constituting the enhancement layer is received via the transmission path 500A, the base layer and the enhancement layer can be appropriately decoded.
[0120] (2.4) First Modification of the Second Embodiment A first modification of the second embodiment will be described below, focusing mainly on the differences from the second embodiment.
[0121] In the second embodiment, the case where the transmission method is RTP has been mainly described. In contrast, in the first modified example, the case where the transmission method is MMT will be described.
[0122] First, in the first modification, the value of a packet counter may be used instead of the above-mentioned sequence number. In other words, a packet counter may be used instead of a sequence number counter.
[0123] Specifically, in MMT, the packet counter may be a counter that counts up packets through two or more packet IDs (hereinafter, PIDs).
[0124] In such a case, packetizing unit 105 of transmitting device 100 may assign the value of the packet counter to the packet based on the order in which units are output from multi-layer encoding unit 101. Comparing unit 205 of receiving device 200 may instruct buffer unit 203A or buffer unit 203B to output the packet having the smaller packet counter value, between the first packet stored in buffer unit 203A and the second packet stored in buffer unit 203B.
[0125] In the first modified example, a sequence number may be used together with the packet counter value. In such a case, a different PID may be used for each transmission path 500, and the sequence number may be counted for each PID. The sequence number may be used to detect packet loss for each transmission path 500.
[0126] Secondly, in the first modified example, the value of an extended counter may be used instead of the above-mentioned sequence number. In other words, an extended counter may be used instead of the sequence number counter.
[0127] Specifically, a field for an extended counter may be newly defined in the extended header of the MMT packet. The extended counter is a counter that is used exclusively for rearrangement of units accompanying the introduction of the importance evaluation unit 104.
[0128] In such a case, the packetizer 105 of the transmitting device 100 may assign the value of the extension counter to the packet based on the order in which the units are output from the multi-layer encoding unit 101. The comparator 205 of the receiving device 200 may instruct the buffer unit 203A or the buffer unit 203B to output the packet having the smaller value of the extension counter, between the first packet stored in the buffer unit 203A and the second packet stored in the buffer unit 203B.
[0129] In the first modified example, a sequence number may be used together with the value of the extended counter. In such a case, the PID may be different for each transmission path 500, and the sequence number may be counted for each PID. The sequence number may be used to detect packet loss for each transmission path 500.
[0130] In the first modified example, the packet counter flag may be set to "0" and the packet counter value may not be used together with the extended counter value. Alternatively, in the first modified example, the packet counter flag may be set to "1" and the packet counter value may be used for other purposes together with the extended counter value.
[0131] (2.5) Other Modifications of the Second Embodiment Although not mentioned in the above-mentioned second embodiment, when RTP is used as the transmission method, audio or other media components and video may be multiplexed using a multiplexing method such as MPEG-2 TS (Transport Stream). In such a case, the packetizing unit 105 of the transmitting device 100 has a TS multiplexing function, and may divide one unit into two or more, store the divided units in TS packets, and then store the two or more TS packets in one packet.
[0132] Although not mentioned in the above-mentioned second embodiment, when MMT is used as the transmission method, ISOBMFF (ISO Base Media File Format) metadata may be used. In such a case, the packetizer 105 of the transmitting device 100 has a function of generating ISOBMFF metadata, and may multiplex the ISOBMFF metadata into the MMT packet as MPU metadata or MFU data.
[0133] Here, the ISOBMFF may be CMAF (Common Media Application Format). The CMAF segment (chunk) may include an initialization segment and a media segment. The initialization segment includes global metadata ("moov") required for initializing the decoding function. The media segment includes media data ("mdat") of a video divided into short pieces and corresponding local metadata ("moof"). In such a case, the importance evaluation unit 104 of the transmitting device 100 may allocate a unit including a parameter set of an initialization segment common to the base layer and the enhancement layer to the transmission path 500A as a specific unit. The initialization segment may be multiplexed into an MMT packet as MPU data. The parameter set may be included in the metadata of the initialization segment or in the media data ("mdat") of the media segment.
[0134] Although not mentioned in the above-mentioned second embodiment, the sequence number, the packet counter, and the time stamp may be a 4-byte (32-bit) field. Assuming a case where the 4-byte field is handled as an unsigned integer, the field can express values from 0 to 4294967295. When the sequence number counter overflows, the value of the sequence number counter returns to 0. When the clock counter overflows, the clock counter returns to a value close to 0. Assuming such an overflow, an error in magnitude comparison due to an overflow may be suppressed by detecting the overflow using a method such as virtually holding the state of bits higher than 32 bits. In addition, the extended counter may have any field length, such as 2 bytes (16 bits), other than the case of 4 bytes. In these cases, an overflow may be detected by the same principle to suppress an error in magnitude comparison.
[0135] (3) Other embodiments In the above embodiment, an example has been described in which the bit stream of L0 (base layer) is transmitted through the transmission path 500A, and the bit stream of L1 (enhancement layer) is transmitted through the transmission path 500B, but this is not limiting. Even when these bit streams are transmitted through the same transmission path 500, when data is lost in the bit stream on the L1 side, the loss is detected and replaced with replacement data, so that the decoding process can be continued without initializing the decoding process. Therefore, the bit stream of L0 (base layer) and the bit stream of L1 (enhancement layer) may be transmitted through the same transmission path 500.
[0136] In the above embodiment, VVC is mainly exemplified as an encoding method. However, the above embodiment is not limited to this. As the encoding method, a multi-layer extension of HEVC (High Efficiency Video Coding) may be used. As the encoding method, a method that can identify layers by the header of a unit may be used.
[0137] In the above embodiment, the case of two layers has been mainly described. However, the above embodiment is not limited to this. The above embodiment may be applied to three or more layers. In such a case, a number of transmitting means, transmission paths, and receiving means corresponding to the number of layers may be provided.
[0138] A program may be provided that causes a computer to execute each process performed by the transmitting device 100 or the receiving device 200. The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. In addition, a circuit that executes each process performed by the transmitting device 100 or the receiving device 200 may be integrated, and at least a part of the transmitting device 100 or the receiving device 200 may be configured as a semiconductor integrated circuit (chip set, SoC).
[0139] The terms "based on" and "depending on / in response to" used in this disclosure do not mean "based only on" or "depending only on", unless otherwise specified. The term "based on" means both "based only on" and "based at least in part on". Similarly, the term "depending on" means both "based only on" and "depending at least in part on". The terms "include", "comprise", and variations thereof do not mean including only the recited items, but may include only the recited items or may include additional items in addition to the recited items. Additionally, the term "or" used in this disclosure is not intended to be an exclusive or. Additionally, any reference to elements using designations such as "first", "second", etc., used in this disclosure is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some manner. In this disclosure, where articles are added by translation, such as, for example, a, an, and the in English, these articles are intended to include the plural unless the context clearly indicates otherwise.
[0140] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention.
[0141] (4) Additional Notes Additional notes will be given regarding the features of the above-described embodiment.
[0142] (4.1) Note 1 The first embodiment described above may be expressed as follows.
[0143] A first feature of the first embodiment is a receiving device comprising a first receiving means for receiving a bit stream of a first layer in multi-layer encoding, a second receiving means for receiving a bit stream of a second layer higher than the first layer, a synthesis means for performing a combining process to combine the bit stream of the first layer and the bit stream of the second layer and output a multi-layer bit stream, and a determination means for determining a reception state in the second receiving means, wherein when the determination means determines that a portion of the bit stream of the second layer is missing, the combining means replaces the bit stream of the missing portion with replacement data and then performs the combining process.
[0144] A second feature of the first embodiment is a receiving device which, in addition to the first feature, further comprises a memory means for storing the replacement data used for the replacement in the second layer, and the replacement data includes prediction mode information for generating a predicted image in which the decoded image in the first layer is enlarged to a display resolution in the second layer, and no-residue information indicating that no prediction residual is included for each image block in the second layer.
[0145] A third feature of the first embodiment is a receiving device in which, in the first or second feature, the first receiving means and the second receiving means each perform receiving processing of the bit stream on a frame-by-frame basis, the determination means determines a missing frame in the bit stream of the second layer corresponding to the missing portion as a replacement frame, and the combining means replaces the replacement frame with the replacement data and then performs the combining processing.
[0146] A fourth feature of the first embodiment is that, in the third feature, the receiving device is configured such that the determination means determines the missing frame and a frame that uses the missing frame as a reference image as a replacement frame group, and the combination means replaces the replacement data for each frame that constitutes the replacement frame group and then performs the combination process.
[0147] A fifth feature of the first embodiment is a receiving device in which, in the first or second feature, the first receiving means and the second receiving means each perform receiving processing of the bit stream in chunk units each consisting of a plurality of frames, the determination means determines a missing chunk including the missing portion in the bit stream of the second layer as a replacement chunk, and the combining means replaces the replacement chunk with the replacement data and then performs the combining processing.
[0148] A sixth feature of the first embodiment is a receiving device in which, in the fifth feature, the determination means determines each frame in the missing chunk and frames that use frames in the missing chunk as reference images as a replacement frame group, and the combination means replaces each frame that constitutes the replacement frame group with the replacement data and then performs the combination process.
[0149] A seventh feature of the first embodiment is a receiving device, in any of the first to sixth features, wherein the bit stream of the first layer is transmitted via a first transmission path, and the bit stream of the second layer is transmitted via a second transmission path different from the first transmission path.
[0150] An eighth feature of the first embodiment is a program that causes a computer to function as the receiving device of the first feature.
[0151] (4.2) Appendix 2 The second embodiment described above may be expressed as follows.
[0152] A first feature of the second embodiment is a transmitting device comprising: an encoding unit that encodes a base layer and an enhancement layer that extends the base layer; a first transmitting unit that transmits a first packet via a first transmission path; a second transmitting unit that transmits a second packet via a second transmission path; and a control unit that allocates a specific unit having a specific importance among units constituting the enhancement layer to the first transmission path and allocates units other than the specific unit to the second transmission path, wherein the first packet includes units constituting the base layer and the specific unit, and the second packet includes units constituting the enhancement layer other than the specific unit.
[0153] A second feature of the second embodiment is the transmitting device according to the first feature, in which the control unit assigns a unit including a parameter set as the specific unit to the first transmission path.
[0154] A third feature of the second embodiment is the transmitting device according to the first or second feature, wherein the control unit assigns a unit constituting a leading picture as the specific unit to the first transmission path.
[0155] A fourth feature of the second embodiment is a transmitting device in which, in any one of the first to third features, the control unit allocates units that constitute a picture that is not referenced by other pictures to the second transmission path as units other than the specific units.
[0156] A fifth feature of the second embodiment is a transmitting device which, in any one of the first to fourth features, includes a packetizing unit which generates packets to be transmitted via the first transmission path and the second transmission path, and the packetizing unit assigns at least one of a sequence number, a packet counter value, an extended counter value, and a timestamp to the packets based on the order in which units are output from the encoding unit.
[0157] A sixth feature of the second embodiment is a receiving device comprising: a first buffer unit that accumulates a first packet received via a first transmission path; a second buffer unit that accumulates a second packet received via a second transmission path; a decoding unit that decodes a base layer and an enhancement layer that enhances the base layer; and a depacketizing unit that outputs the first packet and the second packet to the decoding unit based on at least one of a sequence number, a packet counter value, an extended counter value, and a timestamp assigned to the first packet and the second packet, wherein at least one of the sequence number, the packet counter value, the extended counter value, and the timestamp are assigned based on the order in which units are output from an encoding unit; the first packet includes, in addition to units constituting the base layer, a specific unit having a specific importance among units constituting the enhancement layer; and the second packet includes units constituting the enhancement layer other than the specific unit.
[0158] A seventh feature of the second embodiment is the receiving device according to the sixth feature, wherein the first packet includes a unit including a parameter set, among packets constituting the enhancement layer.
[0159] An eighth feature of the second embodiment is a receiving device in which, in the sixth or seventh feature, the first packet includes a unit that constitutes the first picture among packets that constitute the enhancement layer.
[0160] A ninth feature of the second embodiment is a receiving device in which, in at least one of the sixth to eighth features, the second packet includes a unit constituting a picture that is not referenced by other pictures, among packets constituting the enhancement layer. [Explanation of symbols]
[0161] 10: Transmission Systems 100: Transmitting device 101: Multi-layer coding unit 103: Layer ID separation section 104: Importance evaluation section 105: Packetizing section 110: Multi-layer coding unit 120: Layer separation section 130: Transmission method 131: Packetizing section 132: Transmission unit 200: Receiving device 201: Receiving unit 203: Buffer section 205: Comparison section 207: Depacketizing Department 209: Multi-layer decoding unit 210: Receiving means 211: Receiving unit 212: Depacketizing Department 213: Buffer section 220: Layer junction 230: Memory section 240: Reception status determination unit 250: Multi-layer decoding unit 500: Transmission line
Claims
1. A first receiving means for receiving a bit stream of a first layer in multi-layer coding; A second receiving means for receiving a bit stream of a second layer higher than the first layer; a combining means for performing a layer combining process for combining the first layer bit stream and the second layer bit stream and outputting a multi-layer bit stream; a determination means for determining a reception state of the second receiving means, When the determining means determines that a portion of the bit stream of the second layer is lost, the combining means performs the layer combining process after replacing the bit stream of the lost portion with replacement data. Receiving device.
2. a memory means for storing the replacement data used for the replacement in the second layer; The replacement data is prediction mode information for generating a predicted image in which the decoded image in the first layer is enlarged to a display resolution in the second layer; and no-residual information indicating that no prediction residual is included for each image block in the second layer.
2. The receiving device according to claim 1.
3. each of the first receiving means and the second receiving means performs a receiving process of a bit stream on a frame-by-frame basis; the determining means determines a lost frame corresponding to the lost portion in the bit stream of the second layer as a replacement frame; The combining means performs the layer combining process after replacing the replacement data with the replacement frame.
3. The receiving device according to claim 1 or 2.
4. the determining means determines, as a replacement frame group, the lost frame and a frame that uses the lost frame as a reference image; The combining means performs the layer combining process after replacing each frame constituting the replacement frame group with the replacement data.
4. The receiving device according to claim 3.
5. each of the first receiving means and the second receiving means performs a receiving process of a bit stream in chunk units each of which is a group of a plurality of frames; The determining means determines a lost chunk including the lost portion in the bitstream of the second layer as a replacement chunk, The combining means performs the layer combining process after replacing the replacement chunk with the replacement data.
3. The receiving device according to claim 1 or 2.
6. the determining means determines, as a replacement frame group, each frame in the lost chunk and a frame that uses a frame in the lost chunk as a reference image; The combining means performs the layer combining process after replacing each frame constituting the replacement frame group with the replacement data.
6. The receiving device according to claim 5.
7. the first layer bit stream is transmitted via a first transmission path; The bit stream of the second layer is transmitted via a second transmission path different from the first transmission path.
3. The receiving device according to claim 1 or 2.
8. A computer is caused to function as the receiving device according to claim 1. program.
Citation Information
Patent Citations
IEC23090-3、
