Decoding device

Abstract

To improve coding efficiency in hierarchical coding.SOLUTION: An encoding device is a device that performs hierarchical encoding on an input video, and includes a region extraction unit that extracts a main video region included in the input video, a region division unit that divides a remaining video region of the input video other than the main video region into a plurality of sub-pictures, a base hierarchical encoding unit that encodes the main video region output by the region extraction unit and outputs a base hierarchical encoding stream, and an extension hierarchical encoding unit that encodes each of the plurality of sub-pictures output by the region division unit and outputs an extension hierarchical encoding stream.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an encoding device, a decoding device, and a program. [Background technology]

[0002] There are high expectations for services that use large-screen, high-definition video, but high-definition video requires a huge amount of data, so services that involve transmission require broadband transmission paths. When using radio waves for transmission, the frequency band is limited, so technology is used to compress and encode video data according to the transmission band.

[0003] However, compression encoding causes degradation of the image quality, and the higher the compression ratio, the greater the degradation. For this reason, there is a limit to the compression ratio required to obtain the quality required for a service. For example, if you try to broadcast 8K images over terrestrial broadcasting (6MHz transmission bandwidth), you will not be able to obtain sufficient image quality, and the service cannot be provided as is.

[0004] In conditions where sufficient transmission bandwidth cannot be secured, the occurrence of encoding degradation can be suppressed by lowering the video resolution, but in this case the quality of the service itself will be reduced. For example, if 8K video is down-converted to 4K and broadcast, the service will naturally have 4K quality.

[0005] Therefore, it is considered that high quality video can be obtained by separately transmitting complementary information using additional transmission means and using the data sent through these transmission paths together. In other words, 4K service is provided over terrestrial waves with limited transmission bandwidth, and additional information is transmitted over a separate transmission path, which is then combined with the 4K data to obtain 8K video. It is considered that hierarchical coding can be used as a means for realizing such a service form (for example, see Patent Document 1). Note that hierarchical coding is also called scalable coding.

[0006] In hierarchical coding, high quality video can be reproduced by combining the base layer and the enhancement layer, and video can be reproduced using the base layer alone, although the quality will be reduced. Several techniques have been proposed to achieve hierarchical coding. One method for scalably encoding video with different pixel counts (resolutions), such as the aforementioned 4K and 8K, is called spatial hierarchical coding.

[0007] There is also known a technique for hierarchically encoding a video signal from coarse information to fine information (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2017-135522 A [Patent Document 2] JP 2011-244403 A Summary of the Invention [Problem to be solved by the invention]

[0009] In spatial scalable coding, for example, low-resolution 4K video is upconverted to generate 8K video, and additional information for generating 8K video from 4K video is transmitted as an extension layer, resulting in poor prediction accuracy and low coding efficiency. This tendency is particularly noticeable in cases where a high compression rate is required, such as terrestrial broadcasting.

[0010] For this reason, in order to obtain 8K video of sufficient quality using spatial hierarchical coding within a limited transmission bandwidth, the additional information required to obtain 8K will increase compared to 4K. Therefore, it is expected that the transmission path capacity required to obtain the same quality will be almost the same compared to transmitting 4K and 8K video as is without hierarchical coding, and the benefits of adopting hierarchical coding are considered to be small.

[0011] Therefore, although conventional hierarchical coding is useful, there is a problem that coding efficiency is poor under certain conditions.

[0012] Therefore, an object of the present invention is to provide an encoding device, a decoding device, and a program that improve the encoding efficiency in hierarchical encoding. [Means for solving the problem]

[0013] A coding device according to a first aspect is a device that cuts out an input video into a plurality of videos and hierarchically codes the plurality of cut out videos. The coding device includes: a cut-out position designation unit that outputs cut-out control information that designates a position and a size of a main video area included in the input video, a region cut-out unit that cuts out the main video area included in the input video based on the cut-out control information, a region division unit that divides a video area other than the main video area constituting the input video into a plurality of sub-pictures, a base hierarchical coding unit that encodes the main video area output by the region cut-out unit and outputs a base hierarchical coding stream including only the encoded main video, and an enhancement hierarchical coding unit that encodes each of the plurality of sub-pictures output by the region division unit and outputs an enhancement hierarchical coding stream that can be used in combination with the base hierarchical coding stream.

[0014] A decoding device according to a second aspect is a device for decoding an encoded stream obtained by hierarchical encoding. The decoding device includes a base hierarchical decoding unit that decodes a base hierarchical encoded stream and outputs a main video area included in an input video, an extension hierarchical decoding unit that decodes an extension hierarchical encoded stream that can be used in combination with the base hierarchical encoded stream and outputs a plurality of sub-pictures constituting a video area other than the main video area of ​​the input video, a region synthesis unit that synthesizes the sub-pictures output by the extension hierarchical decoding unit and outputs a video area other than the main video area, and a hierarchical synthesis unit that synthesizes the main video area output by the base hierarchical decoding unit and the video area other than the main video area output by the region synthesis unit to restore the input video.

[0015] A program according to a third aspect causes a computer to function as the encoding device according to the first aspect.

[0016] A program according to the fourth aspect causes a computer to function as the decryption device according to the second aspect. Effect of the Invention

[0017] According to the present invention, it is possible to provide an encoding device, a decoding device, and a program for improving the encoding efficiency in hierarchical encoding. [Brief description of the drawings]

[0018] [Figure 1] 1 is a diagram showing a configuration of a video transmission system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of an encoding device according to an embodiment. [Diagram 3] 11A to 11C are diagrams illustrating a method for specifying a cut-out position according to the embodiment. [Figure 4] FIG. 4 is a diagram showing a first division example in an area dividing section according to the embodiment. [Diagram 5] FIG. 11 is a diagram showing a second division example in the region dividing section according to the embodiment. [Figure 6] FIG. 13 is a diagram showing a third example of division in the region dividing section according to the embodiment. [Figure 7] FIG. 2 is a diagram illustrating a configuration of an extension hierarchical encoding unit according to the embodiment. [Figure 8] FIG. 2 is a diagram illustrating a configuration of a decoding device according to an embodiment. [Figure 9] FIG. 13 is a diagram illustrating a configuration of an enhancement layer decoding unit according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] An encoding device and a decoding device according to an embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0020] (Video Transmission System) First, a video transmission system according to an embodiment will be described. Fig. 1 is a diagram showing a configuration of a video transmission system 100 according to an embodiment.

[0021] As shown in Fig. 1, a video transmission system 100 includes a transmitting device 10, a receiving device 20A, and a receiving device 20B. The transmitting device 10 transmits video to the receiving devices 20A and 20B via a transmission path. The transmitting device 10 includes an encoding device 1 that performs hierarchical encoding on input video. In the following, a case where the video to be transmitted is an 8K video will be illustrated, but it is not necessarily limited to 8K.

[0022] The encoding device 1 outputs a base hierarchically encoded stream and an extended hierarchically encoded stream by hierarchical encoding. In the example shown in Fig. 1, the transmitting device 10 transmits the base hierarchically encoded stream to the receiving devices 20A and 20B via a broadcast transmission path 30. For example, the broadcast transmission path 30 is a transmission path that uses terrestrial waves. The transmitting device 10 provides a 4K video transmission service via terrestrial waves with a limited transmission band.

[0023] Moreover, the transmitting device 10 transmits the enhanced hierarchical encoded stream to the receiving device 20B via a communication transmission path 40. For example, the communication transmission path 40 is a wired communication line such as the Internet. The transmitting device 10 separately transmits the enhanced hierarchical encoded stream as complementary information using an additional transmission means.

[0024] The receiving device 20A receives the base hierarchical encoded stream via the broadcast transmission path 30. The receiving device 20A has a decoding device 2A that decodes 4K video from the base hierarchical encoded stream. The receiving device 20A is able to play back video from the base hierarchical encoded stream, although the quality is reduced.

[0025] The receiving device 20B receives the base layer encoded stream via a broadcast transmission path 30, and also receives the extension layer encoded stream via a communication transmission path 40. The receiving device 20B has a decoding device 2B that combines the base layer encoded stream and the extension layer encoded stream to decode 8K video. The receiving device 20B is capable of playing back high-quality video by combining the base layer and the extension layer.

[0026] In this way, the video transmission system 100 transmits 4K and 8K video by hierarchical coding. In the case of a large-screen, high-definition video service such as 8K, it is considered that viewers do not keep their eyes evenly on the entire screen, but usually watch while concentrating their attention on a certain part of the screen.

[0027] Taking this into consideration, when generating 4K images from 8K, there is no need to reduce the entire screen; by cropping the 4K image (without reducing it) to include the focal points where important information is concentrated, it is possible to convey sufficient information.

[0028] The encoding device 1 of the transmission device 10 according to the embodiment generates 4K video by extracting a main video area (hereinafter referred to as the "base layer area") from an 8K video, uses the extracted 4K base layer area as the base layer in hierarchical encoding, and transmits the remaining area (hereinafter referred to as the "base layer area") as an enhancement layer. This makes it possible to provide 8K video services without reducing encoding efficiency while providing 4K video services.

[0029] Here, the position where the 4K base layer region is cut out may be fixed or may be changed for each frame. Also, although an example of a base layer region in which a 4K image is cut out from an 8K image is shown, the size of the base layer region may be changed to suit the performance of a device that displays the base layer.

[0030] The decoder 2A of the receiving device 20A decodes the base layer decoded stream to obtain 4K base layer video. Meanwhile, the decoder 2B of the receiving device 20B decodes the base layer region from the base layer stream and decodes the enhancement layer region from the enhancement layer stream, and synthesizes the base layer region and the enhancement layer region to obtain 8K video. In this way, 4K video and 8K video can be obtained.

[0031] Therefore, the video transmission system 100 according to the embodiment is expected to realize diverse and high-quality services by enabling both 4K (base layer) and 8K (base layer + enhancement layer) video without reducing encoding efficiency. Using this, 4K and 8K hierarchical services can be realized by broadcasting 4K on narrow-band terrestrial waves and transmitting additional information via wired communication such as the Internet.

[0032] (encoding device) Next, a coding device according to an embodiment will be described. Fig. 2 is a diagram showing a configuration of a coding device 1 according to an embodiment.

[0033] As shown in FIG. 2, the encoding device 1 includes an extraction position designation unit 11, a region extraction unit 12, a base hierarchical encoding unit 13, a region division unit 14, and an enhancement hierarchical encoding unit 15.

[0034] The cut-out position designation unit 11 outputs cut-out control information that designates the position and size of the base hierarchical layer region to the region cut-out unit 12, the region division unit 14, and the extended hierarchical coding unit 15. When determining the cut-out position, it is desirable to take care at the time of program production that important information is included in the base hierarchical layer region. Alternatively, it is also possible to designate the cut-out position using a separate means for determining an area to be cut out as important information from the 8K video.

[0035] Fig. 3 is a diagram showing a method for specifying a cut-out position according to an embodiment. As shown in Fig. 3, the cut-out position specification unit 11 specifies the position and size of a 4K basic layer region to be cut out from an 8K input video as cut-out control information. For example, the cut-out position specification unit 11 outputs the coordinate values ​​(x, y) of the top left corner of the basic layer region and the number of horizontal and vertical pixels (w, h) as the cut-out control information.

[0036] Here, the cut-out position may be fixed or may be changed according to the contents of the video content. The coordinate values ​​may be specified by inputting values ​​determined at the time of content creation, or may be specified by a method of automatically detecting and specifying from the input video. For example, the cut-out position specification unit 11 may specify a basic hierarchical region to be cut out as important information from the 8K video by face recognition processing or the like on the input video. The size of the basic hierarchical region to be cut out may also be changed according to the contents of the video content.

[0037] Also, while an example is shown in which the base layer area is arranged so that the long side of the 4K base layer area is parallel to the long side of the 8K input video, this does not necessarily have to be parallel, and the base layer area may be tilted relative to the horizontal direction. However, if they are not parallel, the division method and encoding process of the enhancement layer, which will be described later, become more complicated.

[0038] The area cut-out unit 12 cuts out a base hierarchical layer area from the input video based on the cut-out control information output by the cut-out position designation unit 11, i.e., the coordinate values ​​(x, y) and the area size (w, h), and outputs the cut-out base hierarchical layer area to the base hierarchical layer encoding unit 13 and the area division unit 14.

[0039] The base layer coding unit 13 codes the base layer region output by the region extraction unit 12, and outputs a base layer coded stream that can be used alone. The coding method used by the base layer coding unit 13 may be any method.

[0040] Based on the cut-out control information output by the cut-out position designation unit 11, the region division unit 14 divides the extended hierarchical region other than the base hierarchical region in the input video into regions of shapes that can be encoded by the extended hierarchical encoding unit 15. Each of these divided regions is called a "sub-picture." Considering that a typical encoding method uses a rectangular region as input for encoding, the shape of each sub-picture is assumed to be rectangular. The division method may be determined in advance, or may be changed according to the content, etc., and the division method may be conveyed to the decoding side by a method such as multiplexing the stream each time.

[0041] FIG. 4 is a diagram showing a first division example in the region dividing section 14 according to the embodiment.

[0042] As shown in Fig. 4, in the first division example, the extended hierarchical area other than the base hierarchical area is divided into four sub-pictures by extending the vertical sides of the base hierarchical area. The divided rectangular areas, i.e., the sub-pictures, are A, B, C, and D. The shape (horizontal and vertical pixel count) of each of the sub-pictures A, B, C, and D is uniquely determined once the values ​​(x, y, w, h) specified as the crop control information are determined. In other words, if the division method of "dividing by extending the vertical sides" is specified, it is possible to restore the original image using only the order of the sub-pictures A, B, C, and D and the (x, y, w, h) information.

[0043] FIG. 5 is a diagram showing a second division example in the region dividing section 14 according to the embodiment.

[0044] 5, in the second division example, the area other than the base layer area is divided into four sub-pictures by extending the top side of the base layer area to the left, extending the left side of the base layer area to the bottom, extending the bottom side of the base layer area to the right, and extending the right side of the base layer area to the top. In this case as well, as long as the division method is specified in advance, restoration is possible using only the values ​​(x, y, w, h) specified as the clipping control information, as in the first division example.

[0045] FIG. 6 is a diagram showing a third division example in the region dividing section 14 according to the embodiment.

[0046] As shown in Fig. 6, in the third division example, the enhancement hierarchical region is divided into smaller regions to obtain multiple sub-pictures. For example, if the enhancement hierarchical region can be divided into small regions of the same shape (size), it has the advantage that all sub-encoders in the enhancement hierarchical encoding process described below can be the same. However, it is necessary to limit (x, y, w, h) to values ​​that allow the enhancement hierarchical region to be divided into an integer number of identical small regions both horizontally and vertically, without any excess or deficiency.

[0047] The extension hierarchical coding unit 15 codes each of the multiple sub-pictures output by the region division unit 14, and outputs an extension hierarchical coding stream. Fig. 7 is a diagram showing a configuration of the extension hierarchical coding unit 15 according to an embodiment. As shown in Fig. 7, the extension hierarchical coding unit 15 has multiple sub-encoders 151a, 151b, 151c, etc., and a stream synthesis unit 152.

[0048] The multiple sub-encoders 151a, 151b, 151c,... respectively encode the multiple sub-pictures output by the region division unit 14, and output the multiple encoded streams to the stream synthesis unit 152. In the examples of Fig. 4 and Fig. 5, since there are four sub-pictures A, B, C, and D, it is sufficient to use four sub-encoders.

[0049] The stream synthesis unit 152 synthesizes the multiple encoded streams output by the multiple sub-encoders 151a, 151b, 151c, etc., with the cut-out control information output by the cut-out position designation unit 11, and outputs an extended hierarchical encoded stream. Any method of synthesis may be used, but the method should be such that it is possible to determine the cut-out control information and which sub-picture the stream corresponds to and separate the stream.

[0050] In this way, the encoding device 1 according to the embodiment is a device that performs hierarchical encoding on an input video, and includes an area extraction unit 12 that extracts a main video area (base hierarchical area) included in the input video, an area division unit 14 that divides the remaining video area (extension hierarchical area) other than the main video area of ​​the input video into multiple sub-pictures, a base hierarchical encoding unit 13 that encodes the main video area output by the area extraction unit 12 and outputs a base hierarchical encoded stream that can be used alone, and an extension hierarchical encoding unit 15 that encodes each of the multiple sub-pictures output by the area division unit 14 and outputs an extension hierarchical encoded stream that can be used in combination with the base hierarchical encoded stream. This makes it possible to realize a variety of high-quality services by making it possible to coexist with 4K (base hierarchical layer) and 8K (base hierarchical layer + extension hierarchical layer) video without reducing encoding efficiency.

[0051] (Decoding device) Next, a decoding device according to an embodiment will be described. Fig. 8 is a diagram showing a configuration of a decoding device 2 according to an embodiment. Here, as the decoding device 2, the configuration of the decoding device 2B shown in Fig. 1 is shown.

[0052] As shown in FIG. 8, the decoding device 2 includes a base layer decoding unit 21, an enhancement layer decoding unit 22, a region synthesis unit 23, and a layer synthesis unit 24.

[0053] The base layer decoding unit 21 decodes the base layer encoded stream and outputs the base layer region. When the decoding device 2 is the decoding device 2A shown in Fig. 1, only the base layer region, which is a 4K video, is reproduced.

[0054] The extension layer decoding unit 22 decodes the extension layer encoded stream and outputs a plurality of sub-pictures constituting an extension layer area and cut-out control information. Fig. 9 is a diagram showing a configuration of the extension layer decoding unit 22 according to an embodiment. As shown in Fig. 9, the extension layer decoding unit 22 has a stream separation unit 221 and a plurality of sub-decoders 222a, 222b, 222c, etc.

[0055] The stream separation unit 221 separates the enhanced hierarchical encoded stream into a plurality of encoded streams corresponding to a plurality of sub-pictures and cut-out control information, and outputs the plurality of encoded streams to a plurality of sub-decoders 222a, 222b, 222c, etc., and outputs the cut-out control information to the region synthesis unit 23 and the hierarchical synthesis unit 24. Here, the method of separating the streams and outputting them to the corresponding sub-decoders corresponds to the synthesis method used by the stream synthesis unit 152 of the encoding device 1.

[0056] The plurality of sub-decoders 222a, 222b, 222c, . . . respectively decode the plurality of sub-pictures from the plurality of coded streams output by the stream separation unit 221, and output the decoded sub-pictures to the region synthesis unit .

[0057] The region synthesis unit 23 synthesizes multiple sub-pictures output by the extension hierarchical decoding unit 22 (multiple sub-decoders 222a, 222b, 222c, etc.) and outputs an extension hierarchical region. The synthesis method corresponds to the division method used by the region division unit 14 of the encoding device 1. As described above, the division method used by the encoding device 1 may be determined in advance, or may be changed according to the content, etc., and communicated each time by multiplexing the method into the stream, for example. The synthesis method is determined by the sub-picture division method and the clipping control information, so the region synthesis unit 23 synthesizes multiple sub-pictures accordingly and outputs an extension hierarchical region.

[0058] Based on the cut-out control information output by the extension layer decoding unit 22, the layer synthesis unit 24 synthesizes the base layer region output by the base layer decoding unit 21 and the extension layer region output by the region synthesis unit 23, and restores and outputs the 8K video.

[0059] Thus, the decoding device 2 according to the embodiment is a device for decoding an encoded stream obtained by hierarchical encoding, and includes a base layer decoding unit 21 that decodes a base layer encoded stream that can be used alone and outputs a main video area (base layer area) included in an input video, an extension layer decoding unit 22 that decodes an extension layer encoded stream that can be used in combination with the base layer encoded stream and outputs a plurality of sub-pictures constituting a remaining video area (extension layer area) other than the main video area of ​​the input video, a region synthesis unit 23 that synthesizes a plurality of sub-pictures output by the extension layer decoding unit 22 and outputs the remaining video area, and a layer synthesis unit 24 that synthesizes the main video area output by the base layer decoding unit 21 and the remaining video area output by the region synthesis unit 23 to restore the input video. This makes it possible to realize a variety of high-quality services by making it possible to provide both 4K (base layer) and 8K (base layer + extension layer) video without reducing encoding efficiency.

[0060] (Other embodiments) In the embodiment, an example of configuring hierarchical coding as two layers using a base layer and an enhancement layer has been described, but it is also possible to use the method according to the embodiment recursively to configure three or more layers, or to combine it with other hierarchical coding methods.

[0061] Information on the area that the viewer pays attention to may be fed back from the receiving device 20 (decoding device 2) to the transmitting device 10 (encoding device 1) using the communication transmission path 40. In this case, the cut-out position designation unit 11 may determine the base hierarchical area to be cut out as important information based on the fed-back information.

[0062] In the embodiment, an example has been described in which one base layer area is extracted from the input video, but the area extraction unit 12 may extract two or more base layer areas from the input video. The receiving device 20 may select one base layer area to decode from the two or more base layer areas, and decode only the selected base layer area.

[0063] A program may be provided that causes a computer to execute each process performed by the encoding device 1. Also, a program may be provided that causes a computer to execute each process performed by the decoding device 2. The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

[0064] The circuits for executing the processes performed by the encoding device 1 may be integrated, and the encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC). The circuits for executing the processes performed by the decoding device 2 may be integrated, and the decoding device 2 may be configured as a semiconductor integrated circuit (chip set, SoC).

[0065] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention. [Explanation of symbols]

[0066] 1: Encoding device 2: Decryption device 2A: Decryption device 2B: Decryption device 10: Transmitting device 11: Cut-out position specification section 12: Region extraction section 13: Basic layer encoding section 14: Area division part 15: Enhanced hierarchical coding unit 20: Receiving device 20A: Receiving device 20B: Receiving device 21: Base layer decoding unit 22: Enhancement layer decoding unit 23: Area synthesis section 24: Hierarchical synthesis section 30: Broadcast transmission channel 40: Communication transmission line 100: Video transmission system 151a: Sub-encoder 151b: Sub-encoder 151c: Sub-encoder 152: Stream synthesis unit 221: Stream separation unit 222a: Sub-decoder 222b: Sub-decoder 222c: Sub-decoder

Claims

1. a first decoding unit that decodes the encoded stream and outputs a first video region included in the input video; a second decoding unit that decodes an encoded stream that can be used in combination with the encoded stream, and outputs one or more sub-pictures that constitute an image area of ​​the input video other than the first image area; an acquisition unit that acquires control information that specifies a position and a size of the first video area; a synthesis unit that synthesizes the first video area output by the first decoding unit and the one or more sub-pictures output by the second decoding unit based on the control information to restore the input video, A decoding device, characterized in that the control information specifying the size is limited in accordance with division into sub-pictures of the same size on the encoding side.

2. a first decoding unit that decodes the encoded stream and outputs a first video region included in the input video; a second decoding unit that decodes an encoded stream that can be used in combination with the encoded stream, and outputs one or more sub-pictures that constitute an image area of ​​the input video other than the first image area; an acquisition unit that acquires control information that specifies a position and a size of the first video area; a synthesis unit that synthesizes the first video area output by the first decoding unit and the one or more sub-pictures output by the second decoding unit based on the control information to restore the input video, A decoding device characterized in that the size of the subpicture indicated by the control information is limited so that the input video can be divided into an integer number of identical subpictures, without any excess or deficiency, in response to division into subpictures of the same size on the encoding side.

Images