Image conversion device and image decoding device

Abstract

To provide an image decoding device that improves a processing efficiency of a parallel processing tool.SOLUTION: In an image decoding device, a conversion part that converts a coding picture containing a slice including an integer number of rectangular regions in each of which an image is divided into blocks of a predetermined size and one of more rows of blocks are collected, converts the coding picture so that a relation between the slice contained in the coding picture and the rectangular region is reconstructed if the coding picture is the same in a slice boundary and a rectangular region boundary in regard to an adaption of a loop filter, and converts the coding picture so that the relation between the slice contained in the coding picture and the rectangular region is not changed if the coding picture is not the same in the slice boundary and the rectangular region boundary in regard to the adaption of the loop filter.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image encoding and decoding technique capable of parallel processing. [Background technology]

[0002] HEVC defines tile and entropy synchronous coding as tools for parallel processing. Improvements to parallel processing tools are being considered for VVC (Versatile Video Coding), the next-generation video coding standard being formulated by JVET (Joint Video Expert Team). [Prior art documents] [Non-patent literature]

[0003] Versatile Video Coding (Draft 6) Summary of the Invention

[0004] The technology in Non-Patent Document 1 is still in the process of being standardized, and there are issues with the processing efficiency of parallel processing tools.

[0005] In order to solve the above problem, an image conversion device of one embodiment of the present invention is an image conversion device that divides an image into blocks of a predetermined size, and converts a coded picture including a slice including an integer number of rectangular areas each including one or more block rows of the blocks, and if the coded picture applies a loop filter at the same time at slice boundaries and rectangular area boundaries, converts the coded picture so as to reconstruct the relationship between the slices and the rectangular areas included in the coded picture, and if the coded picture applies a loop filter at the same time at slice boundaries and rectangular area boundaries, converts the coded picture so as not to change the relationship between the slices and the rectangular areas included in the coded picture.

[0006] Another aspect of the present invention is an image decoding device that divides an image into blocks of a predetermined size, decodes a code string of a slice including an integer number of rectangular areas each including one or more rows of the blocks, the code string of the slice being included in an NAL unit, and decodes an entry point indicating a start byte position of the second or subsequent rectangular areas included in the slice from an NAL unit other than the NAL unit including the slice.

[0007] Yet another embodiment of the present invention is also an image decoding device that divides an image into blocks of a predetermined size, decodes a code string of a slice including an integer number of rectangular areas each including one or more rows of the blocks, and when a flag indicating that the slice may include two or more rectangular areas indicates that the slice may include two or more rectangular areas, decodes a flag indicating whether or not to include in a slice header an entry point indicating a start byte position of the second or subsequent rectangular areas included in the slice, from a picture parameter set that sets parameters of a picture.

[0008] According to the present invention, a parallel processing tool can be realized with high efficiency. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example in which pictures in an encoded stream are divided into bricks. [Diagram 3] A diagram explaining the syntax of a PPS regarding allocation information of CTUs to tiles and allocation information of CTU rows to bricks. [Figure 4] FIG. 1 is a diagram illustrating an example of a brick index. [Diagram 5] A diagram explaining part of the syntax of a PPS. [Figure 6] A diagram explaining the syntax of a portion of the slice header. [Figure 7] 13 is a flowchart illustrating an operation of a conversion unit. [Figure 8] A diagram showing the syntax of the SEI message. [Figure 9] A diagram showing the syntax of the SEI payload. [Figure 10] FIG. 1 is a diagram illustrating the syntax of an entry point SEI message. [Figure 11] 11 is a diagram illustrating a coded stream output by a conversion unit. FIG. [Figure 12] FIG. 13 is a diagram for explaining a part of the syntax of a PPS according to a modification of the first embodiment. [Figure 13] FIG. 11 is a diagram illustrating a configuration of a second embodiment. [Figure 14] FIG. 2 is a diagram illustrating an example of a hardware configuration of a coding / decoding device according to a first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The embodiment of the present invention is based on the technology considered in VVC (Versatile Video Coding), which is a next-generation video coding standard being formulated by JVET (Joint Video Expert Team). In VVC, similar to conventional AVC and HEVC, technologies such as block division, intra prediction, inter prediction, transform, quantization, entropy coding, and loop filter are used.

[0011] The following defines the techniques and technical terms used in the embodiments of the present invention. Hereinafter, unless otherwise specified, the terms image and picture are used interchangeably. In the embodiments of the present invention, the image encoding unit performs encoding, and the image decoding unit performs decoding.

[0012] <Coding tree unit and coding block> A picture is equally divided into blocks of a predetermined size. These blocks of a predetermined size are defined as coding tree blocks (CTBs). Recursive division is possible inside each coding tree block. The block to be coded / decoded after coding tree block division is defined as a coding block (CU). Intra prediction and inter prediction are coded (decoded) in coding block units. Moreover, coding tree blocks and coding blocks are collectively defined as blocks. Efficient coding (decoding) is possible by performing appropriate block division. The size of the coding tree block can be a fixed value previously agreed upon between the encoder and decoder, or a configuration can be adopted in which the size of the coding tree block determined by the encoder is transmitted to the decoder.

[0013] In addition, in a picture, one luma coding tree block and two chroma coding tree blocks are defined as a coding tree unit (CTU) as a coding unit. There are CTUs that use one luma coding tree block as a unit and CTUs that use coding tree blocks of three color planes as a unit. The size of the CTU (CTB) is coded (decoded) by the SPS (Sequence Parameter Set).

[0014] <Tiles and Bricks> A tile is a rectangular area in a picture that is made up of multiple CTUs. A brick is a rectangular area in a tile that is made up of one or more CTU rows. A CTU does not belong to multiple tiles, but only to one tile. A CTU does not belong to multiple bricks, but only to one brick. A brick does not belong to multiple tiles, but only to one tile. The allocation of CTUs to tiles and the allocation of CTU rows to bricks are set on a picture-by-picture basis. The information on the allocation of CTUs to tiles and the information on the allocation of CTU rows to bricks are coded (decoded) by PPS (Picture Parameter Set). As described above, a picture is divided into multiple tiles whose areas do not overlap. Furthermore, tiles are divided into multiple bricks whose areas do not overlap. Hereinafter, a rectangular area in a picture composed of multiple CTUs will be referred to as a brick, which means a rectangular area composed of one or more rows of CTUs in a tile, in which a single CTU does not belong to multiple tiles, but only to one tile, and a single CTU does not belong to multiple rectangular areas, but only to one rectangular area.

[0015] With the parameter information of the SPS, which sets the parameters of the encoded stream, and the PPS, which sets the parameters of the encoded picture, and the slice header, each encoded brick can be decoded independently. Therefore, with the SPS, PPS, and slice header, multiple encoded bricks can be simultaneously encoded (decoded) in parallel. An encoded brick is defined as a code string of a brick generated by encoding a brick. In the following, an encoded picture is defined as a code string of a picture generated by encoding a picture, and an encoded slice is defined as a code string of a slice generated by encoding a slice.

[0016] Figure 3 is a diagram for explaining the syntax of PPS regarding the CTU allocation information to tiles and the CTU row allocation information to bricks. The brick_splitting_present_flag is a flag indicating whether to split a tile into two or more bricks. If the brick_splitting_present_flag is 1, one or more tiles are split into two or more bricks. If the brick_splitting_present_flag is 0, all tiles are not split into two or more bricks. If the brick_splitting_present_flag is 1, the splitting information to bricks is encoded (decoded) for each tile.

[0017] When a tile is not split into bricks, one tile is treated as one brick. The tile index is an index assigned in raster scan order to the tiles in the picture. The brick index is an index assigned to the bricks in the picture. When the picture is split into a plurality of tiles, the brick index is an index assigned in raster scan order to each brick within the tile in tile index order with respect to the tiles in the picture. The brick index is an integer value that is 0 or more and less than the number of bricks (NumBricksInPic) in the picture. The raster scan order is the order of first scanning horizontally, then moving vertically and scanning horizontally again. Figure 4 is a diagram for explaining an example of the brick index. Figure 4(a) shows an example of 8 bricks in the picture, and Figure 4(b) shows an example of 5 bricks in the picture.

[0018] <CTU processing order> CTUs are processed in raster scan order within a brick, and bricks are processed in raster scan order within a tile.

[0019] <Slice and slice mode> A slice is an area that groups together an integer number of bricks. There are two modes for grouping blocks in a slice: rectangular slice mode and raster scan slice mode. Figure 5 is a diagram for explaining part of the syntax of the PPS. The rectangular slice mode and raster scan slice mode are explained using Figure 5. If a flag (single_brick_per_slice_flag) indicating that a slice may contain two or more bricks indicates that the slice may contain two or more bricks (single_brick_per_slice_flag is 0), rect_slice_flag is encoded (decoded). If a slice contains only one brick (single_brick_per_slice_flag is 1), rect_slice_flag is 0. If rect_slice_flag is 1, the rectangular slice mode is used, and if rect_slice_flag is 0, the raster scan slice mode is used.

[0020] <Rectangle slice mode> In rectangular slice mode, bricks in a rectangular area are grouped together as a slice. In rectangular slice mode, the order of processing bricks is not raster scan order, so the correspondence between slices and bricks cannot be uniquely determined. Therefore, the correspondence between slices and bricks can be obtained by encoding (decoding) information indicating the correspondence between slices and bricks using PPS.

[0021] The number of slices in a picture is obtained by subtracting 1 from the number of slices in the picture, and num_slices_in_pic_minus1 is coded (decoded) in the PPS to obtain the number of slices in the picture. The bottom_right_brick_idx_delta and brick_idx_delta_sign_flag, which indicate the bottom right brick index included in each slice in the picture, are coded (decoded) in the PPS to derive the correspondence between slices and bricks in the picture.

[0022] As described above, in the rectangular slice mode, the information indicating the correspondence between slices and bricks is coded (decoded) in PPS, which allows freedom in the order of processing bricks. For example, as shown in Fig. 4(b), by assigning the large screen to the tile to which bricks 0 and 1 belong and the small screen to the other bricks, the processing order of the large screen can be prioritized.

[0023] <Raster scan slice mode> In the raster scan slice mode, bricks are sliced ​​in raster scan order. Therefore, in the raster scan slice mode, it is not necessary to code (decode) the correspondence between the positions of the bricks and the slices in PPS. In the raster scan slice mode, the information indicating the correspondence between the slices and the bricks is not coded (decoded) in PPS, so that the correspondence between the slices and the bricks can be made more flexible. For example, in FIG. 4(b), the number of bricks included in the slice can be adjusted during encoding, such as whether the slice is composed of only brick 0 or of two, brick 0 and brick 1. This makes it possible to adjust the amount of code per slice.

[0024] <Slice ID> The slice ID is an ID to uniquely identify a slice. In raster scan slice mode, the slice ID of the ith slice[i] in raster scan order is i, and the slice ID is a value that is 0 or more and num_slices_in_pic_minus1 or less. In rectangular slice mode, the slice ID can also be explicitly coded (decoded) as an arbitrary value in the PPS. In rectangular slice mode, if the slice ID is not explicitly coded (decoded) in the PPS, the slice ID of the ith slice[i] in raster scan order is i, and the slice ID is a value that is 0 or more and num_slices_in_pic_minus1 or less.

[0025] <Loop filter boundary control flag> Here, the boundary control flag of the loop filter will be explained using the syntax in Fig. 5. loop_filter_across_bricks_enabled_flag is a flag indicating whether or not to apply the loop filter to a brick boundary; if it is 1, the loop filter is applied, and if it is 0, the loop filter is not applied. loop_filter_across_slices_enabled_flag is a flag indicating whether or not to apply the loop filter to a slice boundary; if it is 1, the loop filter is applied, and if it is 0, the loop filter is not applied. When a brick boundary and a slice boundary overlap, if both flags are 1, the loop filter is applied, and if either or both flags are 0, the loop filter is not applied.

[0026] <slice header> Next, the slice header will be described. Fig. 6 is a diagram for explaining a part of the syntax of the slice header. If the mode is rectangular slice mode or NumBricksInPic, which is the number of bricks in a picture, is greater than 1, the slice address (slice_address) is coded (decoded). If the mode is raster scan slice mode and single_brick_per_slice_flag is 0, num_bricks_in_slice_minus1, which indicates the number of slices included in the slice, is coded (decoded).

[0027] In the raster scan slice mode, the slice address indicates the first brick ID included in the slice. The brick ID is an ID assigned to the CTU included in each brick in the picture. CTUs that belong to the same brick have the same brick ID. The brick ID is an integer value that is equal to or greater than 0 and less than the number of bricks in the picture (NumBricksInPic). In the raster scan slice mode, the position of the brick in the picture is defined by the PPS, so the start position of the slice in the picture can be determined most quickly by setting the slice address to the first brick ID included in the slice. The bit length of slice_address is Ceil(Log2(NumBricksInPic)), and slice_address is a value that is equal to or greater than 0 and less than or equal to (NumBricksInPic-1). Here, in the raster scan slice mode, the information indicating the correspondence between slices and bricks is not coded (decoded) in the PPS, so the slice address is set to the first brick ID included in the slice. In raster scan slice mode, the bricks are included in a slice consecutively in raster scan order, so if you know the first brick ID and the number of slices included in the slice, you can uniquely determine the relationship between slices and bricks. Note that Ceil(x) is a function that takes the smallest integer value equal to or greater than x.

[0028] In the rectangular slice mode, information indicating the correspondence between slices and bricks is coded (decoded) in the PPS. Therefore, the brick ID can be derived from the slice ID, and the slice address indicates the slice ID. Here, the slice address is set to the slice ID in the rectangular slice mode, but it may be set to the first brick ID included in the slice as in the raster scan slice mode. By setting the slice address to the first brick ID included in the slice as in the raster scan slice mode in the rectangular slice mode as well, extra circuits and modules are not required, and the memory amount and circuit scale of the encoder and decoder can be reduced. In addition, if the signalled_slice_id_flag of the PPS is 0, management of the slice ID is not required, and the memory amount of the encoder and decoder can be further reduced.

[0029] <entry point> Next, the start byte position of the coding brick in the slice data will be described with reference to FIG. 6. As shown in FIG. 6, the start byte position of the coding brick in the slice data is coded (decoded) in the slice header as an entry point (entry_point_offset_minus1[i]), so the decoder can obtain the start byte position of the coding brick by decoding the slice header. Here, NumEntryPoints is a value obtained by subtracting 1 from the number of bricks in the slice. The start byte position of the first coding brick in the slice data is 0, which is self-evident, so it is not coded (decoded), and the start byte positions of the second and subsequent coding bricks in the slice data are coded (decoded). entry_point_offset_present_flag is a flag that determines whether or not to code (decode) the entry point in the slice header. If entry_point_offset_present_flag is 1, the entry point is coded (decoded), and if entry_point_offset_present_flag is 0, the entry point is not coded (decoded). offset_len_minus1 is a value indicating the bit length of entry_point_offset_minus1. As shown in FIG. 5, entry_point_offset_present_flag is coded (decoded) in PPS when single_tile_in_pic_flag is 0 or entropy_coding_sync_enabled_flag is 1. Here, single_tile_in_pic_flag is a flag indicating whether or not a picture is one tile. If single_tile_in_pic_flag is 1, the picture is composed of one tile, and if single_tile_in_pic_flag is 0, the picture is composed of multiple tiles. When a picture is composed of one tile, it is not permitted for the tile to be composed of multiple bricks, and one tile is composed of one brick. entropy_coding_sync_enabled_flag is a flag indicating whether or not parallel processing of CTU columns in a brick is permitted.If entropy_coding_sync_enabled_flag is 1, parallel processing of CTU columns within a brick is allowed, and if entropy_coding_sync_enabled_flag is 0, parallel processing within a brick is not allowed.

[0030] <sei> Supplemental enhancement information (SEI) is information that is not necessary for the process of generating pixels of a decoded image, but is necessary for the operation of a decoder. For example, SEI includes Buffering Period SEI, which is information for accurately operating a hypothetical reference decoder (HRD), and Picture Timing SEI, which determines the output timing of a decoded image. SEI is encoded (decoded) as an SEI message.

[0031] <NALユニット> An NAL unit consists of a header and a payload. The payload type is coded (decoded) in the header, and the payload contains a code sequence of the type indicated by the payload type. Payload types include SPS, PPS, coded slice, SEI message, etc. For example, if the payload type indicates that a slice is included, the slice code sequence is stored in the payload, and if the payload type indicates that an SEI message is included, the SEI message code sequence is stored in the payload.

[0032] <Encoded stream> In VVC and HEVC, the SPS, PPS, coded slices, SEI messages, etc. are stored in the NAL unit to form a coded stream. A coded slice consists of a slice header and slice data. A coding tree block is coded into the slice data. A coded stream contains one or more coded pictures.

[0033] <hrd> HRD is a hypothetical reference decoder for checking whether a coded stream complies with the coding standard. HRD defines a CPB (coded picture buffer) for decoding the coded stream and a DPB (decoded picture buffer) for outputting the decoded pictures. HRD checks whether the CPB and DPB do not break when the coded stream is decoded based on the input / output timing of the CPB and DPB coded in the coded stream. If the CPB and DPB do not break, the coded stream complies with the coding standard. HRD type 1 checks whether the coded slices of the coded stream conform to the standard. HRD type 1 does not check NALs that contain SPS, PPS, and SEI messages. The input / output timing of the CPB and DPB is coded (decoded) in the SEI message.

[0034] (First embodiment) <Configuration of the first embodiment> A first embodiment of the present invention will be described. First, the configuration of this embodiment will be described. FIG. 1 is a diagram for explaining the configuration of the first embodiment. This embodiment is composed of a data server 2000 and an image analysis device 3000. The data server 2000 is composed of a storage unit 2001, a conversion unit 2002, and a transmission unit 2003. The data server 2000 has a function of converting an encoded stream, and is therefore also called an image conversion device. The image analysis device 3000 is composed of a reception unit 3001, an image decoding unit 3002, and an image analysis unit 3003.

[0035] An encoded stream is stored in the storage unit 2001. The image decoding unit 3002 is capable of decoding the encoded stream stored in the storage unit 2001. The image decoding unit 3002 has a maximum number of bricks that can be processed in parallel (hereinafter, the maximum number of bricks for parallel processing) of eight.

[0036] <Operation of the First Embodiment> Next, the operation of this embodiment will be described. The operation of each component will be described with reference to Fig. 1. First, the operation of the data server 2000 and the image analyzing device 3000 will be described.

[0037] Based on a request from the image analyzing device 3000, the data server 2000 reads out the coded stream and characteristic parameters of the image decoding unit 3002 from the storage unit 2001 and inputs them to the conversion unit 2002. The characteristic parameters are information on whether the image decoding unit 3002 supports the entry point SEI, and the characteristic parameters are assumed to be held in advance in the storage unit 2001. The conversion unit 2002 converts the input coded stream and inputs the converted coded stream to the transmission unit 2003. The detailed operation of the conversion unit 2002 will be described later. The transmission unit 2003 transmits the input coded stream to the image analyzing device 3000.

[0038] The receiving unit 3001 inputs the coded stream input from the transmitting unit 2003 to the image decoding unit 3002. The image decoding unit 3002 decodes the input coded stream in parallel with the maximum number of brick parallel processing, outputs a decoded image, and inputs it to the image analysis unit 3003. Here, even if a frame rate for outputting a decoded image is set in the input coded stream, the image decoding unit 3002 outputs the decoded image without being bound by the frame rate. The image analysis unit 3003 analyzes the input decoded image and outputs the analysis result. For example, the image analysis unit 3003 measures the image quality and recognizes faces, people, etc., of the input decoded image.

[0039] As described above, when the decoded image is analyzed by the image analysis unit 3003, the analysis result output by the image analysis unit 3003 can be obtained in a short time by outputting the decoded image from the image decoding unit 3002 without being restricted by the frame rate of the encoded stream.

[0040] <Encoded stream> The coded stream stored in the storage unit 2001 will be described. FIG. 2 is a diagram for explaining an example in which a picture in a coded stream is divided into bricks. For simplicity of explanation, a picture is divided into tiles, and the tiles are not divided into bricks. In other words, one tile is one brick. FIG. 2(a) shows an example in which a screen is divided horizontally into eight bricks (tiles) B0 (T0) to B7 (T7) for each CTU row. FIG. 2(b) shows an example in which a screen is divided vertically into eight bricks (tiles) B0 (T0) to B7 (T7) for each CTU column. Brick IDs 0 to 7 are assigned to bricks B0 to B7, respectively. Here, the number of bricks in a picture, NumBricksInPic, is 8.

[0041] Here, one picture is assumed to consist of one slice. Therefore, slice 0 contains eight bricks, B0 to B7. Therefore, single_brick_per_slice_flag in the PPS is set to 0. The coding stream may be in raster scan slice mode or rectangular slice mode. However, entry_point_offset_present_flag, which is a flag indicating whether to code an entry point in the slice header, is set to 0.

[0042] Here, the effect of setting entry_point_offset_present_flag to 0 will be described. When entry_point_offset_present_flag is set to 1, it is necessary to code the entry point obtained as a result of encoding the brick in the slice header, and even if the bricks in the slice are encoded in parallel to generate an encoded stream, the encoded stream cannot be transmitted until the number of bytes indicating the size of the brick is determined. Therefore, it is necessary to transmit the encoded stream in slice units, and low latency can only be achieved in slice units. On the other hand, when entry_point_offset_present_flag is set to 0, it is not necessary to encode the entry point of the brick in the slice header, and the encoded stream can be transmitted in brick units, enabling low latency in brick units.

[0043] <Detailed operation of the conversion unit> Next, a detailed operation of the conversion unit 2002 will be described. Fig. 7 is a flowchart for explaining the operation of the conversion unit. The process shown in Fig. 7 is executed for all coded pictures included in the coded stream.

[0044] It is checked whether or not to apply a loop filter at slice boundaries and brick boundaries (S100). Specifically, it is checked whether the values ​​of loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are the same. If the values ​​of whether or not to apply a loop filter at slice boundaries and brick boundaries are not the same (YES in S100), it is checked whether the decoder is compatible with the entry point SEI (S101). If the values ​​of whether or not to apply a loop filter at slice boundaries and brick boundaries are the same (NO in S100), the slices are reconstructed (S104).

[0045] If the decoder is an entry point SEI-compatible decoder (YES in S101), the entry point is added to the entry point SEI (S102). Detailed operations for adding the entry point to the entry point SEI will be described later. If the decoder is not an entry point SEI-compatible decoder (NO in S101), the entry point is added to the slice header (S103).

[0046] Here, if the application of a loop filter is not the same between slice boundaries and brick boundaries, changing the bricks included in each slice will change the decoded image, and re-encoding will be necessary. Therefore, if the application of a loop filter is not the same between slice boundaries and brick boundaries, the encoded stream is converted so as not to change the configuration of the bricks included in each slice.

[0047] On the other hand, if the application of a loop filter is the same for slice boundaries and brick boundaries, re-encoding is not required because the decoded image does not change even if the bricks included in each slice are changed. Therefore, if the application of a loop filter is the same for slice boundaries and brick boundaries, the coded stream is converted to change the configuration of bricks and slices.

[0048] Here, a step of checking whether the values ​​of loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are the same (S100) is provided in consideration of decoders suitable for parallel processing of slices. When targeting a general decoder, it is better to start with the process of checking whether the decoder is compatible with the entry point SEI (S101) without checking whether the values ​​of loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are the same (S100).

[0049] <Adding an entry point to a slice header> Here, an operation of adding entry points to a slice header will be described. To add entry points to a slice header, information on the entry points of bricks B1 to B7 is coded (decoded) into the coded stream according to the syntax in Fig. 6.

[0050] <Slice reconstruction> Next, a detailed operation of reconstructing a slice will be described. In the coded stream stored in the storage unit 2001, one slice contains eight bricks. This is reconstructed so that one slice contains one brick. Note that single_brick_per_slice_flag in the PPS is set to 1.

[0051] <Adding an entry point to the entry point SEI> Next, a detailed operation of adding an entry point to the entry point SEI will be described. To add an entry point to the entry point SEI, information on the entry points of bricks B1 to B7 is encoded (decoded) according to the syntax of Figs. 8, 9, and 10.

[0052] 8 is a diagram showing the syntax of the SEI message. The payload type (payloadType) is derived from payload_type_byte. If the payload type is 128, the entry point SEI message is encoded (decoded) into the SEI payload (sei_payload). The entry point SEI message is the same as the entry point SEI.

[0053] Figure 9 shows the syntax of the SEI payload. In addition to the entry point SEI message, SEI messages such as the Buffering Period SEI message, Picture Timing, and Picture Timing SEI message are encoded (decoded) in the SEI payload. There are two NAL unit types that store SEI messages: PREFIX_SEI_NUT, which is encoded (decoded) before the encoding slice, and SUFFIX_SEI_NUT, which is encoded (decoded) after the encoding slice.

[0054] FIG. 10 is a diagram for explaining the syntax of the entry point SEI message. The syntax of the entry point SEI message will be explained using FIG. 10. NumSlicesInPic is the number of slices in a picture, which can be obtained by PPS in the case of rectangular slice mode, and can be obtained by counting the number of slices in the case of raster scan slice mode. offset_len_minus1[s] for the Sth slice and NumEntryPoints number of entry_point_offset_minus1 are encoded (decoded). offset_len_minus1 and entry_point_offset_minus1 are the same as described above.

[0055] <Encoded stream after conversion> Fig. 11 is a diagram illustrating the coded stream output by the conversion unit 2002. Fig. 11(i) is an example of an input coded stream, which is coded in the following order: SPS, PPS, coded slices constituting coded picture 0, coded slices 0, ..., coded slices constituting coded picture N.

[0056] FIG. 11(a) is an example of an encoded stream in which an entry point has been added to a slice header. FIG. 11(a) will be described below. The NAL units of the SPS, PPS, and encoded slice 0 form an encoded stream. Encoded slice 0 is composed of a slice header to which an entry point has been added (Modified S-Header in FIG. 11(a)) and slice data including eight encoded bricks from encoded brick 0 (Brick 0 in FIG. 11(a)) to encoded brick 7 (Brick 7 in FIG. 11(a)). Here, the slice header to which an entry point has been added is different from the slice header of the input encoded stream. Note that the slice header to which an entry point has been added is necessary to decode encoded bricks 0 to 7.

[0057] Fig. 11(b) is an example of a coded stream in which slices have been reconstructed. Fig. 11(b) will be explained below. The NAL units of the SPS, PPS, and coded slices 0 to 7 form a coded stream. Each coded slice i (i = 0, 1, ... 7) is composed of slice data including slice header i (New S-Header i in Fig. 11(b)) and coded brick i (Brick i in Fig. 11(b)). Here, slice header i is different from the slice header of the input coded stream. Note that slice header i is required to decode coded brick i.

[0058] In this way, parallelization in units of coding slices can be easily achieved by converting one coding slice into a coding picture including one coding brick and setting single_brick_per_slice_flag in the PPS to 1. In this case, an entry point is not required, and access in units of NAL units is sufficient.

[0059] Here, the coded picture is converted so that one coding slice includes one brick, but the present invention is not limited to this and may be modified as long as the bricks included in the slice are reconfigured according to the maximum number of brick parallel processes of the image decoding unit 3002. The coded picture may be converted so that one coding slice includes multiple bricks. For example, when the maximum number of brick parallel processes of the image decoding unit 3002 is four, the coded picture is converted so that one coding slice includes two bricks.

[0060] Fig. 11(c) is an example of a coded stream in which an entry point is added to the entry point SEI. Fig. 11(c) will be described below. The NAL units of SPS, PPS, coding slice 0, and SEI form a coded stream. Coded slice 0 is composed of a slice header (S-Header in Fig. 11(c)) and slice data including eight coding bricks, from coding brick 0 (Brick 0 in Fig. 11(c)) to coding brick 7 (Brick 7 in Fig. 11(c)). Here, the slice header is the same as the slice header of the input coded stream. Note that a slice header is required to decode coding brick 0 to coding brick 7.

[0061] The size of the encoded stream after conversion in which the entry point is added to the slice header is minimized compared to other methods, but the slice header needs to be re-encoded with bit accuracy, and the size of the encoded slice changes.

[0062] The encoded stream with the reconstructed encoded slices is composed of eight encoded slices, so the size of the encoded stream after conversion is larger than with other methods, but the structure of the encoded stream is simpler because an entry point is no longer necessary. Therefore, although parallelization on an encoding brick basis is not supported, it can be decoded in parallel by a decoder that supports parallelization on an encoding slice basis.

[0063] The coded stream in which the entry point is added to the entry point SEI can be easily generated by inserting the entry point SEI after the last coded slice in the coded picture without changing the input coded stream. In this way, by coding (decoding) the entry point SEI after the coded slice, it is possible to notify the decoder of the entry point in the coded slice while giving freedom to the relationship between the slice and the brick in the raster scan slice mode. In addition, since the coded stream is not changed, the coding stream conformance of type 1 of the HRD is not affected. Therefore, it is not necessary to convert the coded stream while verifying the conformity of the HRD, and the coded stream can be converted reliably and easily.

[0064] <Modification> A modified example of the first embodiment will be described. This modified example has a different PPS syntax from this embodiment. FIG. 12 is a diagram for explaining a part of the syntax of the PPS of the modified example of the first embodiment. The condition for encoding (decoding) entry_point_offset_present_flag is different from that of FIG. 5 of the first embodiment. As shown in FIG. 12, entry_point_offset_present_flag is encoded (decoded) in the PPS when the flag indicating that a slice may include two or more bricks indicates that a slice may include two or more bricks (single_brick_per_slice_flag is 0) or when entropy_coding_sync_enabled_flag is 1.

[0065] As described above, by encoding (decoding) entry_point_offset_present_flag when single_brick_per_slice_flag is 0, even if the picture is composed of multiple tiles (single_tile_in_pic_flag is 0), if the slice contains only one brick, it is not necessary to encode entry_point_offset_present_flag. Therefore, it is possible to eliminate the state where entry_point_offset_present_flag is 1 when the slice contains only one brick, and reduce the amount of coding. Also, by implicitly setting entry_point_offset_present_flag to 0 when entry_point_offset_present_flag does not exist, it is possible to define that no entry point exists when the slice contains only one brick. Also, if entry_point_offset_present_flag is 0 in the slice header, it is not necessary to check the number of bricks included in the slice, and therefore the amount of processing can be reduced.

[0066] (Second embodiment) <Configuration of the second embodiment> A second embodiment of the present invention will be described. Fig. 13 is a diagram for explaining the configuration of the second embodiment. The second embodiment of the present invention differs from the first embodiment in that the data server 2000 is an image coding device 1000. Only the points different from the first embodiment will be described below.

[0067] The image decoding unit 3002 is capable of decoding the encoded stream encoded by the image encoding unit 1002 .

[0068] Based on a request from the image analysis device 3000, the image encoding device 1000 reads image data and characteristic parameters of the image decoding unit 3002 from the storage unit 1001, and inputs them to the image encoding unit 1002. The image encoding unit 1002 encodes the input image data, and inputs the encoded stream to the transmission unit 1003.

[0069] The image encoding unit 1002 differs from the conversion unit 2002 in that it encodes the encoded stream to be output from image data. The encoded stream output by the image encoding unit 1002 has the same structure as the encoded stream output by the conversion unit 2002.

[0070] The following describes the operation of the image encoding unit 1002. In the conversion unit 2002, loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are obtained by decoding from the encoded stream, but in the image encoding unit 1002, loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are determined by the image encoding unit 1002.

[0071] If the decision as to whether to apply a loop filter is not the same between the slice boundary and the brick boundary, the image decoding unit 3002 checks whether it is an entry point SEI-compatible decoder. If the decision as to whether to apply a loop filter is the same between the slice boundary and the brick boundary, the coded picture is coded so that one slice includes one brick.

[0072] If the image decoding unit 3002 is an entry point SEI-compatible decoder, the image decoding unit 3002 encodes the coded picture by using the entry point as the entry point SEI. If the image decoding unit 3002 is not an entry point SEI-compatible decoder, the image decoding unit 3002 encodes the entry point into a slice header and then encodes the coded picture.

[0073] As described above, the image coding device 1000 can output a coded stream that can be decoded by the image decoding unit 3002 based on a request from the image analysis device 3000. Moreover, the effects of the coded stream structure in this embodiment are the same as those in the first embodiment.

[0074] In all the embodiments described above, the coded stream output by the image coding device has a specific data format so that it can be decoded according to the coding method used in the embodiment. The coded stream may be provided by being recorded on a computer-readable recording medium such as an HDD, SSD, flash memory, or optical disk, or may be provided from a server via a wired or wireless network. Therefore, an image decoding device corresponding to this image coding device can decode the coded stream in this specific data format regardless of the providing means.

[0075] When a wired or wireless network is used to exchange the coded stream between the image coding device and the image decoding device, the coded stream may be converted into a data format suitable for the transmission mode of the communication channel before being transmitted. In this case, a transmitting device is provided that converts the coded stream output by the image coding device into coded data in a data format suitable for the transmission mode of the communication channel and transmits the coded data to the network, and a receiving device is provided that receives the coded data from the network, restores the coded stream, and supplies the coded data to the image decoding device. The transmitting device includes a memory that buffers the coded stream output by the image coding device, a packet processing unit that packetizes the coded stream, and a transmitting unit that transmits the packetized coded data via the network. The receiving device includes a receiving unit that receives the packetized coded data via the network, a memory that buffers the received coded data, and a packet processing unit that packetizes the coded data to generate a coded stream and provides it to the image decoding device.

[0076] When a wired or wireless network is used to exchange the encoded stream between the image encoding device and the image decoding device, in addition to the transmitting device and the receiving device, a relay device may be provided that receives the encoded data transmitted by the transmitting device and supplies it to the receiving device. The relay device includes a receiving unit that receives packetized encoded data transmitted by the transmitting device, a memory that buffers the received encoded data, and a transmitting unit that transmits the packetized encoded data to the network. The relay device may further include a receiving packet processing unit that processes the packetized encoded data into packets to generate an encoded stream, a recording medium that accumulates the encoded stream, and a transmitting packet processing unit that packetizes the encoded stream.

[0077] Moreover, a display unit for displaying images decoded by the image decoding device may be added to the configuration to form a display device.

[0078] Moreover, an imaging unit may be added to the configuration and the captured image may be input to the image encoding device, thereby forming an imaging device.

[0079] 14 shows an example of a hardware configuration of a coding / decoding device of the present application. The coding / decoding device includes the configurations of an image coding device and an image decoding device according to an embodiment of the present invention. The coding / decoding device 9000 includes a CPU 9001, a codec IC 9002, an I / O interface 9003, a memory 9004, an optical disk drive 9005, a network interface 9006, and a video interface 9009, and each unit is connected by a bus 9010.

[0080] The image encoding unit 9007 and the image decoding unit 9008 are typically implemented as a codec IC 9002. The image encoding process of the image encoding device according to the embodiment of the present invention is executed by the image encoding unit 9007, and the image decoding process of the image decoding device according to the embodiment of the present invention is executed by the image encoding unit 9007. The I / O interface 9003 is realized by, for example, a USB interface, and is connected to an external keyboard 9104, mouse 9105, etc. The CPU 9001 controls the encoding / decoding device 9000 so as to execute an operation desired by the user based on a user operation input via the I / O interface 9003. The user operation by the keyboard 9104, mouse 9105, etc. includes a selection of whether to execute an encoding or decoding function, a setting of encoding quality, an input / output destination of the encoded stream, an input / output destination of an image, etc.

[0081] When a user desires to play back an image recorded on the disc recording medium 9100, the optical disc drive 9005 reads out an encoded bit stream from the inserted disc recording medium 9100, and sends the read encoded stream to an image decoding unit 9008 of the codec IC 9002 via a bus 9010. The image decoding unit 9008 executes image decoding processing in the image decoding device according to the embodiment of the present invention on the input encoded bit stream, and sends the decoded image to an external monitor 9103 via a video interface 9009. The encoding / decoding device 9000 also has a network interface 9006, and can be connected to an external distribution server 9106 or a mobile terminal 9107 via a network 9101. When a user desires to play back an image recorded on the distribution server 9106 or the mobile terminal 9107 instead of the image recorded on the disc recording medium 9100, the network interface 9006 acquires an encoded stream from the network 9101, instead of reading out the encoded bit stream from the input disc recording medium 9100. Furthermore, when a user wishes to play back the image recorded in memory 9004, image decoding processing is performed on the encoded stream recorded in memory 9004 in an image decoding device according to an embodiment of the present invention.

[0082] When a user desires to encode an image captured by an external camera 9102 and record the encoded stream in memory 9004, the video interface 9009 inputs the image from the camera 9102 and sends the image to an image encoding unit 9007 of the codec IC 9002 via a bus 9010. The image encoding unit 9007 executes image encoding processing in the image encoding device according to the embodiment of the present invention on the image input via the video interface 9009 to generate an encoded bit stream. The encoded bit stream is then sent to the memory 9004 via the bus 9010. When a user desires to record the encoded stream in the disc recording medium 9100 instead of the memory 9004, the optical disc drive 9005 writes the encoded stream to the inserted disc recording medium 9100.

[0083] It is also possible to realize a hardware configuration having an image encoding device but not an image decoding device, or a hardware configuration having an image decoding device but not an image encoding device. Such a hardware configuration can be realized, for example, by replacing the codec IC 9002 with an image encoding unit 9007 or an image decoding unit 9008, respectively.

[0084] The above-mentioned encoding and decoding processes may be realized not only as a transmission, storage, and receiving device using hardware, but also as firmware stored in a ROM (read-only memory) or flash memory, or as software for a computer, etc. The firmware program or software program may be provided by recording it on a recording medium readable by a computer, etc., or may be provided from a server via a wired or wireless network, or may be provided as data broadcasting of terrestrial or satellite digital broadcasting.

[0085] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Industrial Applicability]

[0086] The present invention can be used in image encoding and decoding techniques. [Explanation of symbols]

[0087] 1000 image encoding device, 1001 storage unit, 1002 image encoding unit, 1003 transmission unit, 2000 image conversion device, 2001 storage unit, 2002 conversion unit, 2003 transmission unit, 3000 image analysis device, 3001 reception unit, 3002 image decoding unit, 3003 image analysis unit< / hrd> < / sei>

Claims

1. 1. An image decoding device that divides an image into blocks of a predetermined size, and decodes a code string of a slice including an integer number of rectangular regions each including one or more rows of the blocks, the image decoding device comprising: An image decoding device characterized in that, when a flag indicating that the slice may include two or more of the rectangular areas indicates that the slice may include two or more of the rectangular areas, a flag indicating whether or not to include in the slice header an entry point indicating the starting byte position of the second or subsequent rectangular areas included in the slice is decoded from a picture parameter set that sets picture parameters.

2. The code sequence of the slice is included in a NAL unit, The image decoding device according to claim 1 , characterized in that an entry point indicating a starting byte position of the second or subsequent rectangular area included in the slice is decoded from a NAL unit other than the NAL unit including the slice.

Images