Image conversion device and image decoding device

Abstract

The present invention provides an image conversion device, which divides an image into blocks of a predetermined size and converts a coded picture containing a stripe, wherein the stripe contains an integer number of rectangular areas, and the rectangular area is a composite of the blocks of more than one block row. In the device, if the coded picture is the same in terms of whether a loop filter is applied at a stripe boundary and a rectangular area boundary, the coded picture is converted in a manner of reconstructing the relationship between the stripe and the rectangular area contained in the coded picture, and if the coded picture is different in terms of whether a loop filter is applied at a stripe boundary and a rectangular area boundary, the coded picture is converted in a manner of not changing the relationship between the stripe and the rectangular area contained in the coded picture.

Description

Technical Field

[0001] The present invention relates to image encoding and decoding technology capable of parallel processing. Background Art

[0002] In HEVC, tile and entropy synchronization coding are defined as tools for parallel processing. In VVC (Versatile Video Coding), a next-generation video coding standard developed by the Joint Video Expert Team (JVET), improvements to parallel processing tools are being discussed.

[0003] Prior Art Literature

[0004] Non-patent literature

[0005] Versatile Video Coding(Draft 6) Summary of the invention

[0006] The technology of Non-Patent Document 1 is in the process of standardization, and there are problems with the processing efficiency of parallel processing tools.

[0007] In order to solve the above-mentioned problem, an image conversion device according to one embodiment of the present invention divides an image into blocks of a predetermined size and converts a coded picture including a stripe, wherein the stripe includes an integer number of rectangular areas, and the rectangular area is a composite of the blocks of more than one block row. If the coded picture is the same as to whether a loop filter is applied at a stripe boundary and a rectangular area boundary, the coded picture is converted in a manner to reconstruct the relationship between the stripe and the rectangular area contained in the coded picture; if the coded picture is the same as to whether a loop filter is applied at a stripe boundary and a rectangular area boundary, the coded picture is converted in a manner to not change the relationship between the stripe and the rectangular area contained in the coded picture.

[0008] Another embodiment of the present invention is an image decoding device. The device is an image decoding device that divides an image into blocks of a predetermined size, decodes a code string of a slice, the slice includes an integer number of rectangular areas, the rectangular area is a composite of the blocks of more than one block row, the code string of the slice is included in a NAL unit, and decodes an entry point from a NAL unit different from the NAL unit including the slice, the entry point indicating the starting byte position of the second and subsequent rectangular areas included in the slice.

[0009] Another aspect of the present invention is also an image decoding device. The device is an image decoding device that divides an image into blocks of a predetermined size, decodes a code string of a slice, the slice including an integer number of rectangular areas, the rectangular area being a composite of the blocks of more than one block row, and when a flag indicating the possibility that the slice includes two or more rectangular areas indicates the possibility that the slice includes two or more rectangular areas, decodes a flag indicating whether an entry point is included in a slice header from a picture parameter set for setting picture parameters, the entry point indicating the start byte position of the second and subsequent rectangular areas included in the slice.

[0010] According to the present invention, a parallel processing tool can be efficiently implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram illustrating the configuration of the first embodiment;

[0012] Figure 2 is a diagram illustrating an example in which a picture in a coded stream is divided into bricks;

[0013] Figure 3 is a diagram illustrating the syntax of a PPS related to allocation information for rectangular CTUs and allocation information for CTU rows for tiles;

[0014] Figure 4 is a diagram illustrating an example of brick indexing;

[0015] Figure 5 is a diagram illustrating the syntax of a portion of the PPS;

[0016] Figure 6 is a diagram illustrating the syntax of a portion of a strip header;

[0017] Figure 7 is a flow chart illustrating the operation of the conversion unit;

[0018] Figure 8 is a diagram showing the syntax of an SEI message;

[0019] Fig. 9 is a diagram showing the syntax of a SEI payload;

[0020] Fig.10 is a diagram illustrating the syntax of an entry point SEI message;

[0021] Fig.11 is a diagram illustrating a coded stream output by a conversion unit;

[0022] Fig.12 This is a diagram for explaining the syntax of a portion of the PPS of a modified example of Implementation 1;

[0023] Fig.13 is a diagram illustrating the configuration of a second embodiment;

[0024] Fig.14 This is a diagram for explaining an example of the hardware configuration of the encoding and decoding device according to the first embodiment. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are based on the technology discussed in the next generation video coding standard VVC (Versatile Video Coding) developed by JVET (Joint Video Expert Team). In VVC, block segmentation, intra-frame prediction, inter-frame prediction, conversion, quantization, entropy coding, loop filter and other technologies are used in the same way as the existing AVC or HEVC.

[0026] The following defines the techniques and technical terms used in the embodiments of the present invention. Unless otherwise specified, images and pictures are used in the same meaning. In the embodiments of the present invention, encoding is performed in the image encoding unit, and decoding is performed in the image decoding unit.

[0027] <Coding Tree Unit and Coding Block>

[0028] The picture is evenly divided into blocks of a predetermined size. The blocks of the predetermined size are defined as coding tree blocks (CTB). The interior of each coding tree block can be recursively divided. The blocks that are the encoding / decoding objects after the coding tree block is divided are defined as coding blocks (CU). Intra-frame prediction and inter-frame prediction are encoded (decoded) in units of coding blocks. In addition, coding tree blocks and coding blocks are collectively defined as blocks. By performing appropriate block division, efficient encoding (decoding) can be performed. The size of the coding tree block can be a fixed value predetermined by the encoder and decoder, or a structure in which the size of the coding tree block determined by the encoder is transmitted to the decoder.

[0029] In addition, in a picture, a coding tree block of luma and two coding tree blocks of chroma are defined as coding tree units (CTUs). CTUs are composed of one coding tree block of luma or three coding tree blocks of color planes. The size of CTU (CTB) is encoded (decoded) in SPS (Sequence Parameter Set).

[0030] <Rectangles and Bricks>

[0031] A rectangle (Tile) is a rectangular area composed of multiple CTUs in a picture. A brick (Brick) is a rectangular area composed of more than one CTU row in the rectangle. One CTU does not belong to multiple rectangles, but only to one rectangle. One CTU does not belong to multiple bricks, but only to one brick. One brick does not belong to multiple rectangles, but only to one rectangle. The allocation of CTUs for rectangles and the allocation of CTU rows for bricks are respectively set in picture units. Moreover, the allocation information of CTUs for rectangles and the allocation information of CTU rows for bricks are respectively encoded (decoded) in PPS (Picture Parameter Set). As described above, the picture is divided into multiple rectangles whose areas do not overlap. In addition, the rectangle is divided into multiple bricks whose areas do not overlap. Afterwards, although the rectangular area composed of multiple CTUs in the picture is represented as a brick, it is a rectangular area composed of more than one CTU row in the rectangle, as long as one CTU does not belong to multiple rectangles but only belongs to one rectangle, or one CTU does not belong to multiple rectangular areas but only belongs to one rectangular area.

[0032] If there are parameter information and a slice header of the SPS that sets the parameters of the coded stream and the PPS that sets the parameters of the coded picture, each coded brick can be decoded separately. Therefore, if there are SPS, PPS, and a slice header, multiple coded bricks can be encoded (decoded) simultaneously and in parallel. A coded brick is a code string of a brick generated by encoding a brick. Hereinafter, a coded picture is a code string of a picture generated by encoding a picture, and a coded strip is a code string of a strip generated by encoding a slice.

[0033] Figure 3 This is a diagram illustrating the syntax of the PPS related to the allocation information of CTUs for rectangles and the allocation information of CTU rows for bricks. brick_splitting_present_flag is a flag indicating whether a rectangle is split into more than two bricks. If brick_splitting_present_flag is 1, one or more rectangles are split into more than two bricks. If brick_splitting_present_flag is 0, all rectangles are not split into more than two bricks. If brick_splitting_present_flag is 1, the splitting information of the bricks is encoded (decoded) for each rectangle.

[0034] When the rectangle is not divided by bricks, 1 rectangle is processed as 1 brick. The rectangle index is the index assigned to the rectangles in the picture in raster scan order. The brick index is the index assigned to the bricks within the picture. When the picture is divided into multiple rectangles, the brick index is the index assigned to each brick within the rectangle in the order of the rectangle index assigned to the rectangles in the picture in raster scan order. The brick index is an integer value greater than or equal to 0 and less than the number of bricks in the picture (NumBricksInPic). The raster scan order means scanning first in the horizontal direction, then moving in the vertical direction and scanning in the horizontal direction again. Figure 4 is a diagram illustrating an example of the brick index. Figure 4 (a) of shows an example where the number of bricks in the picture is 8, Figure 4 (b) of shows an example where the number of bricks in the picture is 5.

[0035] <CTU Processing Order>

[0036] CTUs are processed in raster scan order within the bricks, and the bricks are processed in raster scan order within the rectangles.

[0037] <Stripes and Stripe Modes>

[0038] A stripe is a region obtained by combining an integer number of bricks. As the mode of combining blocks in a stripe, there are a rectangular stripe mode and a raster scan stripe mode. Figure 5 is a diagram illustrating part of the syntax of the PPS. Use Figure 5 to illustrate the rectangular stripe mode and the raster scan stripe mode. When the flag (single_brick_per_slice_flag) indicating the possibility that a stripe contains more than 2 bricks indicates the possibility that a stripe contains more than 2 bricks (single_brick_per_slice_flag is 0), rect_slice_flag is encoded (decoded). When a stripe contains only 1 brick (single_brick_per_slice_flag is 1), rect_slice_flag is 0. If rect_slice_flag is 1, it is the rectangular stripe mode, and if rect_slice_flag is 0, it is the raster scan stripe mode.

[0039] <Rectangular Stripe Mode>

[0040] In rectangular strip mode, tiles in a rectangular area are merged to form a strip. In rectangular strip mode, since the order of processing tiles is not the raster scan order, the correspondence between a strip and a tile is not uniquely determined. Therefore, by encoding (decoding) information indicating the correspondence between a strip and a tile in the PPS, the correspondence between a strip and a tile can be obtained.

[0041] In the PPS, num_slices_in_pic_minus1 is encoded (decoded) as the value obtained by subtracting 1 from the number of slices in the picture to obtain the number of slices in the picture. bottom_right_brick_idx_delta and brick_idx_delta_sign_flag indicating the bottom right brick index included in each slice in the picture are encoded (decoded) in the PPS to derive the correspondence between slices and bricks in the picture.

[0042] As described above, in the rectangular stripe mode, by encoding (decoding) information indicating the correspondence between stripes and tiles in the PPS, it is possible to give freedom to the order in which tiles are processed. Figure 4 As shown in (b), by allocating a large screen to the rectangle to which brick 0 and brick 1 belong, and allocating a small screen to other bricks, the processing order of the large screen can be prioritized.

[0043] <Raster Scan Strip Mode>

[0044] The raster scan strip mode sets the bricks as strips in the raster scan order. Therefore, in the raster scan strip mode, the correspondence between the positions of bricks and strips may not be encoded (decoded) in the PPS. In the raster scan strip mode, by not encoding (decoding) the information indicating the correspondence between strips and bricks in the PPS, the correspondence between strips and bricks can be given freedom. For example, in Figure 4 In (b), the number of tiles included in the slice can be adjusted during encoding, for example, whether the slice is composed of only tile 0 or of both tile 0 and tile 1. Thus, the encoding amount of the slice can be adjusted.

[0045] <Strip ID>

[0046] The stripe ID is an ID for uniquely identifying a stripe. In the case of the raster scan stripe mode, the stripe ID of the i-th stripe [i] arranged in the raster scan order is i, and the stripe ID is a value greater than or equal to 0 and less than or equal to num_slices_in_pic_minus1. In the case of the rectangular stripe mode, the stripe ID can also be explicitly encoded (decoded) as an arbitrary value in the PPS. In the case of the rectangular stripe mode, if the stripe ID is not explicitly encoded (decoded) in the PPS, the stripe ID of the i-th stripe [i] arranged in the raster scan order is i, and the stripe ID is a value greater than or equal to 0 and less than or equal to num_slices_in_pic_minus1.

[0047] <Boundary control flags for loop filter>

[0048] Here, use Figure 5 The syntax of loop_filter_across_bricks_enabled_flag is used to describe the boundary control flag of the loop filter. loop_filter_across_bricks_enabled_flag is a flag indicating whether the loop filter is applied at the brick boundary. If it is 1, the loop filter is applied, and if it is 0, the loop filter is not applied. loop_filter_across_slices_enabled_flag is a flag indicating whether the loop filter is applied at the slice boundary. If it is 1, the loop filter is applied, and if it is 0, the loop filter is not applied. When the brick boundary and the slice boundary overlap, the loop filter is applied if both flags are 1, and the loop filter is not applied if either or both flags are 0.

[0049] <Strip header>

[0050] Next, the slice header will be described. Figure 6 This is a diagram that explains the syntax of a portion of the slice header. If the slice mode is rectangular or the number of bricks in the picture, NumBricksInPic, is greater than 1, the slice address (slice_address) is encoded (decoded). In addition, if the slice mode is raster scan and single_brick_per_slice_flag is 0, num_bricks_in_slice_minus1, which indicates the number of slices contained in the slice, is encoded (decoded).

[0051] In the case of raster scan strip mode, the strip address indicates the initial brick ID contained in the strip. The brick ID is an ID assigned to the CTU contained in each brick in the picture. CTUs belonging to the same brick have the same brick ID. The brick ID is an integer value greater than or equal to 0 and less than the number of bricks in the picture (NumBricksInPic). In the case of raster scan strip mode, since the position of the brick in the picture is defined by the PPS, the starting position of the strip in the picture can be determined most quickly by setting the strip address to the initial brick ID contained in the strip. The bit length of slice_address is Ceil(Log2(NumBricksInPic)), and slice_address is a value greater than or equal to 0 and less than or equal to (NumBricksInPic-1). Here, in the case of raster scan strip mode, since the information indicating the correspondence between the strip and the brick is not encoded (decoded) in the PPS, the strip address is set to the initial brick ID contained in the strip. In the case of raster scan strip mode, since bricks are continuously included in the strip in the raster scan order, the relationship between the strip and the brick can be uniquely determined as long as the initial brick ID and the number of strips contained in the strip are known. In addition, Ceil(x) is a function set to the minimum integer value greater than or equal to x.

[0052] In the case of the rectangular strip mode, information indicating the correspondence between the strip and the brick is encoded (decoded) in the PPS. Therefore, the brick ID can be derived from the strip ID, and the strip address represents the strip ID. Here, in the case of the rectangular strip mode, the strip address is set to the strip ID, but it can also be set to the initial brick ID contained in the strip as in the raster scan strip mode. In the case of the rectangular strip mode, the strip address is also set to the initial brick ID contained in the strip as in the raster scan strip mode, so that no extra circuits or module bricks are required, and the memory and circuit scale of the encoder and decoder can be reduced. In addition, if the signalled_slice_id_flag of the PPS is 0, there is no need to manage the strip ID, which can further reduce the memory of the encoder or decoder.

[0053] <entry point>

[0054] Next, use Figure 6 The starting byte position of the coded brick in the stripe data is described. Figure 6As shown, the starting byte position of the coded brick in the slice data is encoded (decoded) into the slice header as an entry point (entry_point_offset_minus1[i]), so that in the decoder, by decoding the slice header, the starting byte position of the coded brick can be obtained. Here, NumEntryPoints is a value obtained by subtracting 1 from the number of bricks in the slice. Since it is obvious that the starting byte position of the first coded brick in the slice data is 0, encoding (decoding) is not performed, and the starting byte positions of the second and subsequent coded bricks in the slice data are encoded (decoded). entry_point_offset_present_flag is a flag for determining whether to encode (decode) the entry point into the slice header. If entry_point_offset_present_flag is 1, the entry point is encoded (decoded), and if entry_point_offset_present_flag is 0, the entry point is not encoded (decoded). offset_len_minus1 is a value indicating the bit length of entry_point_offset_minus1. In addition, if Figure 5 As shown, when single_tile_in_pic_flag is 0 or entropy_coding_sync_enabled_flag is 1, entry_point_offset_present_flag is encoded (decoded) in the PPS. Here, single_tile_in_pic_flag is a flag indicating whether the picture is 1 rectangle. If single_tile_in_pic_flag is 1, the picture consists of 1 rectangle. If single_tile_in_pic_flag is 0, the picture consists of multiple rectangles. When the picture consists of 1 rectangle, it is not allowed for the rectangle to be composed of multiple bricks, and 1 rectangle is composed of 1 brick. entropy_coding_sync_enabad_flag is a flag indicating whether parallel processing of CTU columns within a brick is allowed. If entropy_coding_sync_enabad_flag is 1, parallel processing of CTU columns within a brick is allowed. If entropy_coding_sync_enabad_flag is 0, parallel processing of CTU columns within a brick is not allowed.

[0055] <sei>

[0056] SEI (Supplemental Enhancement Information) is information that is not required in the process of generating pixels for a decoded image but is required in the operation of the decoder. For example, SEI includes Buffering Period SEI and Picture Timing SEI. Buffering Period SEI is information for enabling the virtual reference decoder, i.e., HRD (Hypothetical Reference Decoder), to work accurately, and Picture Timing SEI determines the output timing of the decoded image. SEI is encoded (decoded) as an SEI message.

[0057] <NAL unit>

[0058] The NAL unit consists of a header and a payload. In the header, the payload type is encoded (decoded), and a bitstream of the type indicated by the payload type is included in the payload. The payload types include SPS, PPS, coded slices, SEI messages, etc. For example, if the payload type indicates a payload type containing a slice, the bitstream of the slice is stored in the payload, and if the payload type indicates a payload type containing an SEI message, the bitstream of the SEI message is stored in the payload.

[0059] <Coded bitstream>

[0060] In VVC or HEVC, SPS, PPS, coded slices, SEI messages, etc. are stored in NAL units to form a coded bitstream. A coded slice consists of a slice header and slice data. Coding tree blocks are encoded in the slice data. The coded bitstream contains one or more coded pictures.

[0061] <hrd>

[0062] HRD is a virtual reference decoder used to check whether the coded stream complies with the coding standard. HRD is defined as CPB (coded picture buffer) and DPB (decoded picture buffer), CPB is used to decode the coded stream, and DPB is used to output the decoded pictures. In HRD, when the coded stream is decoded according to the input and output timing of CPB and DPB encoded in the coded stream, it is checked that there is no failure in CPB and DPB. If there is no failure in CPB and DPB, the coded stream complies with the coding standard. In type 1 of HRD, the standard matching degree of the coded slices of the coded stream is checked. In type 1 of HRD, NAL containing SPS, PPS, SEI messages is not within the inspection object. The input and output timing of CPB and DPB are encoded (decoded) in the SEI message.

[0063] (First Embodiment)

[0064] <Configuration of First Embodiment>

[0065] A first embodiment of the present invention will be described. First, the configuration of this embodiment will be described. Figure 1 1 is a diagram for explaining the structure of the first embodiment. This embodiment is composed of a data server 2000 and an image analysis device 3000. The data server 2000 is composed of a storage unit 2001, a conversion unit 2002, and a transmission unit 2003. The data server 2000 has a function of converting a coded stream, and is therefore also called an image conversion device. The image analysis device 3000 is composed of a receiving unit 3001, an image decoding unit 3002, and an image analysis unit 3003.

[0066] The storage unit 2001 stores a coded stream. The image decoding unit 3002 can decode the coded stream stored in the storage unit 2001. The image decoding unit 3002 sets the maximum number of tiles that can be processed in parallel (hereinafter referred to as the maximum number of tiles to be processed in parallel) to eight.

[0067] <Operation of the First Embodiment>

[0068] Next, the operation of this embodiment will be described. Figure 1 The operation of each component will be described. First, the operation of the data server 2000 and the image analysis device 3000 will be described.

[0069] The data server 2000 reads the coded stream and the characteristic parameters of the image decoding unit 3002 from the storage unit 2001 according to the request of the image analysis device 3000, and inputs them to the conversion unit 2002. The characteristic parameters are information on whether the image decoding unit 3002 supports the entry point SEI, and the characteristic parameters are stored in the storage unit 2001 in advance. The conversion unit 2002 converts the input coded stream and inputs the converted coded stream to the transmission unit 2003. The detailed operation of the conversion unit 2002 will be described later. The transmission unit 2003 transmits the input coded stream to the image analysis device 3000.

[0070] The receiving unit 3001 inputs the coded stream input from the sending unit 2003 to the image decoding unit 3002. The image decoding unit 3002 decodes the input coded stream in parallel at the maximum brick parallel processing number, outputs the decoded image, and inputs it to the image analysis unit 3003. Here, even if the frame rate of the output decoded image is set for the input coded stream, the image decoding unit 3002 outputs the decoded image without being restricted by the frame rate. The image analysis unit 3003 analyzes the input decoded image and outputs the analysis result. For example, the image analysis unit 3003 measures the image quality of the input decoded image, recognizes faces or people, etc.

[0071] As described above, when the image analysis unit 3003 analyzes a decoded image, the decoded image is output from the image decoding unit 3002 without being restricted by the frame rate of the coded stream, and the analysis result output from the image analysis unit 3003 can be obtained in a short time.

[0072] <Encoded stream>

[0073] The coded stream stored in the storage unit 2001 is described. Figure 2 This is a diagram illustrating an example of dividing a picture in a coded stream into tiles. Here, in order to simplify the description, the picture is divided into rectangles, but the rectangles are not divided into tiles. That is, one rectangle is one tile. Figure 2 (a) is an example of dividing the screen into 8 bricks (rectangles) from B0 (T0) to B7 (T7) in the horizontal direction for each CTU row. Figure 2 (b) is an example of dividing the picture into 8 bricks (rectangles) from B0 (T0) to B7 (T7) in the vertical direction for each CTU column. Bricks B0 to B7 are assigned 0 to 7 as brick IDs respectively. Here, the number of bricks in the picture NumBricksInPic is 8.

[0074] Here, it is assumed that one picture is composed of one slice. Therefore, slice 0 contains eight bricks from B0 to B7. Therefore, single_brick_per_slice_flag of PPS is 0. The coded stream can be in raster scan slice mode or rectangular slice mode. However, the flag of whether the entry point is encoded in the slice header, that is, entry_point_offset_present_flag, is set to 0.

[0075] Here, the effect of entry_point_offset_present_flag being 0 is explained. If entry_point_offset_present_flag is set to 1, the entry point obtained in the encoded result of the brick needs to be encoded in the slice header. Even if the bricks in the slice are encoded in parallel to generate a coded stream, the coded stream cannot be sent until the number of bytes representing the size of the brick is determined. Therefore, the coded stream needs to be sent in units of slices, and the delay can only be reduced in units of slices. On the other hand, when entry_point_offset_present_flag is set to 0, the entry point of the brick does not need to be encoded in the slice header, the coded stream can be sent in units of bricks, and the delay can be reduced in units of bricks.

[0076] <Detailed Operation of the Conversion Unit>

[0077] Next, the detailed operation of the conversion unit 2002 will be described. Figure 7 This is a flowchart illustrating the operation of the conversion unit. Figure 7 It is executed for all coded pictures included in the coded stream.

[0078] Check whether the loop filter is applied at the same slice boundary and brick boundary (S100). Specifically, check whether the values ​​of loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are the same. When the loop filter is applied at different slice boundaries and brick boundaries ("Yes" of S100), check whether it is a decoder that supports entry point SEI (S101). When the loop filter is applied at the same slice boundary and brick boundary ("No" of S100), reconstruct the slice (S104).

[0079] If the decoder supports the entry point SEI ("Yes" in S101), the entry point is added to the entry point SEI (S102). The detailed operation of adding the entry point to the entry point SEI will be described later. If the decoder does not support the entry point SEI ("No" in S101), the entry point is added to the slice header (S103).

[0080] Here, when whether or not to apply the loop filter differs between the slice boundary and the tile boundary, if the tiles included in each slice are changed, the decoded image will change, and thus re-encoding is required. Therefore, when whether or not to apply the loop filter differs between the slice boundary and the tile boundary, the coded stream is converted in a manner that avoids changing the composition of the tiles included in each slice.

[0081] On the other hand, when whether or not the loop filter is applied is the same at the slice boundary and the brick boundary, the decoded image does not change even if the bricks included in each slice are changed, so there is no need to re-encode. Therefore, when whether or not the loop filter is applied is the same at the slice boundary and the brick boundary, the coded stream is converted in such a way that the composition of the bricks and the slices is changed.

[0082] Here, considering a decoder suitable for parallel processing of slices, a step of checking whether the values ​​of loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are the same (S100) is set. In the case of a general decoder, the process of checking whether the values ​​of loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag are the same (S100) may not be performed, and it is better to start from the process of checking whether it is a decoder that supports entry point SEI (S101).

[0083] <Add entry point to strip header>

[0084] Here, the action of adding an entry point to a slice header is described. Figure 6 The syntax of adding the entry point to the slice header is to encode (decode) the entry point information of brick B1 to brick B7 into the encoded stream.

[0085] <Reconstruction of strips>

[0086] Next, the detailed operation of reconstructing a slice is described. In the coded stream stored in the storage unit 2001, one slice includes eight bricks. This is reconstructed so that one slice includes one brick. In addition, the single_brick_per_slice_flag of the PPS is set to 1.

[0087] <Add entry point to entry point SEI>

[0088] Next, the detailed operation of adding the entry point to the entry point SEI is described. Figure 8 , Fig. 9 , Fig.10 The syntax of adding an entry point to the entry point SEI is to encode (decode) the entry point information of brick B1 to brick B7.

[0089] Figure 8 This is a diagram showing the syntax of the SEI message. The payload type (payloadType) is derived from the payload_type_byte. When the payload type is 128, the entry point SEI message is encoded (decoded) into the SEI payload (sei_payload). The entry point SEI message and the entry point SEI have the same meaning.

[0090] Fig. 9 This is a diagram showing the syntax of the SEI payload. In the SEI payload, in addition to the entry point SEI message, SEI messages such as the Buffering Period SEI message, Picture Timing, and Picture Timing SEI message are encoded (decoded). There are two NAL unit types that store SEI messages, one is PREFIX_SEI_NUT, which is encoded (decoded) before the coded slice, and the other is SUFFIX_SEI_NUT, which is encoded (decoded) after the coded slice.

[0091] Fig.10 is a diagram illustrating the syntax of an entry point SEI message. Fig.10 The syntax of the entry point SEI message is explained. NumSlicesInPic is the number of slices in the picture, which can be obtained from the PPS in the case of rectangular slice mode and can be obtained by counting the number of slices in the case of raster scan slice mode. Offset_len_minus1[s] and NumEntryPoints entry_point_offset_1 related to the Sth slice are encoded (decoded). Offset_len_minus1 and entry_point_offset_minus1 have the same meaning as above.

[0092] <Converted coded stream>

[0093] Fig.11 This is a diagram for explaining the coded stream output by the conversion unit 2002. Fig.11 (i) is an example of an input coded stream. Encoding is performed in the order of SPS, PPS, coded slices constituting coded picture 0 (picture 0), coded slice 0 (Slice 0) constituting coded picture 1, ..., coded slices constituting coded picture N (picture N).

[0094] Fig.11 (a) is an example of a coded stream in which an entry point is added to a slice header. Fig.11 (a) is used to illustrate. The NAL units of SPS, PPS, and coded slice 0 form a coded stream. Coding slice 0 is composed of slice data, which includes a slice header ( Fig.11 (a) Modified S-Header) and from coded brick 0 ( Fig.11 (a) Brick 0) to Coded Brick 7 ( Fig.11 The eight coding bricks are Brick 7 of (a). Here, the slice header with the entry point added is different from the slice header of the input coded stream. In addition, in order to decode coding bricks 0 to 7, the slice header with the entry point added is required.

[0095] Fig.11 (b) is an example of a coded stream of a reconstructed slice. Fig.11 (b) is used to illustrate. SPS, PPS, and the NAL units of coded slices 0 to 7 form a coded stream. Coding slice i (i = 0, 1, ... 7) is composed of slice data, which includes slice header i ( Fig.11 (b) New S-Header i) and Coded Brick i ( Fig.11 Here, the slice header i is different from the slice header of the input coded stream. In addition, the slice header i is required to decode the coded brick i.

[0096] In this way, parallelization in units of coded slices can be easily achieved by converting one coded slice into a coded picture containing one coded brick and setting the single_brick_per_slice_flag of the PPS to 1. In this case, no entry point is required, as long as access can be made in units of NAL units.

[0097] Here, although the coded picture is converted so that one coded strip contains one coded brick, the present invention is not limited thereto, and the bricks contained in the strip can be reconstructed according to the maximum number of bricks processed in parallel by the image decoding unit 3002. The coded picture may also be converted so that one coded strip contains a plurality of bricks. For example, when the maximum number of bricks processed in parallel by the image decoding unit 3002 is 4, the coded picture is converted so that one coded strip contains two bricks.

[0098] Fig.11 (c) is an example of a coded stream in which an entry point is added to the entry point (EntryPoint) SEI. Fig.11 (c) is used to illustrate. The NAL units of SPS, PPS, coded slice 0, and SEI form a coded stream. Coding slice 0 is composed of slice data, which includes a slice header ( Fig.11 (c) S-Header) and from the coded brick 0 ( Fig.11 (c) Brick 0) to Coded Brick 7 ( Fig.11 (c) Brick 7) These 8 coding bricks. Here, the slice header is the same as the slice header of the input coded stream. In addition, the slice header is required to decode coding bricks 0 to 7.

[0099] Compared with other methods, the size of the converted coded stream of the coded stream in which the entry point is added to the slice header is the smallest. However, the slice header needs to be re-encoded with bit accuracy, and the size of the coded slice changes.

[0100] The code stream for reconstructing the coded slices consists of 8 coded slices, so the size of the converted coded stream becomes larger than that of other methods, but since no entry point is required, the structure of the coded stream becomes simple. Therefore, although parallelization in units of coded bricks is not supported, parallel decoding is possible with decoders that support parallelization in units of coded slices.

[0101] In the coded stream in which the entry point is added to the entry point SEI, the input coded stream remains unchanged, and the entry point SEI is only inserted after the last coded slice in the coded picture, so that the converted coded stream can be generated in advance. In this way, by encoding (decoding) the entry point SEI after the coded slice, the entry point in the coded slice can be notified to the decoder while giving freedom to the relationship between the strip and the brick in the raster scan strip mode. In addition, since the coded stream is not changed, the type 1 coded stream performance of the HRD is not affected. Therefore, there is no need to convert the coded stream while verifying the compatibility of the HRD, and the coded stream can be converted reliably and easily.

[0102] <Modification>

[0103] Next, a description will be given of a modification of Embodiment 1. This modification is different from the present embodiment in the syntax of PPS. Fig.12 This is a diagram for explaining the syntax of a portion of the PPS of a modified example of the first embodiment. Figure 5 Compared with , the conditions for encoding (decoding) entry_point_offset_present_flag are different. Fig.12 As shown, when the flag indicating the possibility that a slice includes more than two bricks indicates the possibility that a slice includes more than two bricks (single_brick_per_slice_flag is 0) or entropy_coding_sync_enabled_flag is 1, entry_point_offset_present_flag is encoded (decoded) in the PPS.

[0104] As described above, entry_point_offset_present_flag is encoded (decoded) when single_brick_per_slice_flag is 0, so that even in the case where the picture is composed of multiple rectangles (single_tile_in_pic_flag is 0), there is no need to encode entry_point_offset_present_flag when the slice includes only 1. Therefore, when the slice includes only 1 tile, the state where entry_point_offset_present_flag is 1 is eliminated, and the amount of code can be reduced. In addition, if entry_point_offset_present_flag does not exist, it is implied that entry_point_offset_present_flag is 0, thereby, when the slice includes only 1 tile, it can be defined that there is no entry point. In addition, if entry_point_offset_present_flag is 0 in the slice header, there is no need to check the number of tiles included in the slice, so the amount of processing can be reduced.

[0105] (Second Embodiment)

[0106] <Configuration of Second Embodiment>

[0107] A second embodiment of the present invention will be described. Fig.13 2 is a diagram for explaining the configuration of the second embodiment. The second embodiment of the present invention is different from the first embodiment in that the data server 2000 is the image coding device 1000. Only the differences from the first embodiment will be described below.

[0108] The image decoding unit 3002 can decode the coded stream coded by the image coding unit 1002 .

[0109] The image coding device 1000 reads the image data and the characteristic parameters of the image decoding unit 3002 from the storage unit 1001 in response to the request of the image analysis device 3000, and inputs them to the image coding unit 1002. The image coding unit 1002 encodes the input image data and inputs the encoded coded stream to the transmission unit 1003.

[0110] The difference between the image coding unit 1002 and the conversion unit 2002 is that the outputted coded stream is coded according to the image data. The structure of the coded stream outputted by the image coding unit 1002 is the same as that of the coded stream outputted by the conversion unit 2002.

[0111] The following describes the operation of the picture coding unit 1002. The conversion unit 2002 decodes and acquires loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag from the coded stream, but the picture coding unit 1002 determines loop_filter_across_bricks_enabled_flag and loop_filter_across_slices_enabled_flag.

[0112] When whether the loop filter is applied is different at the slice boundary and the tile boundary, the image decoding 3002 checks whether it is a decoder supporting the entry point SEI. When whether the loop filter is applied is the same at the slice boundary and the tile boundary, the coded picture is encoded so that 1 slice contains 1 tile.

[0113] If the image decoding unit 3002 is a decoder supporting entry point SEI, the entry point is used as the entry point SEI to encode the coded picture. If the image decoding unit 3002 is not a decoder supporting entry point SEI, the entry point is encoded in the slice header to encode the coded picture.

[0114] As described above, the image coding device 1000 can output a coded stream decodable by the image decoding unit 3002 based on a request from the image analysis device 3000. The effects of the structure of the coded stream are also the same as those of the first embodiment.

[0115] In all the above-described embodiments, the coded stream output by the image coding device has a specific data format so that it can be decoded according to the coding method used in the embodiment. The coded stream can be provided by recording in a computer-readable recording medium such as an HDD, SSD, flash memory, optical disk, etc., or provided by a server via a wired or wireless network. Therefore, the image decoding device corresponding to the image coding device can decode the coded stream of the specific data format regardless of the provision method.

[0116] In order to exchange the coded stream between the image coding device and the image decoding device, the coded stream can also be converted into a data form suitable for the communication path transmission method for transmission when using a wired or wireless network. In this case, a sending device and a receiving device are provided. The sending device converts the coded stream output by the image coding device into coded data in a data form suitable for the communication path transmission method and sends it to the network. The receiving device receives the coded data from the network, restores it to the coded stream, and provides it to the image decoding device. The sending device includes a memory, a packet processing unit, and a sending unit. The memory buffers the coded stream output by the image coding device, the packet processing unit groups the coded stream, and the sending unit sends the grouped coded data via the network. The receiving device includes a receiving unit, a memory, and a packet processing unit. The receiving unit receives the grouped coded data via the network. The memory buffers the received coded data. The packet processing unit performs packet processing on the coded data and generates a coded stream, and provides it to the image decoding device.

[0117] In order to exchange the coded stream between the image coding device and the image decoding device, in the case of using a wired or wireless network, in addition to the sending device and the receiving device, a relay device may be provided, and the relay device receives the coded data sent by the sending device and provides it to the receiving device. The relay device includes a receiving unit, a memory, and a sending unit, the receiving unit receives the grouped coded data sent by the sending device, the memory buffers the received coded data, and the sending unit sends the grouped coded data to the network. In addition, the relay device may include a receiving packet processing unit, a recording medium, and a sending packet processing unit, the receiving packet processing unit performs packet processing on the grouped coded data and generates a coded stream, the recording medium stores the coded stream, and the sending packet processing unit groups the coded stream.

[0118] Furthermore, a display unit may be added as a display device in the configuration, and the display unit may display the image decoded by the image decoding device.

[0119] Furthermore, the imaging device may be used by adding an imaging unit to the configuration and inputting the captured image into the image encoding device.

[0120] Fig.14 An example of the hardware configuration of the codec device of the present invention is shown. The codec device includes the configuration of the image coding device and the image decoding device of the embodiment of the present invention. The codec device 9000 includes a CPU 9001, a codec IC 9002, an I / O interface 9003, a memory 9004, an optical disk drive 9005, a network interface 9006, and a video interface 9009, and each part is connected via a bus 9010.

[0121] The image encoding unit 9007 and the image decoding unit 9008 are generally implemented as a codec IC 9002. The image encoding process of the image encoding device of the embodiment of the present invention is performed by the image encoding unit 9007, and the image decoding process in the image decoding device of the embodiment of the present invention is performed by the image decoding unit 9008. The I / O interface 9003 is implemented by a USB interface, for example, and is connected to an external keyboard 9104, a mouse 9105, etc. The CPU 9001 controls the codec device 9000 based on the user operation input via the I / O interface 9003 to perform the action desired by the user. The operation performed by the user through the keyboard 9104, the mouse 9105, etc. includes the selection of the function of performing encoding or decoding, the setting of the encoding quality, the input and output destination of the encoded stream, the input and output destination of the image, etc.

[0122] When the user wishes to perform an operation such as reproducing an image recorded in the disk recording medium 9100, the optical disk drive 9005 reads a coded bit stream from the inserted disk recording medium 9100, and sends the read coded stream to the image decoding unit 9008 of the codec IC 9002 via the bus 9010. The image decoding unit 9008 performs the image decoding process in the image decoding device of the embodiment of the present invention on the input coded bit stream, and sends the decoded image to the external monitor 9103 via the video interface 9009. In addition, the codec device 9000 has a network interface 9006, and can be connected to an external distribution server 9106 and a portable terminal 9107 via a network 9101. When the user wishes to change the image recorded in the disk recording medium 9100 or reproduce the image recorded in the distribution server 9106 or the portable terminal 9107, the network interface 9006 changes the coded bit stream read from the disk recording medium 9100, and obtains the coded stream via the network 9101. Furthermore, when a user wishes to reproduce an image recorded in the memory 9004 , image decoding processing in the image decoding apparatus according to the embodiment of the present invention is performed on the coded stream recorded in the memory 9004 .

[0123] When the user wishes to perform an operation such as encoding an image captured by an external camera 9102 and recording it in the memory 9004, the video interface 9009 inputs the image from the camera 9102 and sends it to the image encoding unit 9007 of the codec IC 9002 via the bus 9010. The image encoding unit 9007 performs the image encoding process in the image encoding device of the embodiment of the present invention on the image input through the video interface 9009, and generates a coded bit stream. Then, the coded bit stream is sent to the memory 9004 via the bus 9010. When the user wishes to record the coded stream in the disk recording medium 9100 instead of the memory 9004, the optical disk drive 9005 writes the coded stream to the inserted disk recording medium 9100.

[0124] A hardware configuration having an image encoding device but no image decoding device or a hardware configuration having an image decoding device but no image encoding device may be realized. Such a hardware configuration may be realized by replacing the codec IC 9002 with the image encoding unit 9007 or the image decoding unit 9008, for example.

[0125] The above-mentioned processing related to encoding and decoding can of course be implemented as a transmission, storage, and receiving device using hardware, or can be implemented by firmware stored in ROM (read-only memory), flash memory, etc. or software such as a computer. The firmware program and software program can be recorded in a readable recording medium such as a computer and provided, or can be provided by a server through a wired or wireless network, or can be provided as a data broadcast of a terrestrial wave or satellite digital broadcast.

[0126] The present invention has been described above based on the embodiments. The above embodiments are illustrative only, and various modifications can be made to the combinations of the components and the processing procedures, and those skilled in the art will understand that such modifications are also within the scope of the present invention.

[0127] Industrial Applicability

[0128] The present invention can be used in image encoding and decoding technology.

[0129] Explanation of symbols

[0130] 1000 Image encoding device

[0131] 1001 Storage

[0132] 1002 Image Coding Unit

[0133] 1003 Sending Department

[0134] 2000 Image conversion device

[0135] 2001 Storage Department

[0136] 2002 Conversion Department

[0137] 2003 Sending Department

[0138] 3000 Image Analysis Device

[0139] 3001 Sending Department

[0140] 3002 Image decoding unit

[0141] 3003 Image Analysis Department< / hrd> < / sei>

Claims

1. An image conversion device, which divides an image into blocks of a predetermined size and converts a coded image containing a strip, wherein the strip contains an integer number of rectangular areas, and the rectangular area is a composite of the blocks of more than one block row, characterized in that: If whether the loop filter is applied to the encoded picture is the same at the slice boundary and the rectangular area boundary, the encoded picture is converted in a manner to reconstruct the relationship between the slice and the rectangular area contained in the encoded picture; if the loop filter is applied to the encoded picture is different at the slice boundary and the rectangular area boundary, the encoded picture is converted in a manner not to change the relationship between the slice and the rectangular area contained in the encoded picture.

Images