Image encoding devices and image decoding devices, methods therewith, and storage media

Abstract

An image encoding device, which includes an encoding component, which is used to divide an image into multiple sub-pictures and encode each sub-picture so that each sub-picture can be decoded independently. The image encoding device includes: a first decision component, which is used to decide whether each sub-picture forming an image is defined as only a rectangle; a second decision component, which is used to decide the basic pixel number, which is the vertical / horizontal size of the grid forming the sub-picture; an allocation component, which is used to allocate numbers to each grid divided by the basic pixel number; and a third decision component, which is used to decide the maximum number of sub-pictures, the maximum number of sub-pictures is the number of numbers allocated by the allocation component, and the sub-picture is a set of grids allocated with the same number, and encodes the value obtained by subtracting 2 from the maximum number determined by the third decision component.

Description

Technical Field

[0001] The present invention relates to an image encoding device, an image encoding method and program, an image decoding device, an image decoding method and program, and image code data, and more specifically, to an image encoding method / decoding method capable of dividing a picture into rectangles and extracting independent code data. Background Art

[0002] As an encoding method for compressed recording of moving images, the HEVC (High Efficiency Video Coding) encoding method (hereinafter referred to as HEVC) is known. In HEVC, in order to improve encoding efficiency, a basic block larger than a conventional macroblock (16×16 pixels) is adopted. A basic block of a large size is called a CTU (Coding Tree Unit), and its size is a maximum of 64×64 pixels. The CTU is further divided into sub-blocks as a unit for prediction or conversion.

[0003] In addition, in HEVC, a picture can be divided into multiple tiles or slices and encoded. A tile or a slice has a small data dependency, and encoding / decoding processing can be performed in parallel. A great advantage of tile or slice segmentation is that processing can be performed in parallel by a multi-core CPU or the like to shorten processing time. Patent document 1 discloses a technology associated with a tile and a slice.

[0004] In recent years, international standardization activities for more efficient coding methods as successors to HEVC have been started. JVET (Joint Video Experts Group) has been established between ISO / IEC and ITU-T, and the VVC (Versatile Video Coding) coding method (hereinafter referred to as VVC) has been standardized. In VVC, there are sub-pictures configured to include one or more stripes and form a rectangle. A picture can be divided into one or more sub-pictures, and each sub-picture can be processed as independent code data.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2014-11638 Summary of the invention

[0008] Problem that the invention aims to solve

[0009] Regarding sub-pictures of VVC, values ​​representing the maximum number of sub-pictures, basic pixel numbers such as the vertical / horizontal size of a grid that defines the boundary position of each sub-picture, an ID of each sub-picture, a control flag corresponding to each sub-picture, etc. are defined as syntax.

[0010] There are still many redundant parts in the syntax of sub-pictures, which leads to an increase in the amount of code.

[0011] Therefore, the present invention has been made to solve the above-mentioned problem, and provides a technology for reducing the code amount of a generated bit stream by eliminating redundancy in syntax regarding a sub-picture.

[0012] Solutions for solving problems

[0013] In order to solve the above-mentioned problem, the image encoding device according to the present invention has the following configuration. That is, the image encoding device is an image encoding device including an encoding component, the encoding component is used to divide the image into a plurality of sub-pictures and encode each sub-picture so that each sub-picture can be decoded independently, characterized in that it includes: a first decision component for determining whether each sub-picture forming the image is defined as only one rectangle; a second decision component for determining the basic pixel number, the basic pixel number is the vertical / horizontal size of the grid forming the sub-picture; an allocation component for allocating numbers to each grid divided by the basic pixel number; and a third decision component for determining the maximum number of the sub-pictures, the maximum number of the sub-pictures being the number of numbers allocated by the allocation component, wherein the sub-picture is a set of grids to which the same number is allocated, and a value obtained by subtracting 2 from the maximum number determined by the third decision component is encoded.

[0014] In addition, the image decoding device according to the present invention has the following configuration. That is, the image decoding device is an image decoding device including an image decoding component, the image decoding component being capable of decoding a bit stream including code data obtained by encoding each of a plurality of sub-pictures, characterized in that it includes: a first acquisition component for acquiring information indicating whether each sub-picture forming the image includes only one rectangle from the bit stream; a second acquisition component for acquiring the maximum number of sub-pictures from the bit stream; and a third acquisition component for acquiring a basic pixel number from the bit stream, the basic pixel number being the vertical / horizontal size of a grid forming the sub-picture, wherein a value obtained by adding 2 to the value acquired from the second acquisition component is set as the maximum number of sub-pictures, the number of vertical / horizontal grids forming the image is acquired according to the vertical / horizontal size of the grid acquired from the third acquisition component, and information indicating to which sub-picture each grid belongs is acquired.

[0015] In addition, the image encoding device according to the present invention has the following configuration. That is, the image encoding device is an image encoding device including an encoding component, the encoding component is used to divide the image into a plurality of sub-pictures and encode each sub-picture so that each sub-picture can be decoded independently, characterized in that it includes: a first determination component for determining whether each sub-picture forming the image is defined as only one rectangle; a second determination component for determining the basic pixel number, the basic pixel number is the vertical / horizontal size of the grid forming the sub-picture; an allocation component for allocating numbers to each grid divided by the basic pixel number; and a third determination component for determining the maximum number of the sub-pictures, the maximum number of the sub-pictures is the number of numbers allocated by the allocation component, wherein the sub-picture is a set of grids to which the same number is allocated, and the number and the basic pixel number are encoded only when the maximum number determined by the third determination component is not less than 2.

[0016] In addition, the image decoding device according to the present invention has the following configuration. That is, the image decoding device is an image decoding device including an image decoding component, the image decoding component is capable of decoding a bit stream including code data obtained by encoding each of a plurality of sub-pictures, and is characterized in that it includes: a first acquisition component for acquiring information indicating whether each sub-picture forming the image includes only one rectangle from the bit stream; a second acquisition component for acquiring the maximum number of the sub-pictures from the bit stream; and a third acquisition component for acquiring a basic pixel number from the bit stream, the basic pixel number being the vertical / horizontal size of a grid forming the sub-picture, wherein the sub-picture is a set of grids assigned the same number, and the number and the basic pixel number are acquired only when the maximum number determined by the second acquisition component is not less than 2.

[0017] Effects of the Invention

[0018] According to the present invention, it is possible to effectively encode syntax regarding sub-pictures forming an image and efficiently improve encoding.

[0019] Other features and advantages of the present invention will become apparent from the following description in conjunction with the accompanying drawings. Note that throughout the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0021] Figure 1 is a block diagram showing the configuration of an image encoding device;

[0022] Figure 2 is a block diagram showing a configuration of an image decoding device;

[0023] Figure 3 is a flowchart showing an image encoding process in an image encoding device;

[0024] Figure 4 is a flowchart showing an image decoding process in an image decoding device;

[0025] Figure 5 is a block diagram showing an example of a hardware configuration;

[0026] Figure 6 is a diagram showing a bit stream configuration;

[0027] Figure 7 is a diagram showing an example of image segmentation;

[0028] Figure 8 is a diagram showing an example of image segmentation;

[0029] Fig. 9 is a diagram showing a bit stream configuration;

[0030] Fig.10 is a diagram showing an example of image segmentation; and

[0031] Fig.11 is a diagram showing a bit stream configuration. DETAILED DESCRIPTION

[0032] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention requiring all of these features is not limited, and a plurality of these features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions thereof are omitted.

[0033] (First embodiment)

[0034] Embodiments of the present invention will now be described with reference to the accompanying drawings. Figure 1 is a block diagram showing an image encoding device according to this embodiment. Figure 1 , reference numeral 101 denotes a terminal configured to input image data.

[0035] Reference numeral 102 denotes an image segmentation unit, which segments an input image into one or more slice rows or one or more slice columns. Each slice is a continuous set of basic blocks that covers a rectangular area in the image. The image segmentation unit 102 also segments the slice into one or more bricks. Each block is a rectangle formed by one or more basic block rows, which are rows of basic blocks in a slice. The image segmentation unit 102 segments the image into strips, each strip being formed by one or more slices in the image or one or more blocks in a slice. A strip is a basic unit of encoding, and header information such as information indicating a strip type is added to each strip.

[0036] In this embodiment, if Figure 8 As shown, the picture is divided into two sub-pictures. In addition, for the sake of simple description, the sub-pictures, strips, slices and blocks have the same vertical / horizontal size and the same position. However, the present invention is not limited thereto.

[0037] Reference numeral 103 denotes a block dividing unit which divides the basic block line image output from the image dividing unit 102 into a plurality of basic blocks and outputs the image of each basic block to a subsequent stage.

[0038] Reference numeral 104 denotes a prediction unit, which determines the sub-block division of the image data of each basic block, and performs intra-frame prediction as intra-frame prediction or inter-frame prediction as inter-frame prediction for each sub-block, thereby generating predicted image data. No intra-frame prediction or motion vector prediction across blocks is performed. In addition, the prediction unit 104 calculates a prediction error based on the input image data and the predicted image data, and outputs the prediction error. In addition, the prediction unit 104 outputs information required for prediction, for example, information such as sub-block division, prediction mode, and motion vector, and prediction error. The information required for prediction will be referred to as prediction information hereinafter.

[0039] Reference numeral 105 denotes a transform / quantization unit that orthogonally transforms a prediction error on a subblock basis to obtain a transform coefficient, and further quantizes the transform coefficient to obtain a quantized coefficient. Reference numeral 106 denotes an inverse quantization / inverse transform unit that inversely quantizes the quantized coefficient output from the transform / quantization unit 105 to reproduce the transform coefficient, and further inversely orthogonally transforms the transform coefficient to reproduce the prediction error. Reference numeral 108 denotes a frame memory that stores reproduced image data.

[0040] Reference numeral 107 denotes an image reproduction unit. The image reproduction unit 107 generates predicted image data by appropriately referring to the frame memory 108 based on the prediction information output from the prediction unit 104, generates reproduced image data according to the predicted image data and the input prediction error, and outputs the reproduced image data. Reference numeral 109 denotes an in-loop filter unit. The in-loop filter unit 109 performs in-loop filtering processing such as deblocking filtering processing or sample adaptive offset on the reproduced image, and outputs the image that has been subjected to the filtering processing.

[0041] Reference numeral 110 denotes a coding unit. The coding unit 110 generates code data by encoding the quantization coefficient output from the transform / quantization unit 105 and the prediction information output from the prediction unit 104, and outputs the code data. Reference numeral 111 denotes an integrated coding unit. The integrated coding unit 111 receives segmentation information from the image segmentation unit 102, and generates header code data. The integrated coding unit 111 also forms a bit stream by combining the header code data with the code data output from the coding unit 110, and outputs the bit stream. Reference numeral 112 denotes a terminal for outputting the bit stream generated by the integrated coding unit 111 to the outside.

[0042] The image encoding operation of the image encoding device according to the first embodiment will be described below. In this embodiment, moving image data is input on a frame basis. However, still image data corresponding to one frame may also be input. Furthermore, in this embodiment, for ease of description, only intra-frame prediction encoding processing will be described. However, the present invention is not limited thereto, and may also be applied to inter-frame prediction encoding processing. Furthermore, in this embodiment, for the sake of description, it will be described assuming that the block division unit 103 divides the image into basic blocks each having a size of 64×64 pixels. However, the present invention is not limited thereto.

[0043] In this embodiment, first, the image segmentation unit 102 segments a frame of image data input from the terminal 101 into sub-pictures, such as Figure 8 As shown. In this embodiment, a case will be described in which one image data is divided into two sub-pictures and is formed by 18 grids. The vertical / horizontal size of one grid is 256×256 pixels, and the vertical / horizontal size of the sub-picture is 768×768 pixels. For each grid of the sub-picture on the left, 0 is set as the index value. For each grid of the sub-picture on the right, 1 is set as the index value. As described above, in this embodiment, one image data is divided into a plurality of grids, and is divided into a plurality of sub-pictures based on a plurality of grids. In addition, each grid is assigned an index value as a number, and the grids forming one sub-picture are assigned the same index value.

[0044] In this embodiment, each sub-picture is formed of one slice and one tile / block. Information on the size of the sub-picture or slice is sent as segmentation information to the integrated coding unit 111. In addition, each block is segmented into basic block line images, each of which is image data of each basic block line, and is sent to the block segmentation unit 103.

[0045] The block division unit 103 divides the input basic block line image into a plurality of basic blocks, and outputs an image of each basic block to the prediction unit 104. In this embodiment, an image of each basic block having a size of 64×64 pixels is output.

[0046] The prediction unit 104 performs prediction processing on the image data of each basic block input from the block division unit 103. More specifically, subblock division for dividing a basic block into finer subblocks is determined, and an intra prediction mode such as horizontal prediction or vertical prediction is determined based on the subblocks.

[0047] The predicted image data is generated based on the determined intra prediction mode and the coded pixels. In addition, a prediction error is generated based on the input image data and the predicted image data, and is output to the transform / quantization unit 105. In addition, the information of the sub-block division or intra prediction mode is output to the encoding unit 110 and the image reproduction unit 107 as prediction information.

[0048] The transform / quantization unit 105 performs an orthogonal transform / quantization on the input prediction error to generate a quantized coefficient. First, an orthogonal transform process corresponding to the size of the sub-block is performed to generate an orthogonal transform coefficient. Next, the orthogonal transform coefficient is quantized to generate a quantized coefficient. The generated quantized coefficient is output to the encoding unit 110 and the inverse quantization / inverse transform unit 106.

[0049] The inverse quantization / inverse transformation unit 106 inversely quantizes the input quantization coefficient to reproduce the transformation coefficient, and further inversely orthogonally transforms the reproduced transformation coefficient to reproduce the prediction error. The reproduced prediction error is output to the image reproduction unit 107.

[0050] The image reproduction unit 107 reproduces a predicted image by appropriately referring to the frame memory 108 based on the prediction information input from the prediction unit 104. Then, image data is reproduced based on the reproduced predicted image and the reproduced prediction error input from the inverse quantization / inverse transform unit 106. The image data is input and stored in the frame memory 108.

[0051] The in-loop filter unit 109 reads out the reproduced image from the frame memory 108 and performs an in-loop filter process such as a deblocking filter process, etc. The image that has been subjected to the filter process is input again to the frame memory 108 and stored again.

[0052] The encoding unit 110 performs entropy encoding on a block basis on the quantization coefficient generated by the transform / quantization unit 105 and the prediction information input from the prediction unit 104, thereby generating code data. The entropy encoding method is not particularly specified, and Golomb encoding, arithmetic encoding, Huffman encoding, etc. can be used. The generated code data is output to the integrated encoding unit 111.

[0053] The integrated coding unit 111 receives the segmentation information from the image segmentation unit 102, generates the coded data of the header, and multiplexes the code data input from the coding unit 110, etc., to form a bit stream. Finally, the bit stream is output from the terminal 112 to the outside.

[0054] The format of the encoded data using VVC encoded by the image encoding device according to this embodiment is as follows: Figure 6 As shown. Figure 6 In the coded data shown, first, there is a sequence parameter set as header information including information about sequence coding, followed by a picture parameter set as header information including information about picture coding, a slice header as header information including information about coding of each slice, and coded data of each block.

[0055] In the sequence parameter set, pic_width_in_luma_samples and pic_height_in_luma_samples exist as image size information. These represent the number of pixels in the horizontal direction and the number of pixels in the vertical direction regarding image brightness. In this embodiment, since Figure 8 The image shown in is encoded, so pic_width_in_luma_samples is 1536, and pic_height_in_luma_samples is 768. In addition, as basic block data segmentation information, there is log2_ctu_size_minus2 indicating the size of the basic block. The number of vertical and horizontal pixels of the basic block is represented by 1<<(log2_ctu_size_minus2+2). In this embodiment, since the basic block has 64×64 pixels, the value of log2_ctu_size_minus2 is 4.

[0056] In addition, as sub-picture information, there is subpics_present_flag which is information indicating whether sub-picture segmentation exists. If subpics_present_flag is 1, this indicates that the image is segmented into one or more sub-pictures. If subpics_present_flag is 0, this indicates that the image is not segmented.

[0057] If subpics_present_flag is 1, the following information is also encoded. That is, the information encoded at this time includes, for example, max_subpics_minus2 indicating the maximum number of subpictures. In addition, the information encoded at this time includes, for example, information such as subpic_grid_col_width_minus1 and subpic_grid_row_height_minus1 indicating the basic number of pixels of the grid. In addition, the information encoded at this time includes, for example, information such as subpic_grid_idx[i][j] as an index value. In addition, the information encoded at this time includes, for example, information about filtering such as subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i].

[0058] Conventionally, the maximum number of sub-pictures is expressed as "maximum number - 1". In this case, the maximum number of sub-pictures is 1, that is, there is one sub-picture, and the design is made so that the code amount is minimized in a state where the image is not actually divided. However, in this embodiment, the maximum number of sub-pictures is expressed as "maximum number - 2". If the image is divided into two sub-pictures, the value of max_subpics_minus2 is 0, and it can be expressed with the minimum code amount.

[0059] The picture parameter set includes slice data information, block data information, stripe data information and basic block data information. The stripe header includes the stripe data information, etc., and is followed by the coded data of each block.

[0060] Figure 3 1 is a flowchart showing the encoding process in the image encoding device according to the first embodiment. First, in step S301, the image segmentation unit 102 segments the image into slices, blocks, and strips as described above, and sends the segmentation information to the integrated encoding unit 111. In step S309, the segmentation information is converted into header information and encoded into a bit stream by the integrated encoding unit 111. The image segmentation unit 102 also segments the image into sub-pictures and sends these sub-pictures to the block segmentation unit 103.

[0061] In step S302, the block division unit 103 divides the basic block line image into basic blocks. In step S303, the prediction unit 104 performs prediction processing on the image data of each basic block generated in step S302, thereby generating sub-block division information, prediction information such as intra-frame prediction mode, and predicted image data. The prediction unit 104 also calculates a prediction error based on the input image data and the predicted image data.

[0062] In step S304 , the transform / quantization unit 105 orthogonally transforms the prediction error calculated in step S303 to generate a transform coefficient, and performs quantization to generate a quantized coefficient.

[0063] In step S305, the inverse quantization / inverse transform unit 106 inversely quantizes the quantized coefficient generated in step S304, thereby reproducing the transform coefficient. The inverse quantization / inverse transform unit 106 also inversely orthogonally transforms the transform coefficient, thereby reproducing the prediction error.

[0064] In step S306, the image reproduction unit 107 reproduces the predicted image based on the prediction information generated in step S303. The image reproduction unit 107 also reproduces image data from the reproduced predicted image and the prediction error generated in step S305.

[0065] In step S307 , the encoding unit 110 encodes the prediction information generated in step S303 and the quantization coefficient generated in step S304 , thereby generating code data.

[0066] In step S308, the image encoding apparatus determines whether encoding of all basic blocks in the slice is completed. If encoding is completed, the process proceeds to step S309. Otherwise, the process returns to step S302 to process the next basic block.

[0067] In step S309 , the integrated encoding unit 111 generates header information based on the segmentation information sent from the image segmentation unit 102 , and performs encoding.

[0068] Will be based on Figure 8 Detailed examples in this embodiment are described by referring to the image segmentation shown in FIG. In this embodiment, since the image is segmented into two sub-pictures, subpics_present_flag is 1. Next, for max_subpics_minus2 indicating the maximum number of sub-pictures, the value of "maximum number - 2" is set, that is, "0" in this embodiment. If this information is encoded by Golomb coding, 1-bit data "0" indicating "0" is encoded. This can improve the encoding efficiency compared to the case where 3-bit data "010" indicating the value "1" is encoded.

[0069] subpic_grid_col_width_minus1 is 63, and subpic_grid_row_height_minus1 is also 63. These two values ​​are defined in units of four pixels, and more specifically, these are values ​​obtained by subtracting 1 from a value obtained by dividing the basic number of pixels of the grid by 4.

[0070] There is subpic_grid_idx[i][j] which is an index value of a sub-picture. Here, [i][j] represents the position of a grid, wherein i represents a row direction, and j represents a column direction. In this embodiment, the range of the row direction is 0 to 2, the range of the column direction is 0 to 5, and [0][0], [0][1], [0][2], [1][0], [1][1], [1][2], [2][0], [2][1], and [2][2] are 0. In addition, [0][3], [0][4], [0][5], [1][3], [1][4], [1][5], [2][3], [2][4], and [2][5] are 1. The code data of the strips forming the sub-picture are combined based on these sub-picture information, thereby creating the code data of the sub-picture.

[0071] In step S310, the image encoding apparatus determines whether encoding of all basic blocks in the frame is completed. If encoding is completed, the process proceeds to step S311. Otherwise, the process returns to step S302 to process the next basic block.

[0072] In step S311 , the in-loop filter unit 109 performs in-loop filter processing on the image data reproduced in step S306 to generate an image that has been subjected to filter processing, and the processing ends.

[0073] With the above configuration and operation, particularly in step S309, if the image is divided into sub-pictures, information indicating the maximum number of sub-pictures is encoded by "maximum number - 2", thereby efficiently encoding the syntax regarding the sub-pictures.

[0074] Note that in this embodiment, a sub-picture is formed of one slice, one tile, or one block. However, the present invention is not limited thereto. Figure 7 An example in which one stripe is divided into a plurality of slices is shown. More specifically, the stripes of index 0 and index 3 are each formed of four slices, the stripes of index 1 and index 2 are each formed of two slices, and the stripe of index 4 is formed of six slices. This embodiment can also be applied to the sub-picture configuration. Note that in this case, the slice and the block have the same basic pixel size.

[0075] Note that in this embodiment, the value obtained by subtracting 2 from the maximum number of sub-pictures is encoded. However, the present invention is not limited to this. Instead, max_subpics_minus1 obtained by subtracting 1 from the maximum number of sub-pictures may be encoded, and when the value is 1 or more, that is, only when the maximum number of sub-pictures is 2 or more, the following parameters may be encoded. That is, parameters such as subpic_grid_col_width_minus1, subpic_grid_row_height_minus1, and subpic_grid_idx[i][j] may be encoded. That is, if the maximum number is 1, the parameters may be omitted so that they are not encoded.

[0076] The format of the encoded data is as follows Fig. 9 In this case, when the maximum number of sub-pictures is 1, the amount of information of subpic_grid_col_width_minus1, subpic_grid_row_height_minus1, and subpic_grid_idx[0][0] can be reduced. This allows efficient encoding.

[0077] Note that in the above description, when the value of max_subpics_minus1 is 1, that is, only when the maximum number of sub-pictures is 2 or more, the following parameters are encoded. That is, when the maximum number of sub-pictures is 2 or more, the parameters subpic_grid_col_width_minus1, subpic_grid_row_height_minus1, and subpic_grid_idx[i][j] are encoded. In addition, only when the maximum number of sub-pictures is 2 or more, the following two values ​​can be encoded. That is, only when the value of max_subpics_minus1 is 1 or more, subpic_treated_as_flag and loop_filter_across_subpic_enabled_flag can be encoded. This makes it possible to further reduce the amount of information and efficiently encode.

[0078] Furthermore, in this embodiment, the example of dividing an image into two sub-pictures has been described. Fig.10An example of dividing an image into, for example, three sub-pictures is shown in . The value that can be represented by the maximum of Ceil(log2(max_subpics_minus1+1)) bits can be set to subpic_grid_idx. When a value of max_subpics_minus1 or less is used, efficient encoding is possible. More specifically, since the image is divided into three parts, the maximum number of sub-pictures is 3. Therefore, max_subpics_minus1 is obtained by subtracting 1 from 3, which is 2. The number of bits at this time is 2, which is the calculation result of Ceil(log2(2+1)), and a value that can be represented by 2 bits can be set. More specifically, four types of values ​​0, 1, 2, and 3 can be set. When the maximum value that can be set is limited to max_subpics_minus1, as shown in Fig.10 As shown, these values ​​are restricted to 0, 1, and 2, and the syntax regarding sub-pictures can be correctly encoded.

[0079] In addition, in this embodiment, the grid is defined based on four pixels. However, the grid may be defined based on CTU rows / columns. If the basic pixel number of CTU is 64×64, the value of subpic_grid_col_width_minus1 is 3, and the value of subpic_grid_row_height_minus1 is also 3. The information of the encoding object is small, and the amount of code can be reduced.

[0080] Note that in this embodiment, subpic_grid_idx, which is an ID indicating the position of a rectangular area of ​​a sub-picture, is encoded in the sequence parameter set. However, the present invention is not limited thereto. Fig.11 The configuration of the bit stream is shown in which subpic_idx, which is the ID of the corresponding sub-picture, is encoded in the slice header, which is the header of each slice. When this configuration is adopted, the amount of information indicating the position of the rectangular area of ​​the sub-picture can be reduced from the number of grids to the number of slices. Figure 8 In the example shown, the number of grids is 18, and the number of sub-pictures = the number of slices. Therefore, since the number of slices is 2, the number of information of the coded object can be reduced from 18 to 2.

[0081] In addition, subpic_idx is arranged in the slice header. However, the present invention is not limited to this, and subpic_idx may be arranged in the tile code data. This can reduce the code amount, as in the case where subpic_idx is stored in the slice header.

[0082] (Second embodiment)

[0083] Figure 21 is a block diagram showing the configuration of an image decoding device according to a second embodiment of the present invention. The image decoding device according to this embodiment can decode a bit stream including code data generated by encoding each of a plurality of sub-pictures. In this embodiment, decoding of the coded data generated in the first embodiment will be described as an example.

[0084] Reference numeral 201 denotes a terminal to which a coded bit stream is input. Reference numeral 202 denotes a separation decoding unit which separates code data related to information on decoding processing and coefficients from the bit stream and sends these to a decoding unit 203. In addition, the separation decoding unit 202 decodes the code data present in the header portion of the bit stream. In this embodiment, the segmentation information is generated by decoding header information on image segmentation (such as the size of a slice, a block, a stripe, and a basic block, etc.), and is output to the image reproduction unit 205. The separation decoding unit 202 performs the decoding operation with the decoder 203. Figure 1 The operation of the integrated encoding unit 111 is shown to be the reverse operation.

[0085] Reference numeral 203 denotes a decoding unit that decodes the code data output from the separation decoding unit 202, thereby reproducing the quantization coefficient and the prediction information. Reference numeral 204 denotes an inverse quantization / inverse transform unit that inversely quantizes the quantization coefficient to obtain a transform coefficient, and inversely orthogonally transforms the transform coefficient to reproduce the prediction error. Reference numeral 206 denotes a frame memory 206 that stores image data of a reproduced picture. Reference numeral 205 denotes an image reproduction unit. The image reproduction unit 205 generates predicted image data by appropriately referring to the frame memory 206 based on the input prediction information. Then, the image reproduction unit 205 generates reproduced image data based on the predicted image data and the prediction error reproduced by the inverse quantization / inverse transform unit 204. The positions of slices, blocks, and strips in a frame are specified based on the partition information input from the separation decoding unit 202, and the generated reproduced image data is output.

[0086] Reference numeral 207 denotes an in-loop filter unit. Figure 1 Similar to the above-described in-loop filter unit 109 shown, the in-loop filter unit 207 performs in-loop filter processing such as deblocking filter processing on the reproduced image, and outputs the image that has been subjected to the in-loop filter processing. Reference numeral 208 denotes a terminal for outputting reproduced image data to the outside.

[0087] The image decoding operation of the image decoding device according to this embodiment will be described below. In this embodiment, the bit stream generated in the first embodiment is input based on the frame. However, a still image bit stream corresponding to one frame may be input. In addition, in this embodiment, for ease of description, only the intra-frame prediction decoding process will be described. However, the present invention is not limited to this and can also be applied to the inter-frame prediction decoding process.

[0088] exist Figure 2 In the embodiment, a bit stream corresponding to one frame input from terminal 201 is input to separation decoding unit 202. Separation decoding unit 202 separates code data related to information about decoding processing and coefficients from the bit stream, and decodes the code data present in the header portion of the bit stream. More specifically, by decoding Figure 6 The basic block data division information, slice data division information, block data division information, stripe data division information, basic block line data synchronization information, and basic block line data position information shown in the figure are used to generate division information, and the division information is sent to the image reproduction unit 205. Next, the code data of each basic block of the picture data is reproduced and output to the decoding unit 203.

[0089] The decoding unit 203 decodes the code data, thereby reproducing the quantized coefficient and the prediction information. The reproduced quantized coefficient is output to the inverse quantization / inverse transform unit 204, and the reproduced prediction information is output to the image reproduction unit 205.

[0090] The inverse quantization / inverse transformation unit 204 inversely quantizes the input quantization coefficient to generate an orthogonal transformation coefficient, and performs inverse orthogonal transformation to reproduce the prediction error. The reproduced prediction error is output to the image reproduction unit 205.

[0091] The image reproduction unit 205 reproduces the predicted image by appropriately referring to the frame memory 206 based on the prediction information input from the decoding unit 203. Image data is reproduced based on the predicted image and the prediction error input from the inverse quantization / inverse transform unit 204. For example, Figure 8 As shown, the shapes and positions of slices, stripes, and blocks in a frame are specified based on the division information input from the separation decoding unit 202, and the generated reproduced image data is input and stored in the frame memory 206. The stored image data is used for reference in prediction.

[0092] Similar to Figure 1 The in-loop filter unit 109 shown in the figure, the in-loop filter unit 207 reads out the reproduced image from the frame memory 206, and performs in-loop filtering processing such as deblocking filtering processing. The image that has been subjected to the filtering processing is input again to the frame memory 206. The reproduced image stored in the frame memory 206 is finally output to the outside from the terminal 208.

[0093] Figure 4 4 is a flowchart showing an image decoding process in an image decoding apparatus according to the second embodiment. First, in step S401, the separation decoding unit 202 separates code data related to information on the decoding process and coefficients from the bit stream and decodes the code data in the header portion. Figure 6Decode the slice data information, block data information, stripe data information, etc. shown, generate information for decoding, and send it to the image reproduction unit 205. In this embodiment, the segmentation of the image stored in the bitstream is as Figure 8 shown.

[0094] From the values of pic_width_in_luma_samples and pic_height_in_luma_samples in the image size information, it is obtained that the image has 1536×768 pixels.

[0095] Next, since the value of log2_ctu_size_minus2 in the basic block data segmentation information is 4, the size of the basic block is obtained as 64×64 pixels from 1<<log2_ctu_size_minus2+2.

[0096] Next, obtain the sub-picture segmentation information. First, since subpics_present_flag is 1, it can be determined that the image is segmented into sub-pictures. Next, obtain the information on the maximum number of sub-pictures represented by max_subpics_minus2. In this embodiment, when the value 0 is obtained and 2 is added to this value, the maximum number of sub-pictures is obtained as 2. After that, obtain the basic pixel information of the grid forming the sub-pictures represented by subpic_grid_col_width_minus1 and subpic_grid_row_height_minus1. Then, the number of vertical and horizontal grids forming the image represented by NumSubPicGridRows and NumSubPicGridCols is derived by a predetermined calculation method. In this embodiment, subpic_grid_col_width_minus1 can be obtained as 63, and subpic_grid_row_height_minus1 can be obtained as 63.

[0097] Calculate NumSubPicGridRows and NumSubPicGridCols by the following equation.

[0098] NumSubPicGridCols=

[0099] (pic_width_in_luma_samples+

[0100] subpic_grid_col_width_minus1*4+3) /

[0101] (subpic_grid_col_width_minus1*4+4)

[0102] NumSubPicGridRows=

[0103] (pic_height_in_luma_samples+

[0104] subpic_grid_row_height_minus1*4+3) /

[0105] (subpic_grid_row_height_minus1*4+4)

[0106] In this embodiment, NumSubPicGridCols is exported as 6, and NumSubPicGridRows is exported as 3.

[0107] Thereafter, index information indicating to which sub-picture each grid represented by subpic_grid_idx[i][j] belongs is obtained as many as the necessary number represented by NumSubpicGridRows and NumSubpicGridCols. In this embodiment, the number is 18. As for the value, 0 is obtained for [0][0], [0][1], [0][2], [1][0], [1][1], [1][2], [2][0], [2][1], and [2][2]. In addition, 1 is obtained for [0][3], [0][4], [0][5], [1][3], [1][4], [1][5], [2][3], [2][4], and [2][5]. Based on this information, it can be restored that the image is divided into two sub-pictures, and as shown in FIG. Figure 8 As shown, two sub-pictures are divided. Therefore, the bit stream generated in the first embodiment in which the syntax regarding the sub-pictures has been effectively encoded can be decoded.

[0108] After acquiring the information, information required for decoding such as slice data division information and tile data division information is acquired. The division information derived from the separation and decoding unit 202 is sent to the image reproduction unit 205 and is used to specify the position of data to be processed in step S404 in the image.

[0109] In step S402, the decoding unit 203 decodes the code data separated in step S401 and reproduces the quantized coefficients and prediction information. In step S403, the inverse quantization / inverse transform unit 204 inversely quantizes the quantized coefficients to obtain transform coefficients, and also performs inverse orthogonal transform to reproduce the prediction error.

[0110] In step S404, the image reproduction unit 205 reproduces the prediction information and the prediction image generated in step S403. The image reproduction unit 205 also reproduces the image data based on the reproduced prediction image and the prediction error generated in step S404. The reproduced image data is synthesized at an appropriate position in the image based on the segmentation information generated in step S401.

[0111] In step S405, the image decoding apparatus determines whether decoding of all basic blocks in the frame is completed. If decoding is completed, the process proceeds to step S406. Otherwise, the process returns to step S402 to process the next basic block.

[0112] In step S406 , the in-loop filter unit 207 performs in-loop filter processing on the image data reproduced in step S404 to generate an image that has been subjected to filter processing, and the processing ends.

[0113] With the above-described configuration and operation, it is possible to decode a bit stream generated by efficiently encoding the syntax regarding the sub-picture generated in the first embodiment.

[0114] Note that in this embodiment, a value obtained by subtracting 2 from the maximum number of sub-pictures is obtained. However, the present invention is not limited to this. Instead, max_subpics_minus1 obtained by subtracting 1 from the maximum number of sub-pictures may be obtained, and when this value is 1 or greater, that is, only when the maximum number of sub-pictures is 2 or greater, the following information may be obtained. That is, subpic_grid_col_width_minus1, subpic_grid_row_height_minus1, and subpic_grid_idx[i][j] may be obtained. That is, if the maximum number is 1, this information may not be obtained. This makes it possible to decode a bit stream in which redundant sub-picture information is reduced.

[0115] Note that in the above description, when the value of max_subpics_minus1 is 1 or more, that is, only when the maximum number of sub-pictures is 2 or more, the subsequent information is acquired. That is, when the maximum number of sub-pictures is 2 or more, the information of subpic_grid_col_width_minus1, subpic_grid_row_height_minus1, and subpic_grid_idx[i][j] is acquired. When the maximum number of sub-pictures is 2 or more, the following two information can also be acquired. That is, only when the value of max_subpics_minus1 is 1 or more, the two values ​​of subpic_treated_as_flag and loop_filter_across_subpic_enabled_flag can also be acquired. This makes it possible to decode a bit stream in which redundant sub-picture information is further reduced.

[0116] In addition, in this embodiment, an example of dividing an image into two sub-pictures has been described. However, the present invention is not limited to this. Fig.10 Even in the case where the illustrated image is divided into three sub-pictures, a bit stream created by an encoding method for setting the maximum value of subpic_grid_idx to max_subpics_minus1 or less can be decoded.

[0117] In addition, in this embodiment, the grid is defined based on four pixels. However, the grid may be defined based on CTU rows / columns. In this case, when the values ​​of subpic_grid_col_width_minus1 and subpic_grid_row_height_minus1 are restored, the bitstream with reduced code volume can also be decoded by calculating based on the basic number of pixels of CTU.

[0118] Note that in this embodiment, subpic_grid_idx, which is an ID indicating the position of a rectangular area of ​​a sub-picture, is acquired from the sequence parameter set. However, the present invention is not limited to this. Fig.11 A configuration of a bit stream is shown in which subpic_idx, which is the ID of the corresponding sub picture, is encoded in a slice header, which is the header of each slice, instead. Fig.11 1 shows a configuration of a bit stream in which subpic_idx, which is the ID of the corresponding sub-picture, is instead encoded in a slice header, which is the header of each slice. When this configuration is adopted, the amount of information indicating the position of a rectangular area of ​​a sub-picture can be reduced from the number of grids to the number of slices. Figure 8In the example shown, the number of grids is 18, and the number of sub-pictures = the number of slices. Therefore, since the number of slices is 2, decoding can be performed by obtaining two subpic_idxs and calculating the same information as 18 subpic_grid_idxs.

[0119] In addition, an embodiment in which subpic_idx is arranged in the slice header has been described. However, the present invention is not limited thereto, and even if subpic_idx is arranged in tile code data, the same information as subpic_grid_idx can be decoded.

[0120] (Third Embodiment)

[0121] In the above embodiment, it is described that it is assumed that Figure 1 The processing units shown in FIG. 2 are formed of hardware. However, the processing performed by the processing units shown in these drawings may be configured by a computer program.

[0122] Figure 5 is a block diagram showing an example of the configuration of hardware of a computer applicable to the image processing apparatus according to each embodiment.

[0123] The CPU 501 controls the entire computer using the computer program and data stored in the RAM 502 or the ROM 503, and also executes the above-mentioned respective processes as the processes to be performed by the image processing apparatus according to the above-mentioned embodiments. That is, the CPU 501 functions as Figure 1 or Figure 2 The processing unit shown.

[0124] The RAM 502 includes an area configured to temporarily store computer programs and data loaded from an external storage device 506 and data acquired from the outside via an I / F (interface) 507. In addition, the RAM 502 includes a work area used by the CPU 501 to perform various processes. That is, for example, the RAM 502 may be allocated as a frame memory, or various types of other areas may be appropriately provided.

[0125] The ROM 503 stores setting data of the computer, a boot program, etc. The operation unit 504 is formed of a keyboard, a mouse, etc. When the user of the computer operates the operation unit 504, various instructions can be input to the CPU 501. The display unit 505 displays the processing result of the CPU 501. In addition, the display unit 505 is formed of, for example, a liquid crystal display.

[0126] The external storage device 506 is a mass information storage device represented by a hard disk drive. The external storage device 506 stores an OS (operating system) and is configured to enable the CPU 501 to implement Figure 1 or Figure 2 The external storage device 506 may also store various image data to be processed.

[0127] Under the control of the CPU 501, the computer program and data stored in the external storage device 506 are appropriately loaded into the RAM 502 and processed by the CPU 501. The I / F 507 can connect a network such as a LAN or the Internet or other devices such as a projection device or a display device. The computer can acquire or send various information via the I / F 507. Reference numeral 508 denotes a bus that connects the above-mentioned units.

[0128] With respect to the operation formed by the above-described configuration, the CPU 501 plays a major role in controlling the operation described with reference to the above-described flowcharts.

[0129] Other embodiments

[0130] The present invention can be implemented by the following process: a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read out and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0131] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to apprise the public of the scope of the present invention, the following claims are made.

[0132] This application claims the benefit of Japanese Patent Application No. 2019-165580, filed on September 11, 2019, which is hereby incorporated by reference.

[0133] Reference numerals list

[0134] 101, 112, 201, 208: Terminal

[0135] 102: Image segmentation unit

[0136] 103: Block segmentation unit

[0137] 104: Prediction Unit

[0138] 105: Transform / Quantization Unit

[0139] 106, 204: Inverse quantization / inverse transform unit

[0140] 107, 205: Image reproduction unit

[0141] 108, 206: Frame memory

[0142] 109, 207: In-loop filter unit

[0143] 110: Coding unit

[0144] 111: Integrated coding unit

[0145] 202: Separation decoding unit

[0146] 203: Decoding unit

Claims

1. An image encoding device for dividing an image into one or more sub-pictures each including a basic block and encoding the one or more sub-pictures, the image encoding device comprising: a first encoding unit configured to encode first information corresponding to a block size of each basic block into a sequence parameter set of a bitstream, wherein the block size of each basic block is derived by performing an arithmetic left shift of 1 on a sum of a value of the first information and a predetermined integer; a second encoding unit configured to encode a flag related to the presence of information of a sub-picture into the sequence parameter set of the bitstream; a third encoding unit configured to, if the flag has a value indicating the presence of information of the sub-picture, encode, into the sequence parameter set of the bitstream, a syntax element corresponding to the number of sub-pictures included in the picture and second information of an ID related to the sub-picture in the picture; and a fourth encoding unit configured to encode third information for identifying an arrangement of a sub-picture in the image into the sequence parameter set of the bitstream, wherein, when the block size derived based on the first information is 64, a value of the third information is equal to a result of subtracting 1 from the value identified in units of 64, wherein, in the case where the value represented by the syntax element is one or more than one, the third information for identifying the arrangement of sub-pictures in the image is encoded into the sequence parameter set of the bitstream, and Wherein, in a case where the value represented by the syntax element is not one or more than one, the third information for identifying the arrangement of sub-pictures in the image is not encoded into the sequence parameter set of the bitstream.

2. The device according to claim 1, wherein: loop_filter_across_subpic_enabled_flag can be coded into the bitstream.

3. The device according to claim 2, wherein: In case the value represented by the syntax element is one or more than one, the loop_filter_across_subpic_enabled_flag is not encoded into the bitstream.

4. An image decoding device capable of decoding a bit stream obtained by encoding an image including one or more sub-pictures each including a basic block, the image decoding device comprising: a first decoding unit configured to decode first information corresponding to a block size of each basic block from a sequence parameter set of the bitstream, wherein the block size of each basic block is derived by performing an arithmetic left shift of 1 on a sum of a value of the first information and a predetermined integer; a second decoding unit configured to decode a flag related to the presence of information of a sub-picture from the sequence parameter set of the bitstream; a third decoding unit configured to, if the flag has a value indicating the presence of information of the sub-picture, decode, from the sequence parameter set of the bitstream, a syntax element corresponding to the number of sub-pictures included in the image and second information of an ID related to the sub-picture in the image; and a fourth decoding unit configured to decode third information for identifying an arrangement of sub-pictures in the image from the sequence parameter set of the bitstream, wherein, in a case where the block size derived based on the first information is 64, a value of the third information is equal to a result of subtracting 1 from the value identified in units of 64, wherein, in the case where the value represented by the syntax element is one or more than one, the third information for identifying the arrangement of sub-pictures in the image is decoded from the sequence parameter set of the bitstream, and Wherein, in a case where the value represented by the syntax element is not one or more than one, the third information for identifying the arrangement of sub-pictures in the image is not decoded from the sequence parameter set of the bitstream.

5. The device according to claim 4, wherein: loop_filter_across_subpic_enabled_flag can be decoded from the bitstream.

6. The device according to claim 5, wherein: In case the value indicated by the syntax element is not one or more than one, the loop_filter_across_subpic_enabled_flag is not decoded from the bitstream.

7. An image encoding method for dividing an image into one or more sub-pictures each including a basic block and encoding the one or more sub-pictures, the image encoding method comprising: encoding first information corresponding to a block size of each basic block into a sequence parameter set of a bitstream, wherein the block size of each basic block is derived by arithmetic left shifting by 1 a sum of a value of the first information and a predetermined integer; encoding a flag related to the presence of information of a sub-picture into the sequence parameter set of the bitstream; In a case where the flag has a value indicating the presence of information of the sub-picture, encoding a syntax element corresponding to the number of sub-pictures included in the image and second information of an ID related to the sub-picture in the image into the sequence parameter set of the bitstream; and encoding third information for identifying arrangement of sub-pictures in the image into the sequence parameter set of the bitstream, wherein, in a case where the block size derived based on the first information is 64, a value of the third information is equal to a result of subtracting 1 from the value identified in units of 64, wherein, in the case where the value represented by the syntax element is one or more than one, the third information for identifying the arrangement of sub-pictures in the image is encoded into the sequence parameter set of the bitstream, and Wherein, in a case where the value represented by the syntax element is not one or more than one, the third information for identifying the arrangement of sub-pictures in the image is not encoded into the sequence parameter set of the bitstream.

8. An image decoding method capable of decoding a bit stream obtained by encoding an image including one or more sub-pictures each including a basic block, the image decoding method comprising: Decoding first information corresponding to a block size of each basic block from a sequence parameter set of the bitstream, wherein the block size of each basic block is derived by arithmetic left shifting by 1 a sum of a value of the first information and a predetermined integer; decoding a flag related to the presence of information of a sub-picture from the sequence parameter set of the bitstream; In a case where the flag has a value indicating that information of the sub-picture exists, decoding a syntax element corresponding to the number of sub-pictures included in the image and second information of an ID related to the sub-picture in the image from the sequence parameter set of the bitstream; and decoding, from the sequence parameter set of the bitstream, third information for identifying an arrangement of sub-pictures in the image, wherein, in a case where the block size derived based on the first information is 64, a value of the third information is equal to a result of subtracting 1 from a value identified in units of 64, wherein, in the case where the value represented by the syntax element is one or more than one, the third information for identifying the arrangement of sub-pictures in the image is decoded from the sequence parameter set of the bitstream, and Wherein, in a case where the value represented by the syntax element is not one or more than one, the third information for identifying the arrangement of sub-pictures in the image is not decoded from the sequence parameter set of the bitstream.

9. A non-transitory computer-readable storage medium storing a program for causing a computer to perform an image encoding method of dividing an image into one or more sub-pictures each including a basic block and encoding the one or more sub-pictures, the image encoding method comprising: encoding first information corresponding to a block size of each basic block into a sequence parameter set of a bitstream, wherein the block size of each basic block is derived by arithmetic left shifting by 1 a sum of a value of the first information and a predetermined integer; encoding a flag related to the presence of information of a sub-picture into the sequence parameter set of the bitstream; In a case where the flag has a value indicating the presence of information of the sub-picture, encoding a syntax element corresponding to the number of sub-pictures included in the image and second information of an ID related to the sub-picture in the image into the sequence parameter set of the bitstream; and encoding third information for identifying arrangement of sub-pictures in the image into the sequence parameter set of the bitstream, wherein, in a case where the block size derived based on the first information is 64, a value of the third information is equal to a result of subtracting 1 from the value identified in units of 64, wherein, in the case where the value represented by the syntax element is one or more than one, the third information for identifying the arrangement of sub-pictures in the image is encoded into the sequence parameter set of the bitstream, and Wherein, in a case where the value represented by the syntax element is not one or more than one, the third information for identifying the arrangement of sub-pictures in the image is not encoded into the sequence parameter set of the bitstream.

10. A non-transitory computer-readable storage medium storing a program for causing a computer to perform an image decoding method, the image decoding method being capable of decoding a bit stream obtained by encoding an image including one or more sub-pictures each including a basic block, the image decoding method comprising: Decoding first information corresponding to a block size of each basic block from a sequence parameter set of the bitstream, wherein the block size of each basic block is derived by arithmetic left shifting by 1 a sum of a value of the first information and a predetermined integer; decoding a flag related to the presence of information of a sub-picture from the sequence parameter set of the bitstream; In a case where the flag has a value indicating that information of the sub-picture exists, decoding a syntax element corresponding to the number of sub-pictures included in the image and second information of an ID related to the sub-picture in the image from the sequence parameter set of the bitstream; and decoding, from the sequence parameter set of the bitstream, third information for identifying an arrangement of sub-pictures in the image, wherein, in a case where the block size derived based on the first information is 64, a value of the third information is equal to a result of subtracting 1 from a value identified in units of 64, wherein, in the case where the value represented by the syntax element is one or more than one, the third information for identifying the arrangement of sub-pictures in the image is decoded from the sequence parameter set of the bitstream, and Wherein, in a case where the value represented by the syntax element is not one or more than one, the third information for identifying the arrangement of sub-pictures in the image is not decoded from the sequence parameter set of the bitstream.

Images