Image decoding device, image decoding method, image encoding device, and image encoding method

Abstract

To further improve practicability.SOLUTION: Reference image information indicating a reference image to which the current picture refers is stored in a header area of an access unit. The reference image information is stored in an AUD NAL unit indicating a delimiter of an access unit or a NEW NAL unit provided separately from the AUD NAL unit, and is applied to all Tile group NAL units in the access unit. This technology can be applied to, for example, an image decoding device and an image encoding device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image decoding device, an image decoding method, an image coding device, and an image coding method, and more particularly to an image decoding device, an image decoding method, an image coding device, and an image coding method that can be made more practical. [Background technology]

[0002] In H.264 / AVC, which is one of the standard specifications for image coding, each image (picture) is divided into one or more slices. Each slice is classified into one of I slices (Intra Slice), P slices (Predictive Slice), and B slices (Bi-predictive Slice). An I slice is a slice that is decoded independently without referring to other images. A P slice is a slice that is decoded by referring to a single other image. A B slice is a slice that is decoded by referring to multiple other images.

[0003] The first picture of a sequence, which consists of only I-slices, is called an IDR (Instantaneous Decoding Refresh) picture. IDR pictures are identified by a Network Abstraction Layer (NAL) unit type value. Pictures following an IDR picture in the same sequence do not refer to pictures before the IDR picture in decoding order, but are only located after the IDR picture in presentation order. Therefore, when random access is attempted to a point in the middle of a video of a certain coded stream (decoding / playback from the middle of the stream, rather than decoding from the beginning), the video can be appropriately decoded from an IDR picture near the specified point in time.

[0004] In the standardization of HEVC (High Efficiency Video Coding), the next-generation image coding standard following H.264 / AVC, it is proposed to identify CRA (Clean Random Access) pictures by the value of the NAL unit type, in addition to IDR pictures. A CRA picture is a picture consisting of only I-slices in the middle of a sequence. A picture following a CRA picture in both decoding order and display order does not refer to a picture preceding the CRA picture in decoding order or a picture preceding the CRA picture in display order. Therefore, when random access to a CRA picture in the middle of a video (decoding a video from a CRA picture) is performed, the decoding process of the picture following the CRA picture in display order can be performed without failure.

[0005] Here, Non-Patent Document 1 discloses a standard for storing reference image information in the Header of a Tile group in order to specify a reference image for each Tile group image of a Multi-Tile image. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] 14496-15:2014, JVET-M1001 Summary of the Invention [Problem to be solved by the invention]

[0007] However, as disclosed in the above-mentioned Non-Patent Document 1, in the past, the reference image information of all Tile groups must be the same, so it was redundant to specify a reference image for each Tile group. Also, in the past, when it was desired to update a reference image as a Long Term Reference, it was not possible to change only the Long Term Reference. For this reason, there was a demand to extend the conventional standard to improve its practicality.

[0008] The present disclosure has been made in consideration of such circumstances, and is intended to make it possible to further improve practicality. [Means for solving the problem]

[0009] An image decoding device according to a first aspect of the present disclosure includes a decoding unit configured to decode an image of a bitstream consisting of an access unit in which at least one NAL unit is arranged, and reference image information indicating a reference image referenced by a current picture is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit that is provided separately from an AUD NAL unit that indicates the boundary of the access unit and is applied to all Tile group NAL units within the access unit, and the identification NAL unit is configured such that only Long Term can be independently changed in settings.

[0010] An image decoding method according to a first aspect of the present disclosure includes an image decoding device performing an image decoding process, decoding an image of a bitstream consisting of access units in which at least one NAL unit is arranged, wherein reference image information indicating a reference image referenced by a current picture is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit that is provided separately from an AUD NAL unit that indicates the boundary of the access unit and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured such that only Long Term can be independently changed in settings.

[0011] In a first aspect of the present disclosure, an image of a bitstream consisting of an access unit in which at least one or more NAL units are arranged is decoded, and reference image information indicating a reference image to which a current picture refers is stored in a header area of ​​the access unit. The reference image information is stored in an identification NAL unit provided separately from an AUD NAL unit indicating a boundary between access units, and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured so that only Long Term can be independently changed.

[0012] An image coding device according to a second aspect of the present disclosure includes an encoding unit configured to encode an image of a bitstream consisting of an access unit in which at least one NAL unit is arranged, and reference image information indicating a reference image to which a current picture refers is stored in a header area of ​​the access unit, the reference image information being stored in an identification NAL unit provided separately from an AUD NAL unit indicating a boundary of the access unit and being applied to all Tile group NAL units within the access unit, and the identification NAL unit being configured such that only Long Term can be independently changed in settings.

[0013] An image coding method according to a second aspect of the present disclosure includes an image coding device performing an image coding process, coding an image of a bit stream including an access unit in which at least one NAL unit is arranged, wherein reference image information indicating a reference image to which a current picture refers is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit provided separately from an AUD NAL unit indicating a boundary of the access unit, and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured such that only Long Term can be independently changed.

[0014] A second aspect of the present disclosure includes encoding an image of a bitstream consisting of an access unit in which at least one or more NAL units are arranged, and reference image information indicating a reference image to which a current picture refers is stored in a header area of ​​the access unit. The reference image information is stored in an identification NAL unit provided separately from an AUD NAL unit indicating a boundary of the access unit, and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured so that only Long Term can be independently changed. [Brief description of the drawings]

[0015] [Figure 1] FIG. [Diagram 2] 1 is a block diagram showing an example configuration of an embodiment of an image encoding device to which the present technology is applied. [Diagram 3] 1 is a block diagram showing an example configuration of an embodiment of an image decoding device to which the present technology is applied. [Figure 4] FIG. 11 is a diagram showing an example of the configuration of a bitstream in a first variation of the first embodiment. [Diagram 5] A figure showing an example of a description of the syntax of an AUD NAL unit in the first variation of the first form. [Figure 6] 11 is a flowchart illustrating a NAL generation process of the image encoding process. [Figure 7] 13 is a flowchart illustrating an AUD encoding process. [Figure 8] 13 is a flowchart illustrating a tile group encoding process. [Figure 9] 11 is a flowchart illustrating a NAL generation process in an image decoding process. [Figure 10] 13 is a flowchart illustrating an AUD decoding process. [Figure 11] 13 is a flowchart illustrating a tile group decoding process. [Figure 12]FIG. 13 is a diagram showing an example of the configuration of a bitstream in a second variation of the first embodiment. [Figure 13] A figure showing an example of the syntax description of a NEW NAL unit in the second variation of the first form. [Figure 14] 11 is a flowchart illustrating a NAL generation process in an image encoding process. [Figure 15] 13 is a flowchart illustrating a NEW encoding process. [Figure 16] 11 is a flowchart illustrating a NAL generation process in an image decoding process. [Figure 17] 13 is a flowchart illustrating a NEW decoding process. [Figure 18] 1A and 1B are diagrams for explaining an image quality restoration technique in a narrow band using a long-term reference; [Figure 19] FIG. 13 is a diagram showing an example of the configuration of a bitstream in a first variation of the second embodiment. [Figure 20] A figure showing an example of description of the syntax of SPS NAL and AUD NAL in the first variation of the second form. [Figure 21] FIG. 13 is a diagram showing an example of the configuration of a bitstream in a second variation of the second embodiment. [Figure 22] FIG. 13 is a diagram showing an example of description of the syntax of SPS NAL and NEW NAL in the second variation of the second form. [Figure 23] 11 is a flowchart illustrating a reference frame list designation / setting process of the image encoding process. [Figure 24] 13 is a flowchart illustrating a process of changing Long Term reference frame information. [Diagram 25] 13 is a flowchart illustrating a reference frame list designation / setting process in the image decoding process. [Figure 26] 13 is a flowchart illustrating a process of changing Long Term reference frame information. [Figure 27]1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <References supporting technical content and technical terms> The scope of the disclosure in this specification is not limited to the contents of the Examples, and the contents of Reference Documents REF1 to REF6 shown in FIG. 1, which were publicly known at the time of filing, are also incorporated by reference into this specification.

[0017] That is, the contents of the reference documents REF1 to REF6 shown in Fig. 1 are also the basis for determining the support requirements. For example, even if the NAL unit structure described in the reference document REF4 and the Byte stream format described in the reference document REF5 are not directly defined in the detailed description of the invention, they are considered to be within the scope of this disclosure and to satisfy the support requirements of the claims. Similarly, even if technical terms such as parsing, syntax, and semantics are not directly defined in the detailed description of the invention, they are considered to be within the scope of this disclosure and to satisfy the support requirements of the claims.

[0018] <Terminology> As used herein, the following terms are defined as follows.

[0019] <Block> A "block" (not a block indicating a processing unit) used in the description as a partial region or processing unit of an image (picture) indicates any partial region in a picture, and its size, shape, characteristics, etc. are not limited, unless otherwise specified. For example, a "block" includes any partial region (processing unit) such as a TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU (Coding Unit), LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), transform block, sub-block, macroblock, tile, or slice.

[0020] <Block size specification> In addition, when specifying the size of such a block, the block size may be specified indirectly in addition to directly. For example, the block size may be specified using identification information for identifying the size. For example, the block size may be specified by a ratio or difference with respect to the size of a reference block (for example, LCU or SCU). For example, when transmitting information specifying a block size as a syntax element or the like, the information may be used to indirectly specify the size as described above. In this way, the amount of information can be reduced, and coding efficiency may be improved. The specification of the block size also includes specification of a range of block sizes (for example, specification of a range of allowable block sizes, etc.).

[0021] <Unit of information processing> The data units in which various information is set and the data units targeted by various processes are arbitrary and are not limited to the above-mentioned examples. For example, these information and processes may be set for each TU (Transform Unit), TB (Transform Block), PU (Prediction Unit), PB (Prediction Block), CU (Coding Unit), LCU (Largest Coding Unit), sub-block, block, tile, slice, picture, sequence, or component, or may target data of these data units. Of course, this data unit may be set for each information or process, and the data units of all information and processes do not need to be unified. The storage location of these pieces of information is arbitrary, and may be stored in the header or parameter set of the above-mentioned data unit. Also, they may be stored in multiple locations.

[0022] <Control information> Control information related to the present technology may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether or not to permit (or prohibit) application of the present technology described above may be transmitted. In addition, for example, control information indicating a target to which the present technology described above is applied (or a target to which it is not applied) may be transmitted. For example, control information that specifies a block size (upper or lower limit, or both), frame, component, or layer to which the present technology is applied (or permitted or prohibited to be applied) may be transmitted.

[0023] <flag> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Thus, the value that this "flag" can take may be, for example, a binary value of 1 / 0, or may be three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, it is assumed that identification information (including flags) is not only included in the bit stream, but also includes difference information of the identification information with respect to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information with respect to the reference information.

[0024] <Associate metadata> Also, various information (metadata, etc.) related to the coded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data, or each piece of data may be made into separate data. For example, information associated with coded data (image) may be transmitted on a transmission path different from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium different from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.

[0025] In this specification, the terms "composite", "multiplex", "add", "integrate", "include", "store", "put in", "insert", "insert" and the like mean combining multiple things into one, for example combining encoded data and metadata into one data, and mean one method of "associating" as described above. In this specification, encoding includes not only the whole process of converting an image into a bitstream, but also a part of the process. For example, it includes not only a process that includes prediction processing, orthogonal transformation, quantization, arithmetic coding, etc., but also a process that collectively refers to quantization and arithmetic coding, a process that includes prediction processing, quantization, and arithmetic coding, and the like. Similarly, decoding includes not only the whole process of converting a bitstream into an image, but also a part of the process. For example, it includes not only a process that includes inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc., but also a process that includes inverse arithmetic decoding and inverse quantization, a process that includes inverse arithmetic decoding, inverse quantization, and prediction processing, and the like.

[0026] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0027] <Image Encoding Device and Image Decoding Device> An image encoding device and an image decoding device to which the present technology is applied will be described with reference to Figs.

[0028] As shown in FIG. 2, the image encoding device 10 includes an encoding unit 11, a determining unit 12, a VCL buffer 13, a non-VCL buffer 14, a file generating unit 15, and a control unit 16.

[0029] The encoding unit 11 is an encoder that operates according to the HEVC standard. The encoding unit 11 acquires an image sequence to be encoded from a video source such as a camera or a television tuner connected to the image encoding device 10. The encoding unit 11 then generates an encoded bitstream by performing various processes such as intra-prediction, inter-frame prediction, orthogonal transformation, quantization, and lossless encoding on each image in the acquired image sequence. Slice data corresponding to the substance of an image is generated as a VCL (Video Coding Layer) NAL unit.

[0030] On the other hand, parameter sets such as a sequence parameter set (SPS), a picture parameter set (PPS), and an adaptation parameter set (APS) can be generated as non-VCL NAL units. The encoding unit 11 outputs the VCL NAL units, i.e., a bit stream of slice data, to the file generation unit 15 via the VCL buffer 13. The encoding unit 11 also outputs the parameter sets to the file generation unit 15 via the non-VCL buffer 14.

[0031] The determination unit 12 determines the type of each picture in the picture sequence to be coded by the coding unit 11. More specifically, in this embodiment, the determination unit 12 determines at least whether each picture is an IDR picture, a CRA picture, or another picture. Both the IDR picture and the CRA picture are pictures consisting of only I slices.

[0032] As described above, an IDR picture is the first picture in a sequence. Pictures following an IDR picture in the same sequence do not refer to pictures before the IDR picture in coding order (decoding order) and are located only after the IDR picture in display order. A CRA picture is a picture located in the middle of a sequence and can be used as a decoding start picture at the time of random access on the decoder side. Pictures following a CRA picture in both coding order (decoding order) and display order do not refer to pictures preceding the CRA picture in coding order (decoding order) and pictures preceding the CRA picture in display order. The determination unit 12 outputs the result of the determination to the encoding unit 11 and the file generation unit 15. The encoding unit 11 adds a NAL unit type indicating the type of each image determined by the determination unit 12 to the NAL header of each NAL unit.

[0033] However, when random access is performed to a CRA picture, only the CRA picture of the random access destination and the picture following the CRA picture in decoding order are the objects of decoding. However, there may be a picture following the CRA picture in decoding order and preceding the CRA picture in display order. In this specification, such a picture is called a leading picture. As can be understood from the definition of a CRA picture, it is permitted for the leading picture to refer to a picture preceding the CRA picture in decoding order.

[0034] When a random access is performed to a CRA picture, a preceding picture that refers to a picture preceding the CRA picture in the decoding order is not decoded correctly. This is because the reference picture of the preceding picture has not been decoded. That is, when a random access is performed, whether or not the preceding picture to be decoded can be decoded correctly depends on the reference relationship of the preceding picture. Therefore, the determination unit 12 may further determine the preceding picture that cannot be decoded correctly when the random access is performed to each CRA picture, and provide the determination result to the file generation unit 15.

[0035] The VCL buffer 13 buffers the VCL NAL units. The slice data of the CRA picture is buffered by the VCL buffer 13.

[0036] The non-VCL buffer 14 buffers the non-VCL NAL units.

[0037] The file generation unit 15 generates a video file that stores a series of encoded image data according to a file format including a header area and a data area, and outputs the file to the storage unit 20. In this specification, an example in which the MP4 format is used as the file format will be mainly described. However, the technology according to the present disclosure is not limited to this example, and can be applied to other types of video file formats having a header area and a data area.

[0038] In the MP4 format, data is stored in objects called boxes, and recorded on an object-by-object basis. Within a file, boxes form a tree structure, with parent boxes containing child boxes. Each box type is identified by a four-letter alphabetic identifier.

[0039] More specifically, the file generation unit 15 inserts a bit stream of slice data corresponding to a VCL NAL unit in decoding order into a data area (e.g., an mdat box) of the video file. The file generation unit 15 also inserts one or more parameter sets corresponding to non-VCL NAL units into a header area (e.g., a moov box) of the video file. Furthermore, in this embodiment, the file generation unit 15 inserts CRA information for identifying one or more CRA pictures determined by the determination unit 12 into the header area of ​​the video file. Furthermore, the file generation unit 15 may include, in the CRA information, leading picture information for identifying leading pictures that are not normally decoded when random access is performed to each CRA picture.

[0040] The control unit 16 controls the encoding process executed in the image encoding device 10. For example, when an instruction to start encoding is detected, the control unit 16 causes the encoding unit 11 to encode a specified image sequence. Furthermore, the control unit 16 causes the file generation unit 15 to generate a video file that stores image data encoded by the encoding unit 11. The control unit 16 may use a hypothetical decoder model called HRD (Hypothetical Reference Decoder) to control the generation of the encoded stream so as not to cause the decoder buffer to collapse.

[0041] As shown in FIG. 3, the image decoding device 30 includes a VCL buffer 31, a non-VCL buffer 32, a parameter memory 33, a decoding unit 34, an output buffer 35, and a control unit 37.

[0042] The VCL buffer 31 buffers a bit stream of image data (typically, slice data) read from a data area (eg, an mdat box) of a video file stored in the storage unit 20.

[0043] The non-VCL buffer 32 buffers parameter sets such as SPS, PPS, and APS, which are read from the header area (eg, moov box) of the moving image file stored in the storage unit 20, as well as header information such as CRA information.

[0044] The parameter memory 33 collectively stores information in the header area of ​​a file acquired via the non-VCL buffer 32. The CRA information that can be recorded in the header area of ​​a moving image file in the various formats described above is held by the parameter memory 33 while the moving image file is open.

[0045] The decoding unit 34 is a decoder that operates according to the HEVC standard. The decoding unit 34 decodes an image sequence from a bit stream obtained from the data area of ​​the video file via the VCL buffer 31. When decoding images, the decoding unit 34 uses parameters in a parameter set stored in the parameter memory 33. The decoding unit 34 rearranges the images in the decoded image sequence in display order, and outputs the rearranged images to the output buffer 35.

[0046] The decoding unit 34 normally accesses slice data stored in a video track in a video file in decoding order, starting from the beginning of the slice data. However, when a random access instruction is detected by the control unit 37, the decoding unit 34 randomly accesses a decoding start picture (in the middle of the video track) specified by the control unit 37, and decodes the image sequence from the decoding start picture. The decoding start picture is one of the IDR picture and the CRA picture in the video track.

[0047] The output buffer 35 is a decoded picture buffer (DPB) for buffering the image decoded by the decoding unit 34. The image buffered by the output buffer 35 is output to a display or a processor (not shown) at the output timing of the image.

[0048] The control unit 37 controls the image decoding process executed in the image decoding device 30. For example, the control unit 37 opens a moving image file stored in the storage unit 20 in response to an instruction from a user, and causes the decoding unit 34 to start decoding the image sequence. When a random access instruction is detected, the control unit 37 uses the CRA information to specify any CRA picture in the image sequence as a decoding start picture. Then, the control unit 37 causes the decoding unit 34 to decode the image sequence from the specified decoding start picture (i.e., from the middle of the moving image track).

[0049] The control unit 37 typically identifies the CRA picture located closest to the timing specified in the random access instruction (for example, the timing indicated by the operated pointer of the seek bar of the video playback window) as the decoding start picture.

[0050] When the CRA information includes the above-mentioned leading picture information, the control unit 37 may skip output from the output buffer 35 (and decoding by the decoding unit 34) of a leading picture that is not correctly decoded and is identified using the leading picture information. By using the leading picture information, it is possible to prevent a corrupted image from being displayed on a display or output to an external processor. In this case, the control unit 37 does not need to judge after the fact whether each image has been correctly decoded or not.

[0051] In addition, the control unit 37 can transmit a command to the control unit 16 of the image encoding device 10.

[0052] <First form of bitstream> A first variation of the first form of the bitstream will be described with reference to Figs.

[0053] As shown in Fig. 4, a bit stream is composed of multiple access units (AU), and at least one NAL unit is arranged in each access unit corresponding to one frame. There are multiple types of NAL units, such as an AUD (Access Unit Delimiter) NAL unit, an SPS (Sequence Parameter Set) NAL unit, a PPS (Picture Parameter Set) NAL unit, and a Tile group NAL unit.

[0054] The AUD NAL unit indicates the boundary of an access unit, and generally, only one AUD NAL unit is placed at the beginning of each access unit. Note that the current AUD already has a syntax that indicates the attributes of all Tile group NALs in an access unit.

[0055] The SPS NAL unit stores sequence parameters required for playing bitstreams. The PPS NAL unit stores sequence parameters required for playing pictures. The Tile group NAL unit stores images for each tile group.

[0056] The bitstream configuration can be assumed to have three cases as shown in the figure.

[0057] For example, in the first case, in all access units, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, and Tile group NAL units are placed consecutively thereafter. That is, in the first case, an SPS NAL unit and a PPS NAL unit are assigned to each access unit.

[0058] Also, in the second case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, and a PPS NAL unit is placed third, with Tile group NAL units placed consecutively thereafter. Then, in the second and subsequent access units, an AUD NAL unit is placed first, a PPS NAL unit is placed second, and Tile group NAL units are placed consecutively thereafter. That is, in the second case, an SPS NAL unit is added to the first access unit, and a PPS NAL unit is added to each access unit.

[0059] In the third case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, and a PPS NAL unit is placed third, with Tile group NAL units placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, with Tile group NAL units placed consecutively thereafter. In other words, in the third case, an SPS NAL unit and a PPS NAL unit are assigned only to the first access unit.

[0060] In the first variation of the first form of the bitstream, in any of the first to third cases, reference image information indicating a reference image to which the current picture refers is arranged in an AUD NAL unit, which is a change from the conventional configuration. For example, this reference image information can specify ref_pic_list_idx in tile_group_header or set a new ref_pic_list_struct. In this way, by arranging reference image information in the AUD NAL unit in the header area of ​​the access unit and applying the reference image information to all Tile group NAL units in the access unit, redundancy can be avoided compared to when a reference image is specified for each Tile group.

[0061] Specifically, in the first variation of the first form of the bitstream, the syntax of the AUD NAL is described as shown in Fig. 5. As shown in the figure, in the syntax of the AUD NAL, the syntax of pic_type is extended, and it is possible to specify ref_pic_list_idx and set a new ref_pic_list_struct.

[0062] An example of NAL generation processing in the image encoding processing executed by the image encoding device 10 will be described with reference to the flowchart shown in FIG.

[0063] For example, when an image sequence is supplied to the image encoding device 10, the NAL generation process is started, and in step S11, the encoding unit 11 determines whether or not there is an NAL unit to be generated from the image sequence.

[0064] If the encoding unit 11 determines in step S11 that there is a NAL unit to be generated from the image sequence, the process proceeds to step S12.

[0065] In step S12, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is an AUD NAL unit.

[0066] In step S12, if the encoding unit 11 determines that the NAL unit to be generated from the image sequence is an AUD NAL unit, the process proceeds to step S13. Then, in step S13, the encoding unit 11 performs an AUD encoding process to generate an AUD NAL unit including reference image information, and supplies the AUD NAL unit to the file generating unit 15 via the non-VCL buffer 14. After that, the process returns to step S11, and the same process is repeated thereafter. On the other hand, in step S12, if the encoding unit 11 determines that the NAL unit to be generated from the image sequence is not an AUD NAL unit, the process proceeds to step S14.

[0067] In step S14, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is an SPS NAL unit.

[0068] If the encoding unit 11 determines in step S14 that the NAL unit to be generated from the image sequence is an SPS NAL unit, the process proceeds to step S15. Then, in step S15, the encoding unit 11 performs an SPS encoding process to generate an SPS NAL unit and supplies the SPS NAL unit to the file generation unit 15 via the non-VCL buffer 14, and then the process returns to step S11, and the same process is repeated thereafter. On the other hand, if the encoding unit 11 determines in step S14 that the NAL unit to be generated from the image sequence is not an SPS NAL unit, the process proceeds to step S16.

[0069] In step S16, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a PPS NAL unit.

[0070] If the encoding unit 11 determines in step S16 that the NAL unit to be generated from the image sequence is a PPS NAL unit, the process proceeds to step S17. Then, in step S17, the encoding unit 11 performs a PPS encoding process to generate a PPS NAL unit, and supplies the PPS NAL unit to the file generation unit 15 via the non-VCL buffer 14. After that, the process returns to step S11, and the same process is repeated thereafter. On the other hand, if the encoding unit 11 determines in step S16 that the NAL unit to be generated from the image sequence is not a PPS NAL unit, the process proceeds to step S18.

[0071] In step S18, the encoding unit 11 determines whether or not the NAL unit to be generated from the image sequence is a Tile group NAL unit.

[0072] If the encoding unit 11 determines in step S18 that the NAL unit to be generated from the image sequence is a Tile group NAL unit, the process proceeds to step S19. Then, in step S19, the encoding unit 11 performs a Tile group encoding process to generate a Tile group NAL unit and supplies it to the file generation unit 15 via the VCL buffer 13, after which the process returns to step S11, and the same process is repeated thereafter. On the other hand, if the encoding unit 11 determines in step S18 that the NAL unit to be generated from the image sequence is not a Tile group NAL unit, the process proceeds to step S20.

[0073] In step S20, the encoding unit 11 performs an encoding process for encoding an NAL unit other than any of the above-mentioned NAL units, and then the process returns to step S11, and the same process is repeated thereafter.

[0074] On the other hand, if the encoding unit 11 determines in step S11 that there is no NAL unit to be generated from the image sequence, the NAL generation process is terminated.

[0075] FIG. 7 is a flowchart illustrating the AUD encoding process performed in step S13 of FIG.

[0076] In step S31, the encoding unit 11 performs processing to specify and set a reference frame list using reference image information, and can, for example, specify ref_pic_list_idx in tile_group_header or set a new ref_pic_list_struct.

[0077] In step S32, the encoding unit 11 generates an AUD NAL unit by performing a frame delimiting process for delimiting frames (access units), and then the AUD encoding process is terminated.

[0078] FIG. 8 is a flowchart illustrating the Tile group encoding process performed in step S19 of FIG.

[0079] In step S41, the encoding unit 11 performs processing for encoding a header other than processing for specifying and setting a reference frame list.

[0080] In step S42, the encoding unit 11 performs the encoding process for encoding the header and subsequent parts, and then the Tile group encoding process is terminated.

[0081] Through the above-described NAL generation process, the image encoding device 10 generates an AUD NAL unit in which reference image information is placed, and generates a video file consisting of a bitstream as described above with reference to FIG. 4, i.e., encodes an image of a bitstream consisting of access units in which at least one NAL unit is placed.

[0082] An example of NAL generation processing in the image decoding processing executed by the image decoding device 30 will be described with reference to the flowchart shown in FIG.

[0083] For example, the process starts when the image decoding device 30 reads out the bitstream stored in the storage unit 20, and in step S51, the decoding unit 34 determines whether or not there is an NAL unit to be decoded from the bitstream.

[0084] In step S51, if the decoding unit 34 determines that there is a NAL unit to be decoded from the bitstream, the process proceeds to step S52.

[0085] In step S52, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is an AUD NAL unit.

[0086] In step S52, if the decoding unit 34 determines that the NAL unit to be decoded from the bitstream is an AUD NAL unit, the process proceeds to step S53. Then, in step S53, the decoding unit 34 performs an AUD decoding process, and supplies reference image information obtained by decoding the AUD NAL unit to the parameter memory 33, and then the process returns to step S51, and the same process is repeated thereafter. On the other hand, in step S52, if the decoding unit 34 determines that the NAL unit to be decoded from the bitstream is not an AUD NAL unit, the process proceeds to step S54.

[0087] In step S54, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is an SPS NAL unit.

[0088] If the decoding unit 34 determines in step S54 that the NAL unit to be decoded from the bitstream is an SPS NAL unit, the process proceeds to step S55. Then, in step S55, the decoding unit 34 performs the SPS decoding process to decode the SPS NAL unit and supplies the acquired parameters to the parameter memory 33, and then the process returns to step S51, and the same process is repeated thereafter. On the other hand, if the decoding unit 34 determines in step S54 that the NAL unit to be decoded from the bitstream is not an SPS NAL unit, the process proceeds to step S56.

[0089] In step S56, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is a PPS NAL unit.

[0090] If the decoding unit 34 determines in step S56 that the NAL unit to be decoded from the bitstream is a PPS NAL unit, the process proceeds to step S57. Then, in step S57, the decoding unit 34 performs PPS decoding processing to decode the PPS NAL unit and supplies the acquired parameters to the parameter memory 33, and then the process returns to step S51, and the same process is repeated thereafter. On the other hand, if the decoding unit 34 determines in step S56 that the NAL unit to be decoded from the bitstream is not a PPS NAL unit, the process proceeds to step S58.

[0091] In step S58, the decoding unit 34 determines whether or not the NAL unit to be decoded from the bitstream is a Tile group NAL unit.

[0092] If the decoding unit 34 determines in step S58 that the NAL unit to be decoded from the bitstream is a Tile group NAL unit, the process proceeds to step S59. Then, in step S59, the decoding unit 34 performs Tile group decoding process to supply an image obtained by decoding the Tile group NAL unit to the output buffer 35, and then the process returns to step S51, and the same process is repeated thereafter. On the other hand, if the decoding unit 34 determines in step S58 that the NAL unit to be decoded from the bitstream is not a Tile group NAL unit, the process proceeds to step S60.

[0093] In step S60, the decoding unit 34 performs a decoding process for decoding NAL units other than any of the above-mentioned NAL units, and then the process returns to step S51, and the same process is repeated thereafter.

[0094] On the other hand, if the decoding unit 34 determines in step S51 that there is no NAL unit to be decoded from the bitstream, the NAL generation process is terminated.

[0095] FIG. 10 is a flowchart illustrating the AUD decoding process performed in step S53 of FIG.

[0096] In step S71, the decoding unit 34 obtains the picture type from the AUD NAL unit.

[0097] In step S72, the decoding unit 34 acquires reference image information from the AUD NAL unit, and performs processing to specify and set a reference frame list in accordance with the reference image information.

[0098] In step S73, the decoding unit 34 decodes the AUD NAL unit by performing a frame delimiting process for delimiting frames (access units), and then the AUD decoding process is terminated.

[0099] FIG. 11 is a flowchart illustrating the Tile group decoding process performed in step S59 of FIG.

[0100] In step S81, the decoding unit 34 performs a process of decoding a header other than the process of specifying and setting a reference frame list.

[0101] In step S82, the decoding unit 34 performs a decoding process for decoding the header and subsequent parts, and then the Tile group decoding process is terminated.

[0102] As described above, image decoding device 30 can decode images according to reference image information obtained from the AUD NAL unit, i.e., decode images of a bitstream consisting of access units in which at least one NAL unit is placed.

[0103] A second variation of the first form of the bitstream will now be described with reference to Figures 12 to 17.

[0104] As shown in Fig. 12, the bitstream is composed of a plurality of access units, and at least one NAL unit is arranged in each access unit, as described above with reference to Fig. 4. Also, as in the first variation of the first form of the bitstream (see Fig. 4), three cases are assumed for the bitstream configuration, as shown in Fig. 12.

[0105] In the second variation of the first form of the bitstream, a NEW NAL unit is used as the NAL unit in addition to the AUD NAL unit, the SPS NAL unit, the PPS NAL unit, and the Tile group NAL unit.

[0106] For example, in the first case, in all access units, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, a NEW NAL unit is placed fourth, and Tile group NAL units are placed consecutively thereafter. That is, in the first case, an SPS NAL unit and a PPS NAL unit are assigned to each access unit, and further a NEW NAL unit is assigned to each access unit.

[0107] In the second case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, a NEW NAL unit is placed fourth, and Tile group NAL units are placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, a PPS NAL unit is placed second, and a NEW NAL unit is placed third, and Tile group NAL units are placed consecutively thereafter. In other words, in the second case, an SPS NAL unit is added to the first access unit, and a PPS NAL unit and a NEW NAL unit are added to each access unit.

[0108] In the third case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, a NEW NAL unit is placed fourth, and Tile group NAL units are placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, a NEW NAL unit is placed second, and Tile group NAL units are placed consecutively thereafter. In other words, in the third case, an SPS NAL unit and a PPS NAL unit are assigned only to the first access unit, and a NEW NAL unit is assigned to each access unit.

[0109] In the second variation of the first form of the bitstream, in any of the first to third cases, reference image information indicating a reference image to which the current picture refers is arranged in the NEW NAL unit, which is a change from the conventional configuration. For example, this reference image information can specify ref_pic_list_idx in the tile_group_header or set a new ref_pic_list_struct. In this way, by arranging reference image information in the NEW NAL unit (identification NAL unit) in the header area of ​​the access unit and applying the reference image information to all Tile group NAL units in the access unit, it is possible to avoid redundancy compared to when a reference image is specified for each Tile group.

[0110] Specifically, in the second variation of the first form of the bitstream, the syntax of the NEW NAL unit is described as shown in Fig. 13. As shown in the figure, in the syntax of the NEW NAL, it is possible to specify ref_pic_list_idx and set a new ref_pic_list_struct.

[0111] An example of NAL generation processing performed in the image encoding processing executed by the image encoding device 10 will be described with reference to the flowchart shown in FIG.

[0112] In steps S101 to S107, the same processes as those in steps S11 to S17 described above with reference to the flowchart in Fig. 6 are performed. Then, in step S108, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a NEW NAL unit.

[0113] If the encoding unit 11 determines in step S108 that the NAL unit to be generated from the image sequence is a NEW NAL unit, the process proceeds to step S109. Then, in step S109, the encoding unit 11 performs a NEW encoding process to generate a NEW NAL unit including reference image information, and supplies the NEW NAL unit to the file generation unit 15 via the non-VCL buffer 14. After that, the process returns to step S101, and the same process is repeated thereafter. On the other hand, if the encoding unit 11 determines in step S108 that the NAL unit to be generated from the image sequence is not a NEW NAL unit, the process proceeds to step S110.

[0114] Then, in steps S110 to S112, the same processes as in steps S18 to S20 described above with reference to the flowchart in FIG. 6 are performed, and then the NAL generation process ends.

[0115] FIG. 15 is a flowchart illustrating the NEW encoding process performed in step S109 of FIG.

[0116] In step S121, the encoding unit 11 performs a process of specifying and setting a reference frame list by reference image information, for example, specifying ref_pic_list_idx in tile_group_header, setting a new ref_pic_list_struct, etc. In this way, the encoding unit 11 generates a NEW NAL unit, and then the NEW encoding process is terminated.

[0117] The Tile group encoding process performed in step S111 in FIG. 14 is performed in the same manner as the Tile group encoding process described above with reference to the flowchart in FIG.

[0118] Through the above-described NAL generation process, the image encoding device 10 generates a NEW NAL unit in which reference image information is placed, and generates a video file consisting of a bitstream as described with reference to Figure 12 above, i.e., it is possible to encode an image of a bitstream consisting of access units in which at least one NAL unit is placed.

[0119] An example of NAL generation processing in the image decoding processing executed by the image decoding device 30 will be described with reference to the flowchart shown in FIG.

[0120] In steps S131 to S137, the same processes as in steps S51 to S57 described above with reference to the flowchart in Fig. 9 are performed. Then, in step S138, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is a NEW NAL unit.

[0121] If the decoding unit 34 determines in step S138 that the NAL unit to be decoded from the bitstream is a NEW NAL unit, the process proceeds to step S139. Then, in step S139, the decoding unit 34 performs a NEW decoding process, and supplies reference image information obtained by decoding the NEW NAL unit to the parameter memory 33, and then the process returns to step S131, and the same process is repeated thereafter. On the other hand, if the decoding unit 34 determines in step S138 that the NAL unit to be decoded from the bitstream is not an AUD NAL unit, the process proceeds to step S140.

[0122] Then, in steps S140 to S142, the same processes as in steps S58 to S60 described above with reference to the flowchart in FIG. 9 are performed, and then the NAL generation process ends.

[0123] FIG. 17 is a flowchart illustrating the NEW decoding process performed in step S139 of FIG.

[0124] In step S151, the decoding unit 34 obtains reference image information from the NEW NAL unit, and decodes the NEW NAL unit by performing a process of specifying and setting a reference frame list according to the reference image information, and then the NEW decoding process is terminated.

[0125] The Tile group decoding process performed in step S141 in FIG. 16 is performed in the same manner as the Tile group decoding process described above with reference to the flowchart in FIG.

[0126] As described above, the image decoding device 30 can decode an image according to the reference image information obtained from the NEW NAL unit, i.e., decode an image of a bitstream consisting of an access unit in which at least one NAL unit is placed.

[0127] <Second Form of Bitstream> The second form of the bitstream will now be described with reference to Figs.

[0128] In the second form of the bitstream, the Long Term Reference, which can be set irregularly, can be dynamically changed.

[0129] First, with reference to FIG. 18, a method of restoring image quality in a narrow band using a long-term reference will be described as an example of dynamically changing a long-term reference.

[0130] For example, when a packet loss occurs at the timing shown in the figure, the image decoding device 30 on the receiving side transmits error occurrence information indicating the occurrence of a packet loss to the image coding device 10 on the transmitting side. Then, the image coding device 10 avoids retransmission of the I-Picture and encodes the frame P' to be encoded after receiving the error occurrence information as a Long Term Reference Picture (LTR frame P'). L ) and encode it.

[0131] Generally, the Long Term Reference Picture is set (updated) irregularly when the motion is relatively large. For example, the Short Term Reference Picture does not need to be changed because it is encoded in a fixed GOP structure, whereas the Long Term Reference Picture can decide whether to change when encoding a picture.

[0132] In this way, by dynamically changing the Long Term Reference, the image decoding device 30 can decode the LTR frame P even if a packet loss occurs. L The encoded frame P' is denoted by LTR frame P L By decoding the image with reference to the above, it is possible to recover from degradation of image quality due to packet loss with less delay.

[0133] Such image quality restoration techniques using long-term references can be used in low-latency or interactive applications with limited bandwidth, such as surveillance cameras, video chat, and streaming and gaming services.

[0134] A first variation of the second form of the bitstream will be described with reference to Figs.

[0135] As shown in Fig. 19, the bitstream is composed of a plurality of access units, as described above with reference to Fig. 4, and at least one NAL unit (AUD NAL unit, SPS NAL unit, PPS NAL unit, and Tile group NAL unit) is arranged in each access unit. As in the first variation of the first form of the bitstream (see Fig. 4), the bitstream may be composed in three cases as shown in Fig. 19.

[0136] In the first variation of the second form of the bitstream, in any of the first to third cases, the AUD NAL unit is extended so that only the Long Term can be independently changed (modify poc_lsb_lt of ref_pic_list_struct(i, ref_pic_list_idx[i]).

[0137] Specifically, as shown in FIG. 20, in the first variation of the second form of the bitstream, the syntax of SPS NAL and AUD NAL is described.

[0138] For example, the long term of the ref_pic_list_struct defined in the SPS NAL unit can be changed in the AUD NAL unit.

[0139] A second variation of the second form of the bitstream will now be described with reference to Figs.

[0140] As shown in Fig. 21, the bitstream is composed of a plurality of access units, as described above with reference to Fig. 4, and at least one NAL unit (AUD NAL unit, SPS NAL unit, PPS NAL unit, Tile group NAL unit, and NEW NAL unit) is arranged in each access unit. Also, as in the second variation of the first form of the bitstream (see Fig. 12), three cases are assumed for the bitstream configuration as shown in Fig. 21.

[0141] In the second variation of the second form of the bitstream, in any of the first to third cases, the NEW NAL unit is extended so that only the Long Term can be independently changed (modify poc_lsb_lt of ref_pic_list_struct(i, ref_pic_list_idx[i]).

[0142] Specifically, as shown in FIG. 22, in the second variation of the second form of the bitstream, the syntax of SPS NAL and NEW NAL is described.

[0143] For example, the long term of the ref_pic_list_struct already defined in the SPS NAL unit can be changed in the NEW NAL unit.

[0144] An example of a reference frame list designation / setting process performed in the image encoding process executed by the image encoding device 10 will be described with reference to the flowchart shown in FIG.

[0145] For example, when an image sequence is supplied to the image encoding device 10, the reference frame list setting process is started, and in step S161, the encoding unit 11 determines whether or not to specify a predefined reference frame list in an SPS NAL unit.

[0146] If it is determined in step S161 that a predefined reference frame list is to be specified in the SPS NAL unit, the process proceeds to step S162, where the encoding unit 11 obtains an index of the reference frame list.

[0147] In step S163, the encoding unit 11 determines whether or not to change the long-term reference frame information, in accordance with the index of the reference frame list acquired in step S162.

[0148] If the encoding unit 11 determines in step S163 that the long-term reference frame information is to be changed, the process proceeds to step S164, where a process of changing the long-term reference frame information is performed (see the flowchart in FIG. 24, which will be described later).

[0149] On the other hand, if it is determined in step S163 that the Long Term reference frame information is not to be changed, or after the change process of the Long Term reference frame information in step S164 is performed, the reference frame list setting process ends.

[0150] On the other hand, if it is determined in step S161 that the defined reference frame list in the SPS NAL unit is not specified, the process proceeds to step S165. In step S165, after the encoding unit 11 performs the encoding process of the reference frame list, the reference frame list specification / setting process ends.

[0151] FIG. 24 is a flowchart for explaining the change process of the Long Term reference frame information performed in step S164 of FIG. 23.

[0152] In step S171, the encoding unit 11 obtains the Long Term number k of the reference frame list of the index obtained in step S162 of FIG. 23, and sets 0 to the parameter i (i = 0).

[0153] In step S172, the encoding unit 11 determines whether the parameter i is less than the Long Term number k.

[0154] In step S172, if the encoding unit 11 determines that the parameter i is less than the Long Term number k (i < k), the process proceeds to step S173.

[0155] In step S173, the encoding unit 11 changes the absolute position of the i-th Long Term reference frame. For example, the encoding unit 11 can set the absolute position of the i-th Long Term reference frame based on a command (information specifying a reference image) from the image decoding device 30. Also, the image decoding device 30 updates the information specifying the reference image for each frame.

[0156] In step S174, the encoding unit 11 increments the parameter i (i=i+1), and then the process returns to step S172.

[0157] On the other hand, if the encoding unit 11 determines in step S172 that the parameter i is not less than the Long Term number k (i≧k), the process of changing the Long Term reference frame information is terminated.

[0158] By using the above-described reference frame list specification and setting process, the image encoding device 10 can change the long-term reference frame information and generate a video file consisting of a bitstream such as that described with reference to Figure 19 or Figure 21 above.

[0159] An example of a reference frame list designation / setting process performed in the image decoding process executed by the image decoding device 30 will be described with reference to the flowchart shown in FIG.

[0160] For example, the process starts when the image decoding device 30 reads out a bitstream stored in the storage unit 20, and in step S181, the decoding unit 34 determines whether or not to specify a predefined reference frame list in the SPS NAL unit.

[0161] If it is determined in step S181 that a predefined reference frame list is to be specified in the SPS NAL unit, the process proceeds to step S182, where the decoding unit 34 obtains an index of the reference frame list.

[0162] In step S183, the decoding unit 34 determines whether or not to change the long-term reference frame information, according to the index of the reference frame list acquired in step S182.

[0163] In step S183, if the decoding unit 34 determines to change the Long Term reference frame information, the process proceeds to step S184, and the process of changing the Long Term reference frame information (see the flowchart in FIG. 26 described later) is performed.

[0164] On the other hand, in step S183, if it is determined not to change the Long Term reference frame information, or after the process of changing the Long Term reference frame information in step S184 is performed, the reference frame list setting process ends.

[0165] On the other hand, in step S181, if it is determined that the reference frame list defined in the SPS NAL unit is not specified, the process proceeds to step S185. In step S185, after the decoding unit 34 performs the decoding process of the reference frame list, the reference frame list specifying / setting process ends.

[0166] FIG. 26 is a flowchart for explaining the process of changing the Long Term reference frame information performed in step S184 of FIG. 25.

[0167] In step S191, the decoding unit 34 obtains the Long Term number k of the reference frame list of the index obtained in step S182 of FIG. 25, and sets 0 to the parameter i (i = 0).

[0168] In step S192, the decoding unit 34 determines whether the parameter i is less than the Long Term number k.

[0169] In step S192, if the decoding unit 34 determines that the parameter i is less than the Long Term number k (i < k), the process proceeds to step S193.

[0170] In step S193, the decoding unit 34 changes the absolute position of the i-th Long Term reference frame according to the Long Term reference frame information.

[0171] In step S194, the decoding unit 34 increments the parameter i (i=i+1), and then the process returns to step S192.

[0172] On the other hand, if the decoding unit 34 determines in step S192 that the parameter i is not less than the Long Term number k (i≧k), the process of changing the Long Term reference frame information is terminated.

[0173] By the above-described reference frame list specification / setting process, the image decoding device 30 can decode images according to the Long Term reference frame information that has been changed in a video file consisting of a bitstream such as that described with reference to Figure 19 or Figure 21 above.

[0174] <Computer configuration example> Next, the above-mentioned series of processes (image decoding method and image encoding method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0175] FIG. 27 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.

[0176] The program can be recorded in advance on the hard disk 105 or the ROM 103 as a recording medium built into the computer.

[0177] Alternatively, the program can be stored (recorded) in a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0178] The program can be installed in the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0179] The computer includes a built-in CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0180] When a command is input by a user operating an input unit 107 via an input / output interface 110, the CPU 102 executes a program stored in a ROM (Read Only Memory) 103 in accordance with the command. Alternatively, the CPU 102 loads a program stored in a hard disk 105 into a RAM (Random Access Memory) 104 and executes the program.

[0181] As a result, CPU 102 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. CPU 102 then outputs the processing result from output unit 106 via input / output interface 110, transmits the result from communication unit 108, and further records the result on hard disk 105, as necessary.

[0182] The input unit 107 includes a keyboard, a mouse, a microphone, etc. The output unit 106 includes an LCD (Liquid Crystal Display), a speaker, etc.

[0183] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0184] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0185] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0186] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0187] Furthermore, for example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0188] Furthermore, for example, the above-mentioned program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0189] Also, for example, each step described in the above flow chart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0190] In addition, the processing of the steps of the program executed by the computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a necessary timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, as long as no contradiction occurs. Furthermore, the processing of the steps of the program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0191] In addition, the present technologies described in this specification can be implemented independently and individually, as long as there is no contradiction. Of course, any of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in another embodiment. In addition, any of the present technologies described above can be implemented in combination with other technologies not described above.

[0192] <Examples of configuration combinations> The present technology can also be configured as follows. (1) A decoding unit that decodes an image of a bitstream that includes an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with The header area of ​​the access unit stores reference image information indicating a reference image to which the current picture refers. Image decoding device. (2) The reference image information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image decoding device according to (1) above. (3) The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image decoding device according to (1) above. (4) The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. An image decoding device according to any one of (1) to (3) above. (5) The long term information is information that identifies the reference image that is determined based on information from a client that is a receiver of the bitstream. The image decoding device according to (4) above. (6) The information from the client is updated every frame. The image decoding device according to (5) above. (7) The long term information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimitation of the access unit, and is applied to all Tile group NAL units in the access unit. An image decoding device according to any one of (4) to (6) above. (8) The long term information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimitation of the access unit, and is applied to all Tile group NAL units in the access unit. An image decoding device according to any one of (4) to (6) above. (9) An image decoding device that performs image decoding processing, Decoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged Including, The header area of ​​the access unit stores reference image information indicating a reference image to which the current picture refers. Image decoding method. (10) An encoding unit that encodes an image of a bitstream that includes an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with Reference picture information indicating a reference picture to which the current picture refers is stored in a header area of ​​the access unit. Image encoding device. (11) The reference image information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image encoding device according to (10) above. (12) The reference image information is stored in an identification NAL unit provided separately from an AUD (Access Unit Delimiter) NAL unit indicating the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image encoding device according to (10) above. (13) The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. An image encoding device according to any one of (10) to (12) above. (14) The long term information is information that identifies the reference image that is determined based on information from a client that is a receiver of the bitstream. The image encoding device according to (13) above. (15) The information from the client is updated every frame. The image encoding device according to (14) above. (16) The long term information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimitation of the access unit, and is applied to all Tile group NAL units in the access unit. An image encoding device according to any one of (13) to (15) above. (17) The long term information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimitation of the access unit, and is applied to all Tile group NAL units in the access unit. An image encoding device according to any one of (13) to (15) above. (18) An image encoding device that performs an image encoding process, Encoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is placed Including, Reference picture information indicating a reference picture to which the current picture refers is stored in a header area of ​​the access unit. Image encoding method.

[0193] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the scope of the present disclosure. In addition, the effects described in this specification are merely examples and are not limiting, and other effects may be obtained. [Explanation of symbols]

[0194] 10 image encoding device, 11 encoding unit, 12 determination unit, 13 VCL buffer, 14 non-VCL buffer, 15 file generation unit, 16 control unit, 20 storage unit, 30 image decoding device, 31 VCL buffer, 32 non-VCL buffer, 33 parameter memory, 34 decoding unit, 35 output buffer, 37 control unit

Claims

1. A decoding unit for decoding an image of a bit stream including an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with Reference picture information indicating a reference picture to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units in the access unit, The NAL unit for identification is configured so that only Long Term can be independently changed. Image decoding device.

2. The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. The image decoding device according to claim 1 .

3. The long term information is information that identifies the reference image that is determined based on information from a client that is a receiver of the bitstream. The image decoding device according to claim 2 .

4. The information from the client is updated every frame. The image decoding device according to claim 3 .

5. An image decoding device that performs image decoding processing, Decoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged Including, Reference picture information indicating a reference picture to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units in the access unit, The NAL unit for identification is configured so that only Long Term can be independently changed. Image decoding method.

6. An encoding unit that encodes an image of a bitstream that includes an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with Reference picture information indicating a reference picture to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units in the access unit, The NAL unit for identification is configured so that only Long Term can be independently changed. Image encoding device.

7. The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. The image encoding device according to claim 6.

8. The long term information is information that identifies the reference image that is determined based on information from a client that is a receiver of the bitstream. The image encoding device according to claim 7.

9. The information from the client is updated every frame. The image encoding device according to claim 8.

10. An image encoding device that performs an image encoding process, Encoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is placed Including, Reference picture information indicating a reference picture to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units in the access unit, The NAL unit for identification is configured so that only Long Term can be independently changed. Image encoding method.

Images