Image decoding device, image decoding method, image encoding device, and image encoding method

Abstract

To further improve practicability.SOLUTION: Reference image information indicating a reference image to which the current picture refers is stored in a header area of an access unit. The reference image information is stored in an AUD NAL unit indicating a delimiter of an access unit or a NEW NAL unit provided separately from the AUD NAL unit, and is applied to all Tile group NAL units in the access unit. This technology can be applied to, for example, an image decoding device and an image encoding device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image decoding device, an image decoding method, an image coding device, and an image coding method, and more particularly to an image decoding device, an image decoding method, an image coding device, and an image coding method that can be made more practical. [Background technology]

[0002] In H.264 / AVC, one of the standard specifications for image coding, each image (picture) is divided into one or more slices. Each slice is classified into an I-slice (Intra Slice), a P-slice (Predictive Slice), or a B-slice (Bi-predictive Slice). An I-slice is a slice that is decoded independently without reference to other images. A P-slice is a slice that is decoded by referencing a single other image. A B-slice is a slice that is decoded by referencing multiple other images.

[0003] The first picture in a sequence, consisting only of I-slices, is called an IDR (Instantaneous Decoding Refresh) picture. IDR pictures are identified by a Network Abstraction Layer (NAL) unit type value. Pictures following an IDR picture in the same sequence do not refer to pictures preceding the IDR picture in decoding order, but are only located after the IDR picture in presentation order. Therefore, when random access is attempted to a point in the middle of a video in a coded stream (decoding / playback from the middle, rather than decoding from the beginning of the stream), the video can be properly decoded from an IDR picture near the specified point in time.

[0004] In the standardization of High Efficiency Video Coding (HEVC), the next-generation image coding standard following H.264 / AVC, it is proposed to identify Clean Random Access (CRA) pictures, separate from IDR pictures, by the value of their NAL unit type. A CRA picture is a picture consisting of only I-slices in the middle of a sequence. Pictures that follow a CRA picture in both decoding order and display order do not reference pictures that precede the CRA picture in decoding order or pictures that precede the CRA picture in display order. Therefore, when random access to a CRA picture in the middle of a video (decoding video from a CRA picture) is performed, the decoding process of the pictures that follow the CRA picture in display order can be performed without failure.

[0005] Non-Patent Document 1 discloses a standard for storing reference image information in the Header of a Tile group in order to identify a reference image for each Tile group image of a Multi-Tile image. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] 14496-15:2014, JVET-M1001 Summary of the Invention [Problem to be solved by the invention]

[0007] As disclosed in the above-mentioned Non-Patent Document 1, conventionally, the reference image information of all tile groups must match, so specifying a reference image for each tile group was redundant. Also, conventionally, when updating a reference image as a long-term reference, it was not possible to change only the long-term reference. For this reason, there was a demand for extending the conventional standard to make it more practical.

[0008] The present disclosure has been made in view of such circumstances, and aims to further improve practicality. [Means for solving the problem]

[0009] An image decoding device according to a first aspect of the present disclosure includes a decoding unit that decodes images of a bitstream consisting of access units in which at least one NAL unit is arranged, wherein reference image information indicating the reference image referenced by the current picture is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit that is provided separately from an AUD NAL unit that indicates the boundary of the access unit, and is applied to all Tile group NAL units within the access unit, and the identification NAL unit is configured so that only Long Term can be independently configured.

[0010] An image decoding method according to a first aspect of the present disclosure includes an image decoding device that performs an image decoding process, decoding an image of a bitstream consisting of access units in which at least one NAL unit is arranged, wherein reference image information indicating a reference image referenced by a current picture is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit that is provided separately from an AUD NAL unit that indicates the boundary of the access unit, and is applied to all Tile group NAL units within the access unit, and the identification NAL unit is configured so that only Long Term can be independently configured.

[0011] In a first aspect of the present disclosure, an image of a bitstream consisting of an access unit in which at least one or more NAL units are arranged is decoded, and reference image information indicating a reference image to which a current picture refers is stored in a header area of ​​the access unit. The reference image information is stored in an identification NAL unit provided separately from an AUD NAL unit indicating a boundary of the access unit, and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured so that only Long Term can be independently changed in setting.

[0012] An image coding device according to a second aspect of the present disclosure includes an encoding unit that encodes an image of a bitstream consisting of an access unit in which at least one NAL unit is arranged, wherein reference image information indicating the reference image referenced by the current picture is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit that is provided separately from an AUD NAL unit that indicates the boundary of the access unit, and is applied to all Tile group NAL units within the access unit, and the identification NAL unit is configured so that only Long Term can be independently configured.

[0013] An image coding method according to a second aspect of the present disclosure includes an image coding device that performs an image coding process, and encodes an image of a bitstream consisting of access units in which at least one or more NAL units are arranged, wherein reference image information indicating a reference image referenced by a current picture is stored in a header area of ​​the access unit, the reference image information is stored in an identification NAL unit that is provided separately from an AUD NAL unit that indicates a boundary of the access unit, and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured so that only Long Term can be independently changed.

[0014] A second aspect of the present disclosure includes encoding an image of a bitstream consisting of access units in which at least one or more NAL units are arranged, and reference image information indicating a reference image to which a current picture refers is stored in a header area of ​​the access unit. The reference image information is stored in an identification NAL unit provided separately from an AUD NAL unit indicating a boundary of the access unit, and is applied to all Tile group NAL units in the access unit, and the identification NAL unit is configured so that only Long Term can be independently changed. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 shows references. [Figure 2] 1 is a block diagram showing an example of the configuration of an embodiment of an image encoding device to which the present technology is applied. [Figure 3] 1 is a block diagram showing an example configuration of an embodiment of an image decoding device to which the present technology is applied. [Figure 4] FIG. 10 is a diagram showing an example of the configuration of a bitstream in a first variation of the first form. [Figure 5] A figure showing an example of a description of the syntax of an AUD NAL unit in the first variation of the first form. [Figure 6] 10 is a flowchart illustrating a NAL generation process in the image encoding process. [Figure 7] 10 is a flowchart illustrating an AUD encoding process. [Figure 8] 10 is a flowchart illustrating a tile group encoding process. [Figure 9] 10 is a flowchart illustrating a NAL generation process in an image decoding process. [Figure 10] 10 is a flowchart illustrating an AUD decoding process. [Figure 11] 10 is a flowchart illustrating a tile group decoding process. [Figure 12]FIG. 10 is a diagram showing an example of the configuration of a bitstream in a second variation of the first embodiment. [Figure 13] FIG. 10 is a diagram showing an example of the syntax description of a NEW NAL unit in the second variation of the first form. [Figure 14] 10 is a flowchart illustrating a NAL generation process in an image encoding process. [Figure 15] 10 is a flowchart illustrating a NEW encoding process. [Figure 16] 10 is a flowchart illustrating a NAL generation process in an image decoding process. [Figure 17] 10 is a flowchart illustrating a NEW decoding process. [Figure 18] 10A and 10B are diagrams illustrating a method for restoring image quality in a narrow band using a long-term reference; [Figure 19] FIG. 10 is a diagram showing an example of the configuration of a bitstream in a first variation of the second embodiment. [Figure 20] A figure showing an example of description of the syntax of SPS NAL and AUD NAL in the first variation of the second form. [Figure 21] FIG. 10 is a diagram showing an example of the configuration of a bitstream in a second variation of the second embodiment. [Figure 22] FIG. 10 is a diagram showing an example of syntax description of SPS NAL and NEW NAL in the second variation of the second form. [Figure 23] 10 is a flowchart illustrating a reference frame list designation / setting process in the image encoding process. [Figure 24] 10 is a flowchart illustrating a process of changing Long Term reference frame information. [Figure 25] 10 is a flowchart illustrating a reference frame list designation / setting process in the image decoding process. [Figure 26] 10 is a flowchart illustrating a process of changing Long Term reference frame information. [Figure 27]1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE INVENTION

[0016] <References supporting technical content and technical terminology> The scope of the disclosure in this specification is not limited to the contents of the examples, and the contents of reference documents REF1 to REF6 shown in Figure 1, which were publicly known at the time of filing, are also incorporated by reference into this specification.

[0017] In other words, the contents of References REF1 to REF6 shown in Figure 1 also serve as the basis for determining the support requirements. For example, even if the NAL unit structure described in Reference REF4 and the Byte stream format described in Reference REF5 are not directly defined in the detailed description of the invention, they are considered to be within the scope of this disclosure and satisfy the support requirements of the claims. Similarly, even if technical terms such as parsing, syntax, and semantics are not directly defined in the detailed description of the invention, they are considered to be within the scope of this disclosure and satisfy the support requirements of the claims.

[0018] <Terminology> As used herein, the following terms are defined as follows:

[0019] <block> Unless otherwise specified, a "block" (not a block indicating a processing unit) used in the description as a partial region of an image (picture) or a processing unit refers to any partial region within a picture, and its size, shape, characteristics, etc. are not limited. For example, a "block" includes any partial region (processing unit) such as a transform block (TB), a transform unit (TU), a prediction block (PB), a prediction unit (PU), a smallest coding unit (SCU), a coding unit (CU), a largest coding unit (LCU), a coding tree block (CTB), a coding tree unit (CTU), a transform block, a sub-block, a macroblock, a tile, or a slice.

[0020] <Block size specification> Furthermore, when specifying such block sizes, the block sizes may be specified not only directly but also indirectly. For example, the block sizes may be specified using identification information for identifying the sizes. Furthermore, for example, the block sizes may be specified by the ratio or difference with respect to the size of a reference block (e.g., LCU, SCU, etc.). For example, when transmitting information specifying the block size as a syntax element, the information indirectly specifying the size as described above may be used as the information. This may reduce the amount of information and improve coding efficiency. Furthermore, the specification of the block sizes may also include specification of a range of block sizes (e.g., specification of a range of allowable block sizes, etc.).

[0021] <Unit of information and processing> The data units in which various pieces of information are set and the data units targeted by various processes are each arbitrary and are not limited to the above-mentioned examples. For example, these pieces of information and processes may be set for each TU (Transform Unit), TB (Transform Block), PU (Prediction Unit), PB (Prediction Block), CU (Coding Unit), LCU (Largest Coding Unit), sub-block, block, tile, slice, picture, sequence, or component, respectively, or may be targeted at data of these data units. Of course, these data units may be set for each piece of information or process, and the data units for all pieces of information and processes do not need to be unified. Note that the storage location of these pieces of information is arbitrary and may be stored in the headers or parameter sets of the above-mentioned data units, or may be stored in multiple locations.

[0022] <Control information> Control information related to the present technology may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether or not to permit (or prohibit) application of the above-described present technology may be transmitted. Also, for example, control information indicating targets to which the above-described present technology is to be applied (or targets to which it is not to be applied) may be transmitted. For example, control information specifying a block size (upper or lower limit, or both), frame, component, or layer to which the present technology is to be applied (or permitted or prohibited to be applied) may be transmitted.

[0023] <flag> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be assumed not only to include the identification information in the bit stream, but also to include difference information of the identification information relative to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0024] <Associate metadata> Furthermore, various types of information (metadata, etc.) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. In other words, data associated with each other may be combined into one piece of data or may be individual pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer to only a portion of the data, rather than the entire data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0025] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "storage," "embed," "insert," and "insert" refer to combining multiple entities into one, such as combining encoded data and metadata into one data, and are one method of the above-mentioned "associating." Furthermore, in this specification, "encoding" includes not only the overall process of converting an image into a bitstream, but also some of the processes. For example, it not only includes processes that encompass prediction processing, orthogonal transform, quantization, arithmetic coding, etc., but also includes a process that collectively refers to quantization and arithmetic coding, a process that encompasses prediction processing, quantization, and arithmetic coding, and so on. Similarly, "decoding" not only includes the overall process of converting a bitstream into an image, but also some of the processes. For example, it not only includes processes that encompass inverse arithmetic decoding, inverse quantization, inverse orthogonal transform, prediction processing, etc., but also includes a process that encompasses inverse arithmetic decoding and inverse quantization, a process that encompasses inverse arithmetic decoding, inverse quantization, and prediction processing, and so on.

[0026] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0027] <Image Encoding Device and Image Decoding Device> An image encoding device and an image decoding device to which the present technology is applied will be described with reference to FIGS.

[0028] As shown in FIG. 2, the image encoding device 10 includes an encoding unit 11, a determining unit 12, a VCL buffer 13, a non-VCL buffer 14, a file generating unit 15, and a control unit 16.

[0029] The encoding unit 11 is an encoder that operates in accordance with the HEVC standard. The encoding unit 11 acquires an image sequence to be encoded from a video source, such as a camera or a television tuner, connected to the image encoding device 10. The encoding unit 11 then generates an encoded bitstream by performing various processes, such as intra-prediction, inter-frame prediction, orthogonal transform, quantization, and lossless encoding, on each image in the acquired image sequence. Slice data, which corresponds to the actual image, is generated as a VCL (Video Coding Layer) NAL unit.

[0030] On the other hand, parameter sets such as SPS (Sequence Parameter Set), PPS (Picture Parameter Set), and APS (Adaptation Parameter Set) can be generated as non-VCL NAL units. The encoding unit 11 outputs the VCL NAL units, i.e., slice data bitstreams, to the file generation unit 15 via the VCL buffer 13. The encoding unit 11 also outputs parameter sets to the file generation unit 15 via the non-VCL buffer 14.

[0031] The determination unit 12 determines the type of each picture in the picture sequence to be coded by the coding unit 11. More specifically, in this embodiment, the determination unit 12 determines at least whether each picture is an IDR picture, a CRA picture, or another picture. Both IDR pictures and CRA pictures are pictures consisting of only I-slices.

[0032] As described above, an IDR picture is the first picture in a sequence. Pictures following an IDR picture in the same sequence do not refer to pictures preceding the IDR picture in coding order (decoding order) and are located only after the IDR picture in display order. A CRA picture is located in the middle of a sequence and can be used as a decoding start picture during random access on the decoder side. Pictures following a CRA picture in both coding order (decoding order) and display order do not refer to pictures preceding the CRA picture in coding order (decoding order) or pictures preceding the CRA picture in display order. The determination unit 12 outputs the determination result to the encoding unit 11 and the file generation unit 15. The encoding unit 11 adds a NAL unit type indicating the type of each image determined by the determination unit 12 to the NAL header of each NAL unit.

[0033] When random access to a CRA picture is performed, only the CRA picture of the random access destination and pictures following the CRA picture in decoding order are subject to decoding. However, there may also be pictures following the CRA picture in decoding order and preceding the CRA picture in display order. In this specification, such pictures are referred to as leading pictures. As can be understood from the definition of a CRA picture, it is permissible for a leading picture to refer to a picture preceding the CRA picture in decoding order.

[0034] When random access is performed to a CRA picture, a leading picture that references a picture preceding the CRA picture in decoding order cannot be decoded correctly because the reference picture of the leading picture has not been decoded. That is, when random access is performed, whether the leading picture to be decoded can be decoded correctly depends on the reference relationship of the leading picture. Therefore, the determination unit 12 may further determine which leading pictures cannot be decoded correctly when random access is performed to each CRA picture, and provide the determination result to the file generation unit 15.

[0035] The VCL buffer 13 buffers the VCL NAL units. The slice data of the CRA picture is buffered by the VCL buffer 13.

[0036] The non-VCL buffer 14 buffers non-VCL NAL units.

[0037] The file generation unit 15 generates a video file storing a series of encoded image data according to a file format including a header area and a data area, and outputs the file to the storage unit 20. In this specification, an example in which the MP4 format is used as the file format will be mainly described. However, the technology according to the present disclosure is not limited to this example and can be applied to other types of video file formats that have a header area and a data area.

[0038] In the MP4 format, data is stored in objects called boxes and recorded in object units. Within a file, boxes form a tree structure, with parent boxes containing child boxes. Each box type is identified by a four-letter identifier.

[0039] More specifically, the file generation unit 15 inserts a bit stream of slice data corresponding to a VCL NAL unit into a data area (e.g., an mdat box) of the video file in decoding order. The file generation unit 15 also inserts one or more parameter sets corresponding to non-VCL NAL units into a header area (e.g., a moov box) of the video file. Furthermore, in this embodiment, the file generation unit 15 inserts CRA information that identifies one or more CRA pictures determined by the determination unit 12 into the header area of ​​the video file. Furthermore, the file generation unit 15 may include, in the CRA information, leading picture information for identifying leading pictures that will not be correctly decoded when random access to each CRA picture is performed.

[0040] The control unit 16 controls the encoding process executed in the image encoding device 10. For example, when an instruction to start encoding is detected, the control unit 16 causes the encoding unit 11 to encode a specified image sequence. The control unit 16 also causes the file generation unit 15 to generate a video file that stores the image data encoded by the encoding unit 11. The control unit 16 may use a hypothetical decoder model called an HRD (Hypothetical Reference Decoder) to control the generation of the encoded stream so as not to cause the decoder buffer to collapse.

[0041] As shown in FIG. 3, the image decoding device 30 includes a VCL buffer 31, a non-VCL buffer 32, a parameter memory 33, a decoding unit 34, an output buffer 35, and a control unit 37.

[0042] The VCL buffer 31 buffers a bit stream of image data (typically, slice data) read from a data area (for example, an mdat box) of a video file stored in the storage unit 20.

[0043] The non-VCL buffer 32 buffers parameter sets such as SPS, PPS, and APS that are read from the header area (for example, moov box) of a video file stored in the storage unit 20, as well as header information such as CRA information.

[0044] The parameter memory 33 collectively stores information in the header area of ​​a file acquired via the non-VCL buffer 32. The CRA information, which can be recorded in the header area of ​​a moving image file in the various formats described above, is held by the parameter memory 33 while the moving image file is open.

[0045] The decoding unit 34 is a decoder that operates in accordance with the HEVC standard. The decoding unit 34 decodes an image sequence from a bitstream obtained from the data area of ​​a video file via the VCL buffer 31. When decoding images, the decoding unit 34 uses parameters in a parameter set stored in the parameter memory 33. The decoding unit 34 rearranges the images in the decoded image sequence in display order, and outputs the rearranged images to the output buffer 35.

[0046] The decoding unit 34 normally accesses slice data stored in a video track in a video file in decoding order, starting from the beginning. However, when a random access instruction is detected by the control unit 37, the decoding unit 34 randomly accesses a decoding start picture (in the middle of the video track) specified by the control unit 37, and decodes the image sequence from that decoding start picture. The decoding start picture is one of the IDR picture and the CRA picture in the video track.

[0047] The output buffer 35 is a decoded picture buffer (DPB) for buffering images decoded by the decoding unit 34. The images buffered by the output buffer 35 are output to a display or processor (not shown) at the output timing of the images.

[0048] The control unit 37 controls the image decoding process executed in the image decoding device 30. For example, the control unit 37 opens a video file stored in the storage unit 20 in response to a user instruction and causes the decoding unit 34 to start decoding the image sequence. When a random access instruction is detected, the control unit 37 uses the CRA information to identify one of the CRA pictures in the image sequence as a decoding start picture. Then, the control unit 37 causes the decoding unit 34 to decode the image sequence from the identified decoding start picture (i.e., from the middle of the video track).

[0049] The control unit 37 typically identifies the CRA picture located closest to the timing specified in the random access instruction (for example, the timing indicated by the pointer operated on the seek bar of the video playback window) as the decoding start picture.

[0050] When the CRA information includes the preceding picture information described above, the control unit 37 may skip output from the output buffer 35 (and decoding by the decoding unit 34) of a leading picture that is identified using the leading picture information and that is not correctly decoded. By using the leading picture information, it is possible to prevent a corrupted image from being displayed on a display or output to an external processor. In this case, the control unit 37 does not need to subsequently determine whether each image has been correctly decoded.

[0051] Furthermore, the control unit 37 can transmit a command to the control unit 16 of the image encoding device 10.

[0052] <First form of bitstream> A first variation of the first form of the bitstream will be described with reference to FIGS.

[0053] As shown in Fig. 4, a bitstream is made up of multiple access units (AU), and each access unit corresponds to one frame and contains at least one NAL unit. There are multiple types of NAL units, such as an AUD (Access Unit Delimiter) NAL unit, an SPS (Sequence Parameter Set) NAL unit, a PPS (Picture Parameter Set) NAL unit, and a Tile group NAL unit.

[0054] The AUD NAL unit indicates the boundary of an access unit, and generally, only one AUD NAL unit is placed at the beginning of each access unit. Note that the current AUD already has a syntax that indicates the attributes of all Tile group NALs in an access unit.

[0055] The SPS NAL unit stores sequence parameters required for bitstream playback. The PPS NAL unit stores sequence parameters required for picture playback. The Tile group NAL unit stores images for each tile group.

[0056] The bitstream configuration can be assumed to have three cases as shown in the figure.

[0057] For example, in the first case, in all access units, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, and Tile group NAL units are placed consecutively thereafter. That is, in the first case, an SPS NAL unit and a PPS NAL unit are assigned to each access unit.

[0058] In the second case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, and a PPS NAL unit is placed third, with Tile group NAL units placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, a PPS NAL unit is placed second, and Tile group NAL units are placed consecutively thereafter. In other words, in the second case, an SPS NAL unit is assigned to the first access unit, and a PPS NAL unit is assigned to each access unit.

[0059] In the third case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, and a PPS NAL unit is placed third, with Tile group NAL units placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, with Tile group NAL units placed consecutively thereafter. In other words, in the third case, an SPS NAL unit and a PPS NAL unit are assigned only to the first access unit.

[0060] In the first variation of the first form of the bitstream, in any of the first to third cases, reference image information indicating a reference image to which the current picture refers is placed in an AUD NAL unit, which is a change from the conventional configuration. For example, this reference image information can be used to specify ref_pic_list_idx in tile_group_header or to set a new ref_pic_list_struct. In this way, by placing reference image information in the AUD NAL unit in the header area of ​​the access unit and applying the reference image information to all Tile group NAL units in the access unit, it is possible to avoid redundancy compared to specifying a reference image for each Tile group.

[0061] Specifically, the syntax of AUD NAL is described in the first variation of the first form of the bitstream as shown in Fig. 5. As shown in the figure, in the syntax of AUD NAL, the syntax of pic_type is extended, and it is possible to specify ref_pic_list_idx and set a new ref_pic_list_struct.

[0062] An example of NAL generation processing in the image encoding processing executed by the image encoding device 10 will be described with reference to the flowchart shown in FIG.

[0063] For example, when an image sequence is supplied to the image encoding device 10, the NAL generation process starts, and in step S11, the encoding unit 11 determines whether or not there is an NAL unit to be generated from the image sequence.

[0064] If the encoding unit 11 determines in step S11 that there is an NAL unit to be generated from the image sequence, the process proceeds to step S12.

[0065] In step S12, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is an AUD NAL unit.

[0066] If the encoding unit 11 determines in step S12 that the NAL unit to be generated from the image sequence is an AUD NAL unit, the process proceeds to step S13. Then, in step S13, the encoding unit 11 performs an AUD encoding process to generate an AUD NAL unit including reference image information and supplies the AUD NAL unit to the file generation unit 15 via the non-VCL buffer 14. After that, the process returns to step S11, and the same processes are repeated thereafter. On the other hand, if the encoding unit 11 determines in step S12 that the NAL unit to be generated from the image sequence is not an AUD NAL unit, the process proceeds to step S14.

[0067] In step S14, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is an SPS NAL unit.

[0068] If the encoding unit 11 determines in step S14 that the NAL unit to be generated from the image sequence is an SPS NAL unit, the process proceeds to step S15. Then, in step S15, the encoding unit 11 performs SPS encoding processing to generate an SPS NAL unit and supplies it to the file generation unit 15 via the non-VCL buffer 14. After that, the process returns to step S11, and the same processes are repeated thereafter. On the other hand, if the encoding unit 11 determines in step S14 that the NAL unit to be generated from the image sequence is not an SPS NAL unit, the process proceeds to step S16.

[0069] In step S16, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a PPS NAL unit.

[0070] If the encoding unit 11 determines in step S16 that the NAL units to be generated from the image sequence are PPS NAL units, the process proceeds to step S17. Then, in step S17, the encoding unit 11 performs PPS encoding processing to generate PPS NAL units and supplies them to the file generation unit 15 via the non-VCL buffer 14. After that, the process returns to step S11, and the same processes are repeated thereafter. On the other hand, if the encoding unit 11 determines in step S16 that the NAL units to be generated from the image sequence are not PPS NAL units, the process proceeds to step S18.

[0071] In step S18, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a Tile group NAL unit.

[0072] If the encoding unit 11 determines in step S18 that the NAL unit to be generated from the image sequence is a Tile group NAL unit, the process proceeds to step S19. Then, in step S19, the encoding unit 11 performs Tile group encoding processing to generate a Tile group NAL unit and supplies it to the file generation unit 15 via the VCL buffer 13, after which the process returns to step S11 and similar processes are repeated thereafter. On the other hand, if the encoding unit 11 determines in step S18 that the NAL unit to be generated from the image sequence is not a Tile group NAL unit, the process proceeds to step S20.

[0073] In step S20, the encoding unit 11 performs an encoding process for encoding an NAL unit other than any of the above-mentioned NAL units, and then the process returns to step S11, and the same process is repeated thereafter.

[0074] On the other hand, if the encoding unit 11 determines in step S11 that there are no NAL units to be generated from the image sequence, the NAL generation process is terminated.

[0075] FIG. 7 is a flowchart illustrating the AUD encoding process performed in step S13 of FIG.

[0076] In step S31, the encoding unit 11 performs processing to specify and set a reference frame list using reference image information, and can, for example, specify ref_pic_list_idx in tile_group_header or set a new ref_pic_list_struct.

[0077] In step S32, the encoding unit 11 generates an AUD NAL unit by performing a frame delimiting process for delimiting frames (access units), and then the AUD encoding process is terminated.

[0078] FIG. 8 is a flowchart illustrating the Tile group encoding process performed in step S19 of FIG.

[0079] In step S41, the encoding unit 11 performs a process of encoding a header other than the process of specifying and setting a reference frame list.

[0080] In step S42, the encoding unit 11 performs the encoding process for encoding the header and subsequent parts, and then the Tile group encoding process ends.

[0081] By the above-described NAL generation process, the image encoding device 10 generates an AUD NAL unit in which reference image information is placed, and generates a video file consisting of a bitstream as described above with reference to Figure 4, i.e., it is possible to encode an image of a bitstream consisting of an access unit in which at least one NAL unit is placed.

[0082] An example of NAL generation processing in the image decoding processing executed by the image decoding device 30 will be described with reference to the flowchart shown in FIG.

[0083] For example, the process starts when the image decoding device 30 reads out the bitstream stored in the storage unit 20, and in step S51, the decoding unit 34 determines whether or not there is an NAL unit to be decoded from the bitstream.

[0084] In step S51, if the decoding unit 34 determines that there is an NAL unit to be decoded from the bitstream, the process proceeds to step S52.

[0085] In step S52, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is an AUD NAL unit.

[0086] If the decoding unit 34 determines in step S52 that the NAL unit to be decoded from the bitstream is an AUD NAL unit, the process proceeds to step S53. Then, in step S53, the decoding unit 34 performs an AUD decoding process and supplies reference image information obtained by decoding the AUD NAL unit to the parameter memory 33. After that, the process returns to step S51, and similar processes are repeated thereafter. On the other hand, if the decoding unit 34 determines in step S52 that the NAL unit to be decoded from the bitstream is not an AUD NAL unit, the process proceeds to step S54.

[0087] In step S54, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is an SPS NAL unit.

[0088] If the decoding unit 34 determines in step S54 that the NAL unit to be decoded from the bitstream is an SPS NAL unit, the process proceeds to step S55. Then, in step S55, the decoding unit 34 performs SPS decoding processing to decode the SPS NAL unit and supplies the acquired parameters to the parameter memory 33, after which the process returns to step S51, and similar processes are repeated thereafter. On the other hand, if the decoding unit 34 determines in step S54 that the NAL unit to be decoded from the bitstream is not an SPS NAL unit, the process proceeds to step S56.

[0089] In step S56, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is a PPS NAL unit.

[0090] If the decoding unit 34 determines in step S56 that the NAL unit to be decoded from the bitstream is a PPS NAL unit, the process proceeds to step S57. Then, in step S57, the decoding unit 34 performs PPS decoding processing to decode the PPS NAL unit and supplies the acquired parameters to the parameter memory 33. After that, the process returns to step S51, and the same processes are repeated thereafter. On the other hand, if the decoding unit 34 determines in step S56 that the NAL unit to be decoded from the bitstream is not a PPS NAL unit, the process proceeds to step S58.

[0091] In step S58, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is a Tile group NAL unit.

[0092] If the decoding unit 34 determines in step S58 that the NAL unit to be decoded from the bitstream is a Tile group NAL unit, the process proceeds to step S59. Then, in step S59, the decoding unit 34 performs Tile group decoding processing to supply an image obtained by decoding the Tile group NAL unit to the output buffer 35, and then the process returns to step S51, and similar processes are repeated thereafter. On the other hand, if the decoding unit 34 determines in step S58 that the NAL unit to be decoded from the bitstream is not a Tile group NAL unit, the process proceeds to step S60.

[0093] In step S60, the decoding unit 34 performs a decoding process to decode NAL units other than any of the above-mentioned NAL units, and then the process returns to step S51, and the same processes are repeated thereafter.

[0094] On the other hand, if the decoding unit 34 determines in step S51 that there is no NAL unit to be decoded from the bitstream, the NAL generation process is terminated.

[0095] FIG. 10 is a flowchart illustrating the AUD decoding process performed in step S53 of FIG.

[0096] In step S71, the decoding unit 34 obtains the picture type from the AUD NAL unit.

[0097] In step S72, the decoding unit 34 acquires reference image information from the AUD NAL unit, and performs processing to specify and set a reference frame list in accordance with the reference image information.

[0098] In step S73, the decoding unit 34 decodes the AUD NAL unit by performing frame delimiting processing to delimit frames (access units), and then the AUD decoding processing is terminated.

[0099] FIG. 11 is a flowchart illustrating the Tile group decoding process performed in step S59 of FIG.

[0100] In step S81, the decoding unit 34 performs a process of decoding the header, other than the process of specifying and setting the reference frame list.

[0101] In step S82, the decoding unit 34 performs a decoding process for decoding the header and subsequent parts, and then the Tile group decoding process ends.

[0102] As described above, the image decoding device 30 can decode images according to the reference image information obtained from the AUD NAL unit, i.e., decode images of a bitstream consisting of access units in which at least one NAL unit is placed.

[0103] A second variation of the first form of the bitstream will be described with reference to FIGS.

[0104] As shown in Fig. 12, the bitstream is composed of a plurality of access units, and at least one NAL unit is arranged in each access unit, as described above with reference to Fig. 4. Furthermore, as in the first variation of the first form of the bitstream (see Fig. 4), three cases are assumed for the bitstream configuration, as shown in Fig. 12.

[0105] In the second variation of the first form of the bitstream, a NEW NAL unit is used as the NAL unit in addition to the AUD NAL unit, the SPS NAL unit, the PPS NAL unit, and the Tile group NAL unit.

[0106] For example, in the first case, in all access units, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, a NEW NAL unit is placed fourth, and Tile group NAL units are placed consecutively thereafter. That is, in the first case, an SPS NAL unit and a PPS NAL unit are assigned to each access unit, and a NEW NAL unit is also assigned to each access unit.

[0107] In the second case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, and a NEW NAL unit is placed fourth, with Tile group NAL units being placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, a PPS NAL unit is placed second, and a NEW NAL unit is placed third, with Tile group NAL units being placed consecutively thereafter. In other words, in the second case, an SPS NAL unit is assigned to the first access unit, and PPS NAL units and NEW NAL units are assigned to each access unit.

[0108] In the third case, in the first access unit, an AUD NAL unit is placed first, an SPS NAL unit is placed second, a PPS NAL unit is placed third, and a NEW NAL unit is placed fourth, with Tile group NAL units being placed consecutively thereafter. In the second and subsequent access units, an AUD NAL unit is placed first, a NEW NAL unit is placed second, and Tile group NAL units are placed consecutively thereafter. In other words, in the third case, an SPS NAL unit and a PPS NAL unit are assigned only to the first access unit, and a NEW NAL unit is assigned to each individual access unit.

[0109] In the second variation of the first form of the bitstream, in any of the first to third cases, reference image information indicating a reference image to which the current picture refers is placed in a NEW NAL unit, which is a change from the conventional configuration. For example, this reference image information can be used to specify ref_pic_list_idx in tile_group_header or to set a new ref_pic_list_struct. In this way, by placing reference image information in a NEW NAL unit (an identification NAL unit) in the header area of ​​an access unit and applying the reference image information to all Tile group NAL units in the access unit, it is possible to avoid redundancy compared to specifying a reference image for each Tile group.

[0110] Specifically, in the second variation of the first form of the bitstream, the syntax of a NEW NAL unit is described as shown in Fig. 13. As shown in the figure, the syntax of the NEW NAL allows specification of ref_pic_list_idx and setting of a new ref_pic_list_struct.

[0111] An example of NAL generation processing performed in the image encoding processing executed by the image encoding device 10 will be described with reference to the flowchart shown in FIG.

[0112] In steps S101 to S107, the same processes as in steps S11 to S17 described above with reference to the flowchart in Fig. 6 are performed. Then, in step S108, the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a NEW NAL unit.

[0113] If the encoding unit 11 determines in step S108 that the NAL unit to be generated from the image sequence is a NEW NAL unit, the process proceeds to step S109. Then, in step S109, the encoding unit 11 performs a NEW encoding process to generate a NEW NAL unit including reference image information and supplies the NEW NAL unit to the file generation unit 15 via the non-VCL buffer 14. After that, the process returns to step S101, and the same processes are repeated thereafter. On the other hand, if the encoding unit 11 determines in step S108 that the NAL unit to be generated from the image sequence is not a NEW NAL unit, the process proceeds to step S110.

[0114] Then, in steps S110 to S112, the same processes as in steps S18 to S20 described above with reference to the flowchart in FIG. 6 are performed, and then the NAL generation process is terminated.

[0115] FIG. 15 is a flowchart illustrating the NEW encoding process performed in step S109 of FIG.

[0116] In step S121, the encoding unit 11 performs a process of specifying and setting a reference frame list using reference image information, for example, specifying ref_pic_list_idx in tile_group_header or setting a new ref_pic_list_struct. After the encoding unit 11 generates a NEW NAL unit in this way, the NEW encoding process ends.

[0117] The Tile group encoding process performed in step S111 of FIG. 14 is performed in the same manner as the Tile group encoding process described above with reference to the flowchart of FIG.

[0118] Through the above-described NAL generation process, the image encoding device 10 generates a new NAL unit in which reference image information is placed, and generates a video file consisting of a bitstream as described above with reference to Figure 12, i.e., it is possible to encode an image of a bitstream consisting of an access unit in which at least one NAL unit is placed.

[0119] An example of NAL generation processing in the image decoding processing executed by the image decoding device 30 will be described with reference to the flowchart shown in FIG.

[0120] In steps S131 to S137, the same processes as in steps S51 to S57 described above with reference to the flowchart in Fig. 9 are performed. Then, in step S138, the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is a NEW NAL unit.

[0121] If the decoding unit 34 determines in step S138 that the NAL unit to be decoded from the bitstream is a NEW NAL unit, the process proceeds to step S139. Then, in step S139, the decoding unit 34 performs a NEW decoding process and supplies reference image information obtained by decoding the NEW NAL unit to the parameter memory 33. After that, the process returns to step S131, and the same processes are repeated thereafter. On the other hand, if the decoding unit 34 determines in step S138 that the NAL unit to be decoded from the bitstream is not an AUD NAL unit, the process proceeds to step S140.

[0122] Then, in steps S140 to S142, the same processes as in steps S58 to S60 described above with reference to the flowchart in FIG. 9 are performed, and then the NAL generation process ends.

[0123] FIG. 17 is a flowchart illustrating the NEW decoding process performed in step S139 of FIG.

[0124] In step S151, the decoding unit 34 obtains reference image information from the NEW NAL unit, and decodes the NEW NAL unit by performing a process of specifying and setting a reference frame list according to the reference image information, and then the NEW decoding process is terminated.

[0125] The Tile group decoding process performed in step S141 of FIG. 16 is performed in the same manner as the Tile group decoding process described above with reference to the flowchart of FIG.

[0126] As described above, the image decoding device 30 can decode images according to the reference image information obtained from the NEW NAL unit, i.e., decode images of a bitstream consisting of access units in which at least one NAL unit is placed.

[0127] <Second form of bitstream> The second form of the bitstream will be described with reference to FIGS.

[0128] In the second form of the bitstream, the Long Term Reference, which can be set irregularly, can be dynamically changed.

[0129] First, with reference to FIG. 18, a method of restoring image quality in a narrow band using a long-term reference will be described as an example of dynamically changing a long-term reference.

[0130] For example, when a packet loss occurs at the timing shown in the figure, the image decoding device 30 on the receiving side transmits error occurrence information indicating the occurrence of a packet loss to the image coding device 10 on the transmitting side. Then, the image coding device 10 avoids retransmission of the I-Picture and encodes the frame P' to be coded after receiving the error occurrence information as a Long Term Reference Picture (LTR frame P'). L ) and encode it.

[0131] Generally, the Long Term Reference Picture is set (updated) irregularly when the motion is relatively large. For example, the Short Term Reference Picture does not need to be changed because it is encoded with a fixed GOP structure, whereas the Long Term Reference Picture can be determined whether to change when encoding a picture.

[0132] In this way, by dynamically changing the Long Term Reference, the image decoding device 30 can decode the LTR frame P even if a packet loss occurs. L The encoded frame P' is the LTR frame P L By decoding the image with reference to the above, it is possible to recover from the degradation of image quality due to packet loss with less delay.

[0133] Such image quality restoration techniques using long-term references are expected to be used in low-latency or interactive applications with limited transmission bandwidth, such as surveillance cameras, video chat, and streaming / game services.

[0134] A first variation of the second form of the bitstream will be described with reference to FIGS.

[0135] As shown in Fig. 19, the bitstream is composed of a plurality of access units, as described above with reference to Fig. 4, and each access unit contains at least one NAL unit (an AUD NAL unit, an SPS NAL unit, a PPS NAL unit, and a Tile group NAL unit). As with the first variation of the first form of the bitstream (see Fig. 4), three cases are assumed for the bitstream configuration, as shown in Fig. 19.

[0136] In the first variation of the second form of the bitstream, in any of the first to third cases, the AUD NAL unit is extended so that only the Long Term can be independently modified (modify poc_lsb_lt of ref_pic_list_struct(i, ref_pic_list_idx[i]).

[0137] Specifically, as shown in FIG. 20, in the first variation of the second form of the bitstream, the syntax of SPS NAL and AUD NAL is described.

[0138] For example, the long term of the ref_pic_list_struct defined in the SPS NAL unit can be changed in the AUD NAL unit.

[0139] A second variation of the second form of the bitstream will be described with reference to FIGS.

[0140] As shown in Fig. 21, the bitstream is composed of a plurality of access units, as described above with reference to Fig. 4, and each access unit contains at least one NAL unit (AUD NAL unit, SPS NAL unit, PPS NAL unit, Tile group NAL unit, and NEW NAL unit). Also, as in the second variation of the first form of the bitstream (see Fig. 12), three cases are assumed for the bitstream configuration, as shown in Fig. 21.

[0141] In the second variation of the second form of the bitstream, in any of the first to third cases, the NEW NAL unit is extended so that only the Long Term can be independently modified (modify poc_lsb_lt of ref_pic_list_struct(i, ref_pic_list_idx[i]).

[0142] Specifically, as shown in FIG. 22, in the second variation of the second form of the bitstream, the syntax of SPS NAL and NEW NAL is described.

[0143] For example, the long term of the ref_pic_list_struct already defined in the SPS NAL unit can be changed in the NEW NAL unit.

[0144] An example of a reference frame list designation / setting process performed in the image encoding process executed by the image encoding device 10 will be described with reference to the flowchart shown in FIG.

[0145] For example, when an image sequence is supplied to the image encoding device 10, the reference frame list setting process is started, and in step S161, the encoding unit 11 determines whether or not to specify a predefined reference frame list in the SPS NAL unit.

[0146] If it is determined in step S161 that a predefined reference frame list is to be specified in the SPS NAL unit, the process proceeds to step S162, where the encoding unit 11 acquires the index of the reference frame list.

[0147] In step S163, the encoding unit 11 determines whether or not to change the long-term reference frame information, in accordance with the index of the reference frame list acquired in step S162.

[0148] If the encoding unit 11 determines in step S163 that the long-term reference frame information is to be changed, the process proceeds to step S164, where a process of changing the long-term reference frame information is performed (see the flowchart in FIG. 24, which will be described later).

[0149] On the other hand, in step S163, if it is determined not to change the Long Term reference frame information, or after the change process of the Long Term reference frame information in step S164 is performed, the reference frame list setting process ends.

[0150] On the other hand, in step S161, if it is determined not to specify the reference frame list defined in the SPS NAL unit, the process proceeds to step S165. In step S165, after the encoding unit 11 performs the encoding process of the reference frame list, the reference frame list specification / setting process ends.

[0151] FIG. 24 is a flowchart for explaining the change process of the Long Term reference frame information performed in step S164 of FIG. 23.

[0152] In step S171, the encoding unit 11 obtains the Long Term number k of the reference frame list of the index obtained in step S162 of FIG. 23, and sets 0 to the parameter i (i = 0).

[0153] In step S172, the encoding unit 11 determines whether the parameter i is less than the Long Term number k.

[0154] In step S172, if the encoding unit 11 determines that the parameter i is less than the Long Term number k (i < k), the process proceeds to step S173.

[0155] In step S173, the encoding unit 11 changes the absolute position of the i-th Long Term reference frame. For example, the encoding unit 11 can set the absolute position of the i-th Long Term reference frame based on a command (information specifying a reference image) from the image decoding device 30. Also, the image decoding device 30 updates the information specifying the reference image for each frame.

[0156] In step S174, the encoding unit 11 increments the parameter i (i=i+1), and then the process returns to step S172.

[0157] On the other hand, if the encoding unit 11 determines in step S172 that the parameter i is not less than the Long Term number k (i≧k), the process of changing the Long Term reference frame information is terminated.

[0158] By performing the above-described reference frame list specification and setting process, the image encoding device 10 can change the long-term reference frame information and generate a video file consisting of a bitstream such as that described above with reference to Figure 19 or Figure 21.

[0159] An example of the reference frame list designation / setting process performed in the image decoding process executed by the image decoding device 30 will be described with reference to the flowchart shown in FIG.

[0160] For example, the process starts when the image decoding device 30 reads out a bitstream stored in the storage unit 20, and in step S181, the decoding unit 34 determines whether or not to specify a predefined reference frame list in the SPS NAL unit.

[0161] If it is determined in step S181 that a predefined reference frame list is specified in the SPS NAL unit, the process proceeds to step S182, where the decoding unit 34 obtains the index of the reference frame list.

[0162] In step S183, the decoding unit 34 determines whether or not to change the long-term reference frame information, according to the index of the reference frame list acquired in step S182.

[0163] In step S183, when the decoding unit 34 determines to change the Long Term reference frame information, the process proceeds to step S184, and the process of changing the Long Term reference frame information (see the flowchart in FIG. 26 described later) is performed.

[0164] On the other hand, in step S183, when it is determined not to change the Long Term reference frame information, or after the process of changing the Long Term reference frame information in step S184 is performed, the reference frame list setting process ends.

[0165] On the other hand, in step S181, when it is determined that the reference frame list defined in the SPS NAL unit is not specified, the process proceeds to step S185. In step S185, after the decoding unit 34 performs the decoding process of the reference frame list, the reference frame list specification / setting process ends.

[0166] FIG. 26 is a flowchart for explaining the process of changing the Long Term reference frame information performed in step S184 of FIG. 25.

[0167] In step S191, the decoding unit 34 obtains the Long Term number k of the reference frame list of the index obtained in step S182 of FIG. 25, and sets 0 to the parameter i (i = 0).

[0168] In step S192, the decoding unit 34 determines whether the parameter i is less than the Long Term number k.

[0169] In step S192, when the decoding unit 34 determines that the parameter i is less than the Long Term number k (i < k), the process proceeds to step S193.

[0170] In step S193, the decoding unit 34 changes the absolute position of the i-th Long Term reference frame according to the Long Term reference frame information.

[0171] In step S194, the decoding unit 34 increments the parameter i (i=i+1), and then the process returns to step S192.

[0172] On the other hand, if the decoding unit 34 determines in step S192 that the parameter i is not less than the Long Term number k (i≧k), the process of changing the Long Term reference frame information is terminated.

[0173] By performing the above-described reference frame list specification and setting process, the image decoding device 30 can decode images in accordance with the long-term reference frame information that has been changed in a video file consisting of a bitstream such as that described with reference to Figure 19 or Figure 21 above.

[0174] <Example of computer configuration> Next, the above-described series of processes (image decoding method and image encoding method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0175] FIG. 27 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0176] The program can be recorded in advance on the hard disk 105 or ROM 103 as a recording medium built into the computer.

[0177] Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0178] The program can be installed into the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0179] The computer includes a CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0180] When a user inputs a command by operating input unit 107 via input / output interface 110, CPU 102 executes a program stored in ROM (Read Only Memory) 103 in accordance with the command. Alternatively, CPU 102 loads a program stored on hard disk 105 into RAM (Random Access Memory) 104 and executes it.

[0181] As a result, CPU 102 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. CPU 102 then outputs the processing results from output unit 106 via input / output interface 110, transmits them from communication unit 108, or records them on hard disk 105, as necessary.

[0182] The input unit 107 is made up of a keyboard, a mouse, a microphone, etc. The output unit 106 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0183] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0184] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0185] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0186] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0187] Furthermore, for example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0188] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0189] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0190] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0191] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0192] <Configuration combination example> The present technology can also be configured as follows. (1) A decoding unit that decodes an image of a bitstream that includes an access unit in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with The header area of ​​the access unit stores reference image information indicating a reference image to which the current picture refers. Image decoding device. (2) The reference image information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The image decoding device according to (1) above. (3) The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image decoding device according to (1) above. (4) The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. The image decoding device according to any one of (1) to (3) above. (5) The long term information is information that identifies the reference image that is determined based on information from a client that is the receiver of the bitstream. The image decoding device according to (4) above. (6) The information from the client is updated every frame. The image decoding device according to (5) above. (7) The long term information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The image decoding device according to any one of (4) to (6) above. (8) The long term information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image decoding device according to any one of (4) to (6) above. (9) An image decoding device that performs image decoding processing, Decoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is placed Including, The header area of ​​the access unit stores reference image information indicating a reference image to which the current picture refers. Image decoding method. (10) An encoding unit that encodes an image of a bitstream consisting of access units in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with Reference picture information indicating a reference picture to which the current picture refers is stored in the header area of ​​the access unit. Image encoding device. (11) The reference image information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The image encoding device according to (10) above. (12) The reference image information is stored in an identification NAL unit provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The image encoding device according to (10) above. (13) The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. The image encoding device according to any one of (10) to (12) above. (14) The long term information is information that identifies the reference image that is determined based on information from a client that is the receiver of the bitstream. The image encoding device according to (13) above. (15) The information from the client is updated every frame. The image encoding device according to (14) above. (16) The long term information is stored in an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The image encoding device according to any one of (13) to (15) above. (17) The long term information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates the delimiter of the access unit, and is applied to all Tile group NAL units in the access unit. The image encoding device according to any one of (13) to (15) above. (18) An image encoding device that performs image encoding processing, Encoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is placed Including, Reference picture information indicating a reference picture to which the current picture refers is stored in the header area of ​​the access unit. Image encoding method.

[0193] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained. [Explanation of symbols]

[0194] 10 image encoding device, 11 encoding unit, 12 determination unit, 13 VCL buffer, 14 non-VCL buffer, 15 file generation unit, 16 control unit, 20 storage unit, 30 image decoding device, 31 VCL buffer, 32 non-VCL buffer, 33 parameter memory, 34 decoding unit, 35 output buffer, 37 control unit

Claims

1. A decoding unit that decodes an image of a bitstream that includes access units in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with Reference image information indicating a reference image to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The NAL unit for identification is configured so that only Long Term can be independently changed. Image decoding device.

2. The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. The image decoding device according to claim 1 .

3. The long term information is information that identifies the reference image that is determined based on information from a client that is the receiver of the bitstream. The image decoding device according to claim 2 .

4. The information from the client is updated every frame. The image decoding device according to claim 3 .

5. An image decoding device that performs image decoding processing, Decoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is placed Including, Reference image information indicating a reference image to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The NAL unit for identification is configured so that only Long Term can be independently changed. Image decoding method.

6. An encoding unit that encodes an image of a bitstream consisting of access units in which at least one NAL (Network Abstraction Layer) unit is arranged. Equipped with Reference image information indicating a reference image to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The NAL unit for identification is configured so that only Long Term can be independently changed. Image encoding device.

7. The reference picture information includes long term information indicating whether the reference picture is a long term reference picture. The image encoding device according to claim 6 .

8. The long term information is information that identifies the reference image that is determined based on information from a client that is the receiver of the bitstream. The image encoding device according to claim 7 .

9. The information from the client is updated every frame. The image encoding device according to claim 8 .

10. An image encoding device that performs image encoding processing, Encoding an image of a bitstream consisting of an access unit in which at least one NAL (Network Abstraction Layer) unit is placed Including, Reference image information indicating a reference image to which the current picture refers is stored in a header area of ​​the access unit; The reference image information is stored in an identification NAL unit that is provided separately from an AUD (Access Unit Delimiter) NAL unit that indicates a delimiter of the access unit, and is applied to all Tile group NAL units within the access unit. The NAL unit for identification is configured so that only Long Term can be independently changed. Image encoding method.

Images