Image decoding device, image decoding method, image encoding device and image encoding method

Abstract

The present invention relates to an image decoding device, an image decoding method, an image encoding device, and an image encoding method capable of further improving practicality. Reference image information indicating a reference image to be referenced by a current picture is stored in a header area of ​​an access unit. The reference image information is stored in an AUD NAL unit indicating the division of an access unit or in a NEW NAL unit set separately from the AUD NAL unit, and is applied to all tile group NAL units in the access unit. The present technology can be applied to, for example, an image decoding device and an image encoding device.

Description

Technical Field

[0001] The present disclosure relates to an image decoding device, an image decoding method, an image encoding device, and an image encoding method, and in particular to an image decoding device, an image decoding method, an image encoding device, and an image encoding method that enable further improvement in practicality. Background Art

[0002] In H.264 / AVC, which is one of the standard specifications of the image coding scheme, each image (picture) is divided into one or more slices. Then, each of these slices is classified into one of an I slice (internal slice), a P slice (predicted slice), and a B slice (bidirectionally predicted slice). An I slice is a slice that is decoded independently without reference to another image. A P slice is a slice that is decoded by reference to another single image. A B slice is a slice that is decoded by reference to multiple other images.

[0003] A picture that consists only of I slices and is placed at the beginning of a sequence is called an Instantaneous Decoding Refresh (IDR) picture. An IDR picture is identified by the value of the Network Abstraction Layer (NAL) unit type. Pictures following an IDR picture in the same sequence do not refer to pictures before the IDR picture in decoding order, but are only located after the IDR picture in presentation order. Therefore, when random access is attempted at a mid-point in time of a video of a certain coded stream (decoding / reproducing from the middle of the stream instead of decoding from the beginning of the stream), the video can be properly decoded based on the IDR pictures near the specified time point.

[0004] In the standardization work of High Efficiency Video Coding (HEVC), which is the next generation image coding scheme after H.264 / AVC, it has been proposed to identify clean random access (CRA) pictures by the value of the NAL unit type separated from the IDR picture. A CRA picture is a picture composed of only I slices and placed in the middle of a sequence. Pictures that follow the CRA picture in both decoding order and presentation order do not refer to pictures that precede the CRA picture in decoding order and pictures that precede the CRA picture in presentation order. Therefore, when a CRA picture is randomly accessed (the video is decoded according to the CRA picture) at an intermediate time point in the video, the decoding process of the pictures that follow the CRA picture in presentation order can be successfully performed.

[0005] Here, Non-Patent Literature 1 discloses a rule for storing reference image information in a header of a tile group so as to specify a reference image for each tile group image in a multi-tile image.

[0006] Citation List

[0007] Non-patent literature

[0008] Non-patent literature 1:14496-15:2014,JVET-M1001 Summary of the invention

[0009] Problems to be solved by the present invention

[0010] Incidentally, as disclosed in the above-mentioned non-patent document 1, in the past, reference image information needs to be matched in all tile groups, and therefore specifying a reference image for each tile group is redundant. Furthermore, in the past, in the case of wishing to update a reference image as a long-term reference, it is not allowed to modify only the long-term reference. Therefore, it is desirable to expand the past criteria to further improve practicality.

[0011] The present disclosure has been made in view of such circumstances and is intended to enable further improvement in practicality.

[0012] Solution to the problem

[0013] An image decoding device according to a first aspect of the present disclosure includes a decoding unit that decodes an image of a bit stream composed of an access unit, wherein at least one or more network abstraction layer (NAL) units are arranged in the access unit, wherein reference image information indicating a reference image to be referenced by a current picture has been stored in a header area of ​​the access unit.

[0014] The image decoding method of the first aspect of the present disclosure includes: an image decoding device that performs image decoding processing decodes an image of a bit stream composed of an access unit, wherein at least one or more NAL units are arranged in the access unit, wherein reference image information indicating a reference image to be referenced by a current picture has been stored in a header area of ​​the access unit.

[0015] In a first aspect of the present disclosure, an image of a bit stream consisting of an access unit is decoded, in which at least one or more NAL units are arranged, and reference image information indicating a reference image to be referenced by a current picture has been stored in a header area of ​​the access unit.

[0016] The image encoding device of the second aspect of the present disclosure includes an encoding unit, which encodes an image of a bit stream composed of access units, wherein at least one or more NAL units are arranged in the access unit, wherein reference image information indicating a reference image to be referenced by a current picture is stored in a header area of ​​the access unit.

[0017] The image encoding method of the second aspect of the present disclosure includes: encoding an image of a bit stream composed of an access unit by an image encoding device that performs image encoding processing, wherein at least one or more NAL units are arranged in the access unit, wherein reference image information indicating a reference image to be referenced by a current picture is stored in a header area of ​​the access unit.

[0018] In a second aspect of the present disclosure, an image of a bit stream consisting of an access unit is encoded, at least one or more NAL units are arranged in the access unit, and reference image information indicating a reference image to be referenced by a current picture is stored in a header area of ​​the access unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a diagram showing references.

[0020] Figure 2 : is a block diagram showing an exemplary configuration of an image encoding device according to an embodiment to which the present technology is applied.

[0021] Figure 3 : is a block diagram showing an exemplary configuration of an image decoding device according to an embodiment to which the present technology is applied.

[0022] Figure 4 is a diagram showing an exemplary configuration of a bit stream in the first modification of the first mode.

[0023] Figure 5 is a diagram showing an exemplary description of the syntax of an access unit delimiter (AUD) NAL unit in the first variation of the first mode.

[0024] Figure 6 This is a flowchart for explaining the NAL generation process in the image encoding process.

[0025] Figure 7 is a flowchart illustrating the AUD encoding process.

[0026] Figure 8 is a flowchart illustrating the tile group encoding process.

[0027] Fig. 9 This is a flowchart for explaining the NAL generation process in the image decoding process.

[0028] Fig.10 is a flowchart illustrating the AUD decoding process.

[0029] Fig.11 is a flowchart illustrating the tile group decoding process.

[0030] Fig.12is a diagram showing an exemplary configuration of a bit stream in the second modification of the first mode.

[0031] Fig.13 is a diagram showing an exemplary description of the syntax of a NEW NAL unit in the second variation of the first mode.

[0032] Fig.14 This is a flowchart for explaining the NAL generation process in the image encoding process.

[0033] Fig.15 It is a flowchart explaining the NEW encoding process.

[0034] Fig.16 This is a flowchart for explaining the NAL generation process in the image decoding process.

[0035] Fig.17 It is a flowchart explaining the NEW decoding process.

[0036] Fig.18 is a diagram illustrating a technique for restoring image quality in a narrow band using a long-term reference.

[0037] Fig.19 is a diagram showing an exemplary configuration of a bit stream in the first modification of the second mode.

[0038] Fig. 20 is a diagram showing an exemplary description of the syntax of a sequence parameter set (SPS) NAL and an AUD NAL in the first variation of the second mode.

[0039] Fig.21 is a diagram showing an exemplary configuration of a bit stream in the second modification of the second mode.

[0040] Fig. 22 is a diagram showing an exemplary description of the syntax of the SPS NAL and the NEW NAL in the second variation of the second mode.

[0041] Fig.23 Detailed description of the flowchart of the reference frame list designation and setting process in the image encoding process.

[0042] Fig.24 is a flow chart illustrating the long-term reference frame information modification process.

[0043] Fig.25 Detailed description of the flowchart of the reference frame list designation and setting process in the image decoding process.

[0044] Fig.26 is a flow chart illustrating the long-term reference frame information modification process.

[0045] Fig. 27is a block diagram illustrating an exemplary configuration of a computer according to an embodiment to which the present technology is applied. DETAILED DESCRIPTION

[0046] <Documents supporting technical content and terminology, etc.>

[0047] The scope of the disclosure in this specification is not limited to the contents of the examples, and Figure 1 The contents of references REF 1 to REF 6 shown in , which were known at the time of filing, are also incorporated into the present specification by reference.

[0048] In other words, Figure 1 The contents described in references REF 1 to REF 6 shown in are also the basis for verifying the support requirements. For example, the NAL unit structure described in reference REF 4 and the byte stream format described in reference REF 5 are also interpreted as being within the scope of the present disclosure and satisfying the support requirements of the claims, even if the above-mentioned technologies are not directly defined in the detailed description of the present invention. In addition, similarly, for example, technical terms such as parsing, syntax, and semantics are also interpreted as being within the scope of the present disclosure and satisfying the support requirements of the claims, even if the technical terms are not directly defined in the detailed description of the present invention.

[0049] <Term>

[0050] In this application, the following terms are defined as follows.

[0051] <block>

[0052] Unless otherwise specified, a "block" (not a block indicating a processing unit) used as a partial area or a processing unit of an image (picture) in this specification indicates any partial area in a picture, and its size, shape, characteristics, etc. are not limited. For example, it is assumed that a "block" includes any partial area (processing unit) such as a transform block (TB), a transform unit (TU), a prediction block (PB), a prediction unit (PU), a minimum coding unit (SCU), a coding unit (CU), a maximum coding unit (LCU), a coding tree block (CTB), a coding tree unit (CTU), a transform block, a subblock, a macroblock, a tile, and a slice.

[0053] <Block size specification>

[0054] In addition, when specifying the size of such a block, the block size can be specified not only directly but also indirectly. For example, the block size can be specified using identification information that identifies the size. In addition, for example, the block size can be specified by a ratio or difference relative to the size of a block (e.g., LCU or SCU) serving as a reference. For example, in the case where information specifying the block size is transmitted as a syntax element or the like, the information indirectly specifying the size as described above can be used as the information. By specifying the size in this manner, the amount of information of the information can be reduced, and in some cases the coding efficiency can be improved. In addition, such specification of the block size also includes the specification of a range of block sizes (e.g., the specification of a range of allowed block sizes).

[0055] <Units of Information and Processing>

[0056] Both the units of data for setting various information and the units of data for which various processes are targeted are optional and are not limited to the above examples. For example, this information can be set for each transform unit (TU), transform block (TB), prediction unit (PU), prediction block (PB), coding unit (CU), maximum coding unit (LCU), sub-block, block, tile, slice, picture, sequence or component, and these processes can target the data in those data units. Of course, these data units can be set for each information or process, and these data units do not have to be consistent between all information or processes. Note that the storage location of this information is optional and can be stored in the header, parameter set, etc. of the above data unit. In addition, this information can be stored in multiple locations.

[0057] <Control Information>

[0058] Control information related to the present technology may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether to allow (or prohibit) the application of the above-mentioned present technology may be transmitted. In addition, for example, control information indicating an object to which the above-mentioned present technology is to be applied (or an object to which the above-mentioned present technology is not to be applied) may be transmitted. For example, control information specifying the size (upper limit, lower limit, or both upper limit and lower limit) of a block to which the present technology is to be applied (or to allow or prohibit the application), a frame, a component, a layer, etc. may be transmitted.

[0059] <Logo>

[0060] Note that in this specification, a "flag" refers to information used to identify multiple states, and includes not only information used when identifying the two states of true (1) and false (0), but also information that can identify three or more states. Therefore, the value that the "flag" can take can be a binary value such as 1 or 0, or a ternary or more-element value. That is, the number of bits that constitute the "flag" is optional, and one bit or more bits can be used. In addition, it is assumed that the identification information (including the flag) has not only a form in which the identification information is included in the bit stream, but also a form in which the difference information of the identification information relative to specific reference information is included in the bit stream. Therefore, in this specification, "flag" and "identification information" not only imply the overall information therein, but also imply the difference information relative to the reference information.

[0061] <Metadata Association>

[0062] In addition, various information (metadata, etc.) about the coded data (bitstream) can be transmitted or recorded in any form, as long as the information is associated with the coded data. Here, the term "association" means, for example, ensuring that a piece of data is available (linkable) when processing another piece of data. In other words, multiple pieces of data associated with each other can be collected into one piece of data, or each piece of data in the multiple pieces of data can be regarded as a single piece of data. For example, information associated with coded data (image) can be transmitted on a transmission path different from the transmission path of the associated coded data (image). In addition, for example, information associated with coded data (image) can be recorded on a recording medium (or a recording area of ​​the same recording medium) different from the recording medium of the associated coded data (image). Note that the "association" can be generated on a part of the data rather than the entire data. For example, an image and information corresponding to the image can be associated with each other in any unit such as multiple frames, a frame, or a part of a frame.

[0063] In addition, in the present specification, terms such as "synthesize", "multiplex", "add", "integrate", "include", "save", "merge", "put in", and "insert" mean collecting multiple items into one item, such as collecting coded data and metadata into one piece of data, and mean a method of the above-mentioned "association". In addition, in the present specification, encoding includes not only the entire process of converting an image into a bit stream, but also a part of the process. For example, encoding includes not only a process including prediction processing, orthogonal transform, quantization, arithmetic coding, etc., but also a process in which quantization and arithmetic coding are collectively referred to, a process including prediction processing, quantization and arithmetic coding, etc. Similarly, decoding includes not only the entire process of converting a bit stream into an image, but also a part of the process. For example, decoding includes not only a process including inverse arithmetic decoding, inverse quantization, inverse orthogonal transform, prediction processing, etc., but also a process including inverse arithmetic decoding and inverse quantization, a process including inverse arithmetic decoding, inverse quantization and prediction processing, etc.

[0064] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the accompanying drawings.

[0065] <Image Coding Device and Image Decoding Device>

[0066] Will refer to Figure 2 and Figure 3 An image encoding device and an image decoding device to which the present technology is applied are described.

[0067] like Figure 2 As shown, the image encoding device 10 has a configuration including an encoding unit 11, a determination unit 12, a video coding layer (VCL) buffer 13, a non-VCL buffer 14, a file generation unit 15, and a control unit 16.

[0068] The encoding unit 11 is an encoder operating according to the HEVC scheme. The encoding unit 11 acquires an image sequence to be encoded from a moving image source such as a camera or a TV tuner connected to the image encoding device 10. Then, the encoding unit 11 generates an encoded bit stream by performing various processes such as intra-frame prediction, inter-frame prediction, orthogonal transformation, quantization, and lossless encoding for each image in the acquired image sequence. Slice data corresponding to the entity of the image is generated as a video coding layer (VCL) NAL unit.

[0069] At the same time, parameter sets such as sequence parameter sets (SPS), picture parameter sets (PPS), and adaptation parameter sets (APS) may be generated as non-VCL NAL units. The encoding unit 11 outputs the bit stream (ie, slice data) of the VCL NAL unit to the file generation unit 15 via the VCL buffer 13. In addition, the encoding unit 11 outputs the parameter set to the file generation unit 15 via the non-VCL buffer 14.

[0070] The determination unit 12 determines the type of each image in the image sequence to be encoded by the encoding unit 11. More specifically, in the present embodiment, the determination unit 12 determines at least whether each image is an IDR picture, a CRA picture, or a picture other than these pictures. Both the IDR picture and the CRA picture are pictures composed of only I slices.

[0071] As mentioned earlier, the IDR picture is a picture at the beginning of a sequence. The pictures after the IDR picture in the same sequence do not refer to the pictures before the IDR picture in the coding order (decoding order), and are located only after the IDR picture in the presentation order. The CRA picture is a picture located in the middle of the sequence, and is used as a decoding start picture at the time of random access on the decoder side. The pictures after the CRA picture in both the coding order (decoding order) and the presentation order do not refer to the pictures before the CRA picture in the coding order (decoding order) and the pictures before the CRA picture in the presentation order. The determination unit 12 outputs the determination result to the encoding unit 11 and the file generation unit 15. The encoding unit 11 assigns the NAL unit type indicating the type of each image determined by the determination unit 12 to the NAL header of each NAL unit.

[0072] By the way, in the case of performing random access on a CRA picture, only the CRA picture as the random access destination and the picture after the CRA picture in decoding order are taken as decoding targets. However, there may be a picture after the CRA picture in decoding order but before the CRA picture in presentation order. In this specification, such a picture is referred to as a preceding picture. As can be seen from the definition of the CRA picture, the preceding picture is allowed to refer to the picture before the CRA picture in decoding order.

[0073] In the case of performing random access on a CRA picture, a previous picture that refers to a picture before the CRA picture in decoding order is not decoded normally. This is because the picture referenced by the previous picture has not been decoded. That is, in the case of performing random access, whether the previous picture targeted for decoding can be decoded normally depends on the reference relationship of the previous picture. Therefore, the determination unit 12 can also determine the previous picture that cannot be decoded normally in the case of performing random access on each CRA picture, and provide the determination result to the file generation unit 15.

[0074] The VCL buffer 13 buffers the VCL NAL unit. The slice data of the CRA picture is buffered by the VCL buffer 13.

[0075] The non-VCL buffer 14 buffers non-VCL NAL units.

[0076] The file generation unit 15 generates a moving image file storing a series of encoded image data according to a file format including a header area and a data area, and outputs the generated moving image file to the storage unit 20. In this specification, an example in which the MP4 format is used as the file format will be mainly described. However, the technology according to the present disclosure is not limited to such an example, and can be applied to other types of moving image file formats having a header area and a data area.

[0077] In the MP4 format, data is stored in objects called boxes and recorded in units of objects. These boxes form a tree structure within a file, and a parent box contains child boxes. The type of each box is identified by a four-letter alphabetic identifier.

[0078] More specifically, the file generation unit 15 inserts the bit stream of the slice data corresponding to the VCL NAL unit into the data area (e.g., mdat box) of the motion image file in the decoding order. In addition, the file generation unit 15 inserts one or more parameter sets corresponding to the non-VCL NAL unit into the header area (e.g., moov box) of the motion image file. In addition, in the present embodiment, the file generation unit 15 inserts CRA information identifying one or more CRA pictures determined by the determination unit 12 into the header area of ​​the motion image file. In addition, the file generation unit 15 may append the previous picture information for identifying the previous picture that is not normally decoded when random access is performed on each CRA picture to the CRA information.

[0079] The control unit 16 controls the encoding process performed in the image encoding device 10. For example, when an instruction to start encoding is detected, the control unit 16 causes the encoding unit 11 to encode a specified image sequence. In addition, the control unit 16 causes the file generation unit 15 to generate a moving image file in which the image data encoded by the encoding unit 11 is stored. The control unit 16 can control the generation of the encoded stream using a hypothetical decoder model called a hypothetical reference decoder (HRD) so as not to cause a malfunction in the decoder buffer.

[0080] like Figure 3 As shown, the image decoding device 30 has a configuration including a VCL buffer 31 , a non-VCL buffer 32 , a parameter memory 33 , a decoding unit 34 , an output buffer 35 , and a control unit 37 .

[0081] The VCL buffer 31 buffers a bit stream of image data (typically slice data) read from a data area (for example, an mdat box) of a moving image file stored in the storage unit 20 .

[0082] The non-VCL buffer 32 buffers parameter sets such as SPS, PPS, and APS and header information such as CRA information read from a header area (eg, moov box) of a moving image file stored in the storage unit 20 .

[0083] The parameter memory 33 stores as a whole the information in the header area of ​​the file acquired via the non-VCL buffer 32. When a moving picture file is opened, the parameter memory 33 stores CRA information which can be recorded in the header area of ​​the moving picture file in the above-mentioned various formats.

[0084] The decoding unit 34 is a decoder operating according to the HEVC scheme. The decoding unit 34 decodes an image sequence in a bit stream acquired from a data area of ​​a moving image file via the VCL buffer 31. The decoding unit 34 uses parameters in a parameter set stored in the parameter memory 33 when decoding an image. The decoding unit 34 sorts the images in the decoded image sequence in presentation order and outputs the sorted images to the output buffer 35.

[0085] Normally, the decoding unit 34 accesses the slice data in the moving picture track stored in the moving picture file in the decoding order in the order starting from the beginning of the slice data. However, when the control unit 37 detects a random access instruction, the decoding unit 34 performs random access to the decoding start picture (in the middle of the moving picture track) specified by the control unit 37, and decodes the image sequence starting from the accessed decoding start picture. The decoding start picture is one of the IDR picture and the CRA picture in the moving picture track.

[0086] The output buffer 35 is a decoded picture buffer (DPB) for buffering the image decoded by the decoding unit 34. The image buffered by the output buffer 35 is output to a display or a processor (not shown) at an output timing of the buffered image.

[0087] The control unit 37 controls the image decoding process performed in the image decoding device 30. For example, the control unit 37 opens the moving image file stored in the storage unit 20 in response to an instruction from the user, and causes the decoding unit 34 to start decoding the image sequence. In addition, in the case where a random access instruction is detected, the control unit 37 specifies a CRA picture in the image sequence as a decoding start picture using CRA information. Then, the control unit 37 causes the decoding unit 34 to start decoding the image sequence from the specified decoding start picture (i.e., from the middle of the moving image track).

[0088] Typically, the control unit 37 specifies the CRA picture closest to the timing specified in the random access instruction (for example, the timing pointed to by the pointer operated on the search bar of the moving image reproduction window) as the decoding start picture.

[0089] In the case where the CRA information contains the above-mentioned previous picture information, the control unit 37 can skip the previous image that has not been normally decoded and is identified using the previous picture information from the output of the output buffer 35 (and the decoding by the decoding unit 34). By using the previous picture information, it is possible to prevent a damaged image from being presented on a display or output to an external processor. At this time, the control unit 37 does not have to determine afterwards whether each image has been normally decoded.

[0090] Furthermore, the control unit 37 may send a command to the control unit 16 of the image encoding device 10 .

[0091] <First Mode of Bitstream>

[0092] Will refer to Figures 4 to 11 A first variation of the first mode of describing the bitstream.

[0093] like Figure 4 As shown, a bitstream is composed of a plurality of access units (AUs), and at least one or more NAL units are arranged in each access unit corresponding to one frame. In addition, there are multiple types of NAL units, and for example, the types include an access unit delimiter (AUD) NAL unit, a sequence parameter set (SPS) NAL unit, a picture parameter set (PPS) NAL unit, a tile group NAL unit, etc.

[0094] The AUD NAL unit indicates a delimiter between access units, and generally, only one AUD NAL unit is always arranged at the beginning of each access unit. Note that currently AUD has been given a syntax indicating the attributes of all tile group NALs in an access unit.

[0095] The SPS NAL unit stores the sequence parameters required for reproducing the bitstream. The PPS NAL unit stores the sequence parameters required for reproducing the picture. The tile group NAL unit stores the image of each tile group.

[0096] Then, assume that the three cases shown in the figure are the configurations of the bit stream.

[0097] For example, in the first case, for all access units, the AUD NAL unit is arranged first, the SPS NAL unit is arranged second, the PPS NAL unit is arranged third, and then the tile group NAL unit is arranged continuously. That is, in the first case, the SPS NAL unit and the PPS NAL unit are allocated to each access unit.

[0098] Furthermore, in the second case, for the first access unit, the AUD NAL unit is arranged first, the SPS NAL unit is arranged second, the PPS NAL unit is arranged third, and then the tile group NAL unit is arranged continuously. Then, for the second access unit and subsequent access units, the AUD NAL unit is arranged first, the PPS NAL unit is arranged second, and then the tile group NAL unit is arranged continuously. That is, in the second case, the SPS NAL unit is allocated to the first access unit, and the PPS NAL unit is allocated to each access unit.

[0099] In addition, in the third case, for the first access unit, the AUD NAL unit is arranged first, the SPS NAL unit is arranged second, the PPS NAL unit is arranged third, and then the continuous map block group NAL units are arranged. Then, for the second access unit and subsequent access units, the AUD NAL unit is arranged first, and then, the map block group NAL units are arranged continuously. That is, in the third case, the SPS NAL unit and the PPS NAL unit are allocated only to the first access unit.

[0100] Then, in the first variation of the first mode of the bitstream, in any one of the first to third cases, the configuration is modified according to the past configuration, wherein reference image information indicating a reference image to be referenced by the current picture is arranged in the AUD NAL unit. For example, the reference image information allows specifying a ref_pic_list_idx placed in a tile_group_header or setting a new ref_pic_list_struct. In this way, by arranging the reference image information in the AUD NAL unit placed in the header area of ​​the access unit, and applying the arranged reference image information to all tile group NAL units in the access unit, redundancy can be prevented compared to specifying a reference image for each tile group.

[0101] Specifically, if Figure 5 As shown, the syntax of the AUD NAL in the first variant of the first mode of the bitstream is described. Figure 5 As shown, in the syntax of AUD NAL, the syntax of pic_type is extended so that ref_pic_list_idx can be specified and a new ref_pic_list_struct can be set.

[0102] Will refer to Figure 6 The flowchart shown in exemplifies a NAL generation process in an image encoding process performed by the image encoding device 10 .

[0103] For example, when an image sequence is supplied to the image encoding device 10 , the NAL generation process starts, and in step S11 , the encoding unit 11 determines whether there is a NAL unit to be generated from the image sequence.

[0104] In a case where the encoding unit 11 determines in step S11 that there is a NAL unit to be generated from the image sequence, the process proceeds to step S12 .

[0105] In step S12 , the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is an AUD NAL unit.

[0106] In the case where the encoding unit 11 determines in step S12 that the NAL unit to be generated from the image sequence is an AUD NAL unit, the process proceeds to step S13. Then, in step S13, the encoding unit 11 performs an AUD encoding process to generate an AUD NAL unit containing reference image information, and supplies the generated AUD NAL unit to the file generation unit 15 via the non-VCL buffer 14. Subsequently, the process returns to step S11, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the encoding unit 11 determines in step S12 that the NAL unit to be generated from the image sequence is not an AUD NAL unit, the process proceeds to step S14.

[0107] In step S14 , the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is an SPS NAL unit.

[0108] In the case where the encoding unit 11 determines in step S14 that the NAL unit to be generated from the image sequence is an SPS NAL unit, the process proceeds to step S15. Then, in step S15, the encoding unit 11 performs SPS encoding processing to generate an SPS NAL unit, and supplies the generated SPS NAL unit to the file generation unit 15 via the non-VCL buffer 14. Subsequently, the process returns to step S11, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the encoding unit 11 determines in step S14 that the NAL unit to be generated from the image sequence is not an SPS NAL unit, the process proceeds to step S16.

[0109] In step S16 , the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a PPS NAL unit.

[0110] In the case where the encoding unit 11 determines in step S16 that the NAL unit to be generated from the image sequence is a PPS NAL unit, the process proceeds to step S17. Then, in step S17, the encoding unit 11 performs a PPS encoding process to generate a PPS NAL unit, and supplies the generated PPS NAL unit to the file generation unit 15 via the non-VCL buffer 14. Subsequently, the process returns to step S11, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the encoding unit 11 determines in step S16 that the NAL unit to be generated from the image sequence is not a PPS NAL unit, the process proceeds to step S18.

[0111] In step S18 , the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a tile group NAL unit.

[0112] In the case where the encoding unit 11 determines in step S18 that the NAL unit to be generated from the image sequence is a tile group NAL unit, the process proceeds to step S19. Then, in step S19, the encoding unit 11 performs a tile group encoding process to generate a tile group NAL unit, and supplies the generated tile group NAL unit to the file generation unit 15 via the VCL buffer 13. Subsequently, the process returns to step S11, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the encoding unit 11 determines in step S18 that the NAL unit to be generated from the image sequence is not a tile group NAL unit, the process proceeds to step S20.

[0113] In step S20, the encoding unit 11 performs an encoding process of encoding another NAL unit other than any of the above-mentioned NAL units. Subsequently, the process returns to step S11, and similar processes are repeatedly performed thereafter.

[0114] On the other hand, in a case where the encoding unit 11 determines in step S11 that there is no NAL unit to be generated from the image sequence, the NAL generation process ends.

[0115] Figure 7 It is explained in Figure 6 Flowchart of the AUD encoding process performed in step S13.

[0116] In step S31 , the encoding unit 11 performs a process of specifying and setting a reference frame list based on the reference image information, and is allowed to specify ref_pic_list_idx placed in tile_group_header or set a new ref_pic_list_struct, for example.

[0117] In step S32, the encoding unit 11 generates an AUD NAL unit by performing a frame delimiting process of delimiting frames (access units), and then, the AUD encoding process ends.

[0118] Figure 8 It is explained in Figure 6 Flowchart of the tile group encoding process performed in step S19 of FIG.

[0119] In step S41 , the encoding unit 11 performs a process of encoding a header in addition to a process of specifying and setting a reference frame list.

[0120] In step S42 , the encoding unit 11 performs an encoding process of encoding a unit after the header, and then, the tile group encoding process ends.

[0121] Through the NAL generation process as described above, the image encoding device 10 can generate an AUD NAL unit in which reference image information is arranged, and generate an AUD NAL unit composed of the reference image information as described above. Figure 4 A moving picture file composed of a bit stream described, that is, an image of a bit stream composed of access units in which at least one or more NAL units are arranged, is encoded.

[0122] Will refer to Fig. 9 The flowchart shown in exemplifies a NAL generation process in an image decoding process performed by the image decoding device 30 .

[0123] For example, when the image decoding device 30 reads out a bit stream stored in the storage unit 20 , the process starts, and in step S51 , the decoding unit 34 determines whether there is a NAL unit to be decoded from the bit stream.

[0124] In a case where the decoding unit 34 determines in step S51 that there is a NAL unit to be decoded from the bit stream, the process proceeds to step S52 .

[0125] In step S52 , the decoding unit 34 determines whether the NAL unit to be decoded from the bit stream is an AUD NAL unit.

[0126] In the case where the decoding unit 34 determines in step S52 that the NAL unit to be decoded from the bit stream is an AUD NAL unit, the process proceeds to step S53. Then, in step S53, the decoding unit 34 provides the reference image information obtained by performing the AUD decoding process to decode the AUD NAL unit to the parameter memory 33. Subsequently, the process returns to step S51, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the decoding unit 34 determines in step S52 that the NAL unit to be decoded from the bit stream is not an AUD NAL unit, the process proceeds to step S54.

[0127] In step S54 , the decoding unit 34 determines whether the NAL unit to be decoded from the bit stream is an SPS NAL unit.

[0128] In the case where the decoding unit 34 determines in step S54 that the NAL unit to be decoded from the bit stream is an SPS NAL unit, the process proceeds to step S55. Then, in step S55, the decoding unit 34 supplies the parameters acquired by performing the SPS decoding process to decode the SPS NAL unit to the parameter memory 33. Subsequently, the process returns to step S51, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the decoding unit 34 determines in step S54 that the NAL unit to be decoded from the bit stream is not an SPS NAL unit, the process proceeds to step S56.

[0129] In step S56 , the decoding unit 34 determines whether the NAL unit to be decoded from the bit stream is a PPS NAL unit.

[0130] In the case where the decoding unit 34 determines in step S56 that the NAL unit to be decoded from the bit stream is a PPS NAL unit, the process proceeds to step S57. Then, in step S57, the decoding unit 34 supplies the parameters acquired by performing the PPS decoding process to decode the PPS NAL unit to the parameter memory 33. Subsequently, the process returns to step S51, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the decoding unit 34 determines in step S56 that the NAL unit to be decoded from the bit stream is not a PPS NAL unit, the process proceeds to step S58.

[0131] In step S58 , the decoding unit 34 determines whether the NAL unit to be decoded from the bitstream is a tile group NAL unit.

[0132] In the case where the decoding unit 34 determines in step S58 that the NAL unit to be decoded from the bit stream is a tile group NAL unit, the process proceeds to step S59. Then, in step S59, the decoding unit 34 supplies the image obtained by performing the tile group decoding process to decode the tile group NAL unit to the output buffer 35. Subsequently, the process returns to step S51, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the decoding unit 34 determines in step S58 that the NAL unit to be decoded from the bit stream is not a tile group NAL unit, the process proceeds to step S60.

[0133] In step S60, the decoding unit 34 performs a decoding process of decoding another NAL unit other than any of the above-mentioned NAL units. Subsequently, the process returns to step S51, and similar processes are repeatedly performed thereafter.

[0134] On the other hand, in a case where the decoding unit 34 determines in step S51 that there is no NAL unit to be decoded from the bit stream, the NAL generation process ends.

[0135] Fig.10 It is explained in Fig. 9 Flowchart of the AUD decoding process performed in step S53 of .

[0136] In step S71 , the decoding unit 34 acquires the picture type from the AUD NAL unit.

[0137] In step S72 , the decoding unit 34 acquires reference image information from the AUD NAL unit, and performs a process of specifying and setting a reference frame list according to the reference image information.

[0138] In step S73 , the decoding unit 34 decodes the AUD NAL unit by performing a frame delimiting process of delimiting frames (access units), and then, the AUD decoding process ends.

[0139] Fig.11 It is explained in Fig. 9 Flowchart of the tile group decoding process performed in step S59 of .

[0140] In step S81, the decoding unit 34 performs a process of decoding a header in addition to a process of specifying and setting a reference frame list.

[0141] In step S82 , the decoding unit 34 performs a decoding process of decoding a unit after the header, and then the tile group decoding process ends.

[0142] As described above, the image decoding device 30 can decode an image based on the reference image information acquired from the AUD NAL unit, that is, decode an image of a bit stream composed of an access unit in which at least one or more NAL units are arranged.

[0143] Will refer to Figures 12 to 17 A second variation of the first mode of describing a bitstream.

[0144] like Fig.12 As shown above, Figure 4 Similar to the case described above, the bit stream is composed of multiple access units, and at least one or more NAL units are arranged in each access unit. Figure 4 ) is similar, assuming that Fig.12 The three cases are shown in FIG. 1 as the configuration of the bit stream.

[0145] Then, in the second variation of the first mode of the bitstream, in addition to the AUD NAL unit, the SPS NAL unit, the PPSNAL unit, and the tile group NAL unit, the NEW NAL unit is also used as the NAL unit.

[0146] For example, in the first case, for all access units, the AUD NAL unit is arranged first, the SPS NAL unit is arranged second, the PPS NAL unit is arranged third, the NEW NAL unit is arranged fourth, and then the tile group NAL unit is arranged continuously. That is, in the first case, the SPS NAL unit and the PPS NAL unit are allocated to each access unit, and the NEW NAL unit is also allocated.

[0147] Furthermore, in the second case, for the first access unit, the AUD NAL unit is arranged first, the SPS NAL unit is arranged second, the PPS NAL unit is arranged third, the NEW NAL unit is arranged fourth, and then the tile group NAL unit is arranged continuously. Then, for the second access unit and subsequent access units, the AUD NAL unit is arranged first, the PPS NAL unit is arranged second, the NEW NAL unit is arranged third, and then the tile group NAL unit is arranged continuously. That is, in the second case, the SPS NAL unit is allocated to the first access unit, the PPS NAL unit is allocated to each access unit, and the NEW NAL unit is also allocated.

[0148] In addition, in the third case, for the first access unit, the AUD NAL unit is arranged first, the SPS NAL unit is arranged second, the PPS NAL unit is arranged third, the NEW NAL unit is arranged fourth, and then the tile group NAL unit is arranged continuously. Then, for the second access unit and subsequent access units, the AUD NAL unit is arranged first, the NEW NAL unit is arranged second, and then the tile group NAL unit is arranged continuously. That is, in the third case, the SPS NAL unit and the PPS NAL unit are allocated only to the first access unit, and the NEW NAL unit is allocated to each access unit.

[0149] Then, in a second variation of the first mode of the bitstream, in any one of the first to third cases, the configuration is modified according to the past configuration, wherein reference image information indicating a reference image to be referenced by the current picture is arranged in a NEW NAL unit. For example, the reference image information allows specification of a ref_pic_list_idx placed in a tile_group_header or setting of a new ref_pic_list_struct. In this way, by arranging the reference image information in a NEW NAL unit (NAL unit for identification) placed in a header area of ​​an access unit and applying the arranged reference image information to all tile group NAL units in the access unit, redundancy can be prevented compared to specifying a reference image for each tile group.

[0150] Specifically, if Fig.13 As shown in FIG. 1 , the syntax of the NEW NAL unit in the second variant of the first mode of the bitstream is described. Fig.13 As shown, in the syntax of NEW NAL, ref_pic_list_idx can be specified and a new ref_pic_list_struct can be set.

[0151] Will refer to Fig.14 The flowchart shown in exemplifies a NAL generation process performed in the image encoding process executed by the image encoding device 10 .

[0152] In steps S101 to S107, the same steps as those described above are performed. Figure 6 The encoding unit 11 is a process similar to the processes in steps S11 to S17 described in the flowchart in . Then, in step S108 , the encoding unit 11 determines whether the NAL unit to be generated from the image sequence is a NEW NAL unit.

[0153] In the case where the encoding unit 11 determines in step S108 that the NAL unit to be generated from the image sequence is a NEW NAL unit, the process proceeds to step S109. Then, in step S109, the encoding unit 11 performs NEW encoding processing to generate a NEW NAL unit containing reference image information, and supplies the generated NEW NAL unit to the file generation unit 15 via the non-VCL buffer 14. Subsequently, the process returns to step S101, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the encoding unit 11 determines in step S108 that the NAL unit to be generated from the image sequence is not a NEW NAL unit, the process proceeds to step S110.

[0154] Then, in steps S110 to S112, the same steps as those in the above reference are performed. Figure 6 The processing is similar to the processing in steps S18 to S20 described in the flowchart in , and then, the NAL generation processing ends.

[0155] Fig.15 It is explained in Fig.14 Flowchart of the NEW encoding process performed in step S109.

[0156] In step S121, the encoding unit 11 performs a process of specifying and setting a reference frame list based on the reference image information, and for example, specifies ref_pic_list_idx placed in tile_group_header or sets a new ref_pic_list_struct. In this way, the encoding unit 11 generates a NEW NAL unit, and then the NEW encoding process ends.

[0157] Note that, as mentioned above Figure 8 The tile group encoding process described in the flowchart in is similarly performed in Fig.14 The tile group encoding process is performed in step S111.

[0158] Through the NAL generation process as described above, the image encoding device 10 can generate a NEW NAL unit in which reference image information is arranged, and generate a NEW NAL unit composed of the reference image information as described above. Fig.12 A moving picture file composed of a bit stream described, that is, an image of a bit stream composed of access units in which at least one or more NAL units are arranged, is encoded.

[0159] Will refer to Fig.16 The flowchart shown in exemplifies a NAL generation process in an image decoding process performed by the image decoding device 30 .

[0160] In steps S131 to S137, the same steps as those described above are performed. Fig. 9The decoding unit 34 performs processing similar to the processing in steps S51 to S57 described in the flowchart in . Then, in step S138 , the decoding unit 34 determines whether the NAL unit to be decoded from the bit stream is a NEW NAL unit.

[0161] In the case where the decoding unit 34 determines in step S138 that the NAL unit to be decoded from the bit stream is a NEW NAL unit, the process proceeds to step S139. Then, in step S139, the decoding unit 34 supplies the reference image information acquired by performing the NEW decoding process to decode the NEW NAL unit to the parameter memory 33. Subsequently, the process returns to step S131, and similar processes are repeatedly performed thereafter. On the other hand, in the case where the decoding unit 34 determines in step S138 that the NAL unit to be decoded from the bit stream is not a NEW NAL unit, the process proceeds to step S140.

[0162] Then, in steps S140 to S142, the same steps as those described above are performed. Fig. 9 The processing is similar to the processing in steps S58 to S60 described in the flowchart in , and then, the NAL generation processing ends.

[0163] Fig.17 It is explained in Fig.16 Flowchart of the NEW decoding process performed in step S139.

[0164] In step S151, the decoding unit 34 acquires the reference image information from the NEW NAL unit, and decodes the NEW NAL unit by performing a process of specifying and setting a reference frame list according to the reference image information. Subsequently, the NEW decoding process ends.

[0165] Note that, as mentioned above Fig.11 The picture tile group decoding process described in the flowchart in is similarly performed in Fig.16 The tile group decoding process is performed in step S141.

[0166] As described above, the image decoding device 30 can decode an image based on the reference image information acquired from the NEW NAL unit, that is, decode an image of a bit stream composed of access units in which at least one or more NAL units are arranged.

[0167] <Second Mode of Bitstream>

[0168] Will refer to Figures 18 to 26 Describes the second mode of the bitstream.

[0169] In the second mode of the bitstream, a long-term reference, which may be set irregularly, may be modified dynamically.

[0170] First, as an example of dynamically modifying a long-term reference, we will refer to Fig.18 A technique for restoring image quality in narrowband using a long-term reference is described.

[0171] For example, in the event that packet loss occurs at the timing shown in the figure, the image decoding device 30 as the receiving side notifies the image encoding device 10 as the transmitting side of error occurrence information indicating that packet loss has occurred. Then, the image encoding device 10 avoids retransmitting the I picture and refers to the long-term reference picture (LTR frame P). L ) encodes the frame P′ to be encoded after receiving the error occurrence information.

[0172] Typically, long-term reference pictures are set (updated) irregularly when the motion is relatively large. For example, short-term reference pictures are encoded with a fixed group of pictures (GOP) structure and therefore do not need to be modified, while it is optional whether to modify long-term reference pictures when pictures are encoded.

[0173] By dynamically modifying the long-term reference in this manner, the image decoding apparatus 30 can decode the image by referring to the LTR frame P even if packet loss has occurred. L By referring to LTR frame P L The encoded frame P' is decoded to recover the image quality degradation caused by packet loss with lower delay.

[0174] As a use case for using such a technique to restore image quality using a long-term reference, for example, low-latency or interactive applications with limited transmission bandwidth are assumed. Specifically, this technique can be used for surveillance cameras, video chats, streaming game services, etc.

[0175] Will refer to Fig.19 and Fig. 20 A first variation of the second mode of describing the bitstream.

[0176] like Fig.19 As shown above, Figure 4 Similar to the case described above, the bitstream is composed of multiple access units, and at least one or more NAL units (AUD NAL unit, SPS NAL unit, PPS NAL unit, and tile group NAL unit) are arranged in each access unit. In addition, as with the first variant of the first mode of the bitstream (see Figure 4 ) is similar, assuming Fig.19 The three cases shown in FIG. 1 are configurations of the bit stream.

[0177] Then, in the first variation of the second mode of the bitstream, in any of the first to third cases, the AUD NAL unit is extended so that only the settings for the long term can be modified independently (modifying poc_lsb_lt in ref_pic_list_struct(i, ref_pic_list_idx[i])).

[0178] Specifically, in a first variation of the second mode of the bit stream, as Fig. 20 As shown, the syntax of SPS NAL and AUDNAL is described.

[0179] For example, a long-term modification of the ref_pic_list_struct already defined in the SPS NAL unit is implemented in the AUD NAL unit.

[0180] Will refer to Fig.21 and Fig. 22 A second variation of the second mode of describing the bitstream.

[0181] like Fig.21 As shown above, Figure 4 Similar to the case described above, the bitstream is composed of a plurality of access units, and at least one or more NAL units (AUD NAL unit, SPS NAL unit, PPS NAL unit, tile group NAL unit, and NEW NAL unit) are arranged in each access unit. Fig.12 ) is similar, assuming that Fig.21 The three cases shown are configurations of the bitstream.

[0182] Then, in the second variant of the second mode of the bitstream, in any of the first to third cases, the NEW NAL unit is extended so that only the settings for the long term can be modified independently (modifying poc_lsb_lt in ref_pic_list_struct(i, ref_pic_list_idx[i])).

[0183] Specifically, if Fig. 22 FIG. 2 shows a diagram describing the syntax of the SPS NAL and the NEW NAL in the second variant of the second mode of the bitstream.

[0184] For example, long-standing modifications to the ref_pic_list_struct already defined in the SPS NAL unit are implemented in the NEW NAL unit.

[0185] Will refer to Fig.23The flowchart shown in exemplifies a reference frame list specification and setting process performed in the image encoding process performed by the image encoding device 10.

[0186] For example, when an image sequence is supplied to the image encoding device 10 , the reference frame list designation and setting process starts, and in step S161 , the encoding unit 11 determines whether a reference frame list already defined in the SPS NAL is designated.

[0187] In the event that determination is made in step S161 that a reference frame list already defined in the SPS NAL unit is specified, the process proceeds to step S162 , and the encoding unit 11 acquires an index of the reference frame list.

[0188] In step S163 , the encoding unit 11 determines whether to modify the long-term reference frame information according to the index of the reference frame list acquired in step S162 .

[0189] In the case where the encoding unit 11 determines in step S163 that the long-term reference frame information is to be modified, the process proceeds to step S164, and a long-term reference frame information modification process (see the later-described Fig.24 Flowchart in ).

[0190] On the other hand, in a case where it is determined in step S163 that the long-term reference frame information is not to be modified, or after the long-term reference frame information modification processing is performed in step S164, the reference frame list designation and setting processing ends.

[0191] Meanwhile, in the event that determination is made in step S161 that the reference frame list already defined in the SPS NAL unit is not specified, the process proceeds to step S165. In step S165, the encoding unit 11 performs encoding processing on the reference frame list, and then the reference frame list specifying and setting processing ends.

[0192] Fig.24 It is explained in Fig.23 Flowchart of the long-term reference frame information modification process performed in step S164.

[0193] In step S171, the encoding unit 11 obtains Fig.23 The long-term number k in the reference frame list of the index obtained in step S162 is calculated, and the parameter i is set to zero (i=0).

[0194] In step S172, the encoding unit 11 determines whether the parameter i is smaller than the long-term number k.

[0195] In a case where the encoding unit 11 determines in step S172 that the parameter i is smaller than the long term number k (i<k), the process proceeds to step S173.

[0196] In step S173, the encoding unit 11 modifies the absolute position of the i-th long-term reference frame. For example, the encoding unit 11 may set the absolute position of the i-th long-term reference frame based on a command (information specifying a reference image) from the image decoding device 30. In addition, the image decoding device 30 updates the information specifying the reference image for each frame.

[0197] In step S174 , the encoding unit 11 increments the parameter i (i=i+1), and then the process returns to step S172 .

[0198] On the other hand, in a case where the encoding unit 11 determines in step S172 that the parameter i is not less than the long-term number k (i≥k), the long-term reference frame information modification process ends.

[0199] Through the reference frame list designation and setting processing as described above, the image encoding device 10 can modify the long-term reference frame information and generate the long-term reference frame information as described above. Fig.19 or Fig.21 Describes a moving picture file consisting of a bit stream.

[0200] Will refer to Fig.25 The flowchart shown in describes an example of reference frame list specification and setting processing performed in the image decoding processing performed by the image decoding device 30.

[0201] For example, when the image decoding device 30 reads out a bit stream stored in the storage unit 20 , the process starts, and in step S181 , the decoding unit 34 determines whether a reference frame list that has been defined in the SPS NAL is specified.

[0202] In the event that determination is made in step S181 that a reference frame list already defined in the SPS NAL unit is specified, the process proceeds to step S182 , and the decoding unit 34 acquires the index of the reference frame list.

[0203] In step S183 , the decoding unit 34 determines whether to modify the long-term reference frame information based on the index of the reference frame list acquired in step S182 .

[0204] In the case where the decoding unit 34 determines in step S183 that the long-term reference frame information is to be modified, the process proceeds to step S184 and a long-term reference frame information modification process (see the later-described Fig.26 Flowchart in ).

[0205] On the other hand, in a case where it is determined in step S183 that the long-term reference frame information is not to be modified, or after the long-term reference frame information modification processing is performed in step S184, the reference frame list designation and setting processing ends.

[0206] Meanwhile, in the event that determination is made in step S181 that the reference frame list already defined in the SPS NAL unit is not specified, the process proceeds to step S185. In step S185, the decoding unit 34 performs decoding processing on the reference frame list, and then the reference frame list specifying and setting processing ends.

[0207] Fig.26 It is explained in Fig.25 Flowchart of the long-term reference frame information modification processing performed in step S184.

[0208] In step S191, the decoding unit 34 obtains Fig.25 The long-term number k in the reference frame list of the index obtained in step S182 is calculated, and the parameter i is set to zero (i=0).

[0209] In step S192, the decoding unit 34 determines whether the parameter i is smaller than the long term number k.

[0210] In a case where the decoding unit 34 determines in step S192 that the parameter i is smaller than the long term number k (i<k), the process proceeds to step S193.

[0211] In step S193, the decoding unit 34 modifies the absolute position of the i-th long-term reference frame according to the long-term reference frame information.

[0212] In step S194 , the decoding unit 34 increments the parameter i (i=i+1), and then the process returns to step S192 .

[0213] On the other hand, in a case where the decoding unit 34 determines in step S192 that the parameter i is not less than the long-term number k (i≥k), the long-term reference frame information modification process ends.

[0214] Through the reference frame list designation and setting process as described above, the image decoding device 30 can Fig.19 or Fig.21 The bit stream described in the moving picture file is composed of modified long-term reference frame information to decode the picture.

[0215] <Exemplary Configuration of Computer>

[0216] Next, the above-mentioned series of processing (image decoding method and image encoding method) can be executed by hardware or by software. In the case where the series of processing is executed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0217] Fig. 27 : is a block diagram showing an exemplary configuration of a computer on which a program that executes the above-described series of processes is installed according to the embodiment.

[0218] The program may be recorded in advance on a hard disk 105 or a read only memory (ROM) 103 as a recording medium built in the computer.

[0219] Alternatively, the program may be saved (recorded) in a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 may be provided as so-called packaged software. Here, examples of the removable recording medium 111 include a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a magnetic disk, and a semiconductor memory.

[0220] Note that not only can the program be installed on the computer from the removable recording medium 111 as described above, but the program can also be downloaded to the computer via a communication network or a broadcasting network to be installed on the built-in hard disk 105. That is, for example, the program can be wirelessly transmitted from a download site to the computer via an artificial satellite for digital satellite broadcasting, or the program can be transmitted to the computer by wire via a network such as a local area network (LAN) or the Internet.

[0221] The computer has a built-in central processing unit (CPU) 102 and an input / output interface 110 is connected to the CPU 102 via a bus 101 .

[0222] When a command is input via the input / output interface 110 in response to, for example, a user operating the input unit 107, the CPU 102 executes a program stored in a read-only memory (ROM) 103 according to the command. Alternatively, the CPU 102 loads a program stored in a hard disk 105 into a random access memory (RAM) 104 and executes the loaded program.

[0223] This causes the CPU 102 to execute the processing according to the above flowchart or the processing executed in the configuration in the above block diagram. Then, the CPU 102 outputs the processing result from the output unit 106 or transmits the result from the communication unit 108 as needed, for example, via the input / output interface 110, and additionally, records the result on, for example, the hard disk 105.

[0224] Note that the input unit 107 is constituted by a keyboard, a mouse, a microphone, etc. Furthermore, the output unit 106 is constituted by a liquid crystal display (LCD), a speaker, and the like.

[0225] Here, in this specification, the processing performed by the computer according to the program does not necessarily have to be performed in chronological order in the order described in the flowchart. That is, the processing performed by the computer according to the program includes processing performed in parallel or individually (for example, parallel processing or processing performed by an object).

[0226] In addition, the program may be processed by a single computer (processor), or alternatively, may be processed by a plurality of computers in a distributed manner. Furthermore, the program may be transmitted to a remote computer and executed.

[0227] In addition, in this specification, a system represents a collection of multiple components (e.g., devices and modules (components)), and it is not important whether all the components are placed in the same cabinet. Therefore, multiple devices housed in different cabinets to be connected to each other via a network and one device in which multiple modules are housed in one cabinet are both considered to be a system.

[0228] In addition, for example, a configuration described as one device (or processing unit) may be divided so as to be configured as a plurality of devices (or processing units). Conversely, the configuration described above as a plurality of devices (or processing units) may be integrated so as to be configured as one device (or processing unit). In addition, of course, a configuration other than the above configuration may be added to the configuration of the corresponding device (or processing unit). In addition, as long as the configuration or action of the system as a whole remains substantially unchanged, a portion of the configuration of a specific device (or processing unit) may be included in the configuration of another device (or processing unit).

[0229] Meanwhile, for example, the present technology may take a cloud computing configuration in which one function is divided and distributed to a plurality of devices so as to be processed in coordination among the plurality of devices via a network.

[0230] In addition, for example, the above-mentioned program can be executed by any device. In that case, it is only necessary that the device has necessary functions (functional blocks, etc.) so that those necessary information can be obtained.

[0231] In addition, for example, each step described in the above flowchart may be shared and executed by multiple devices, or may be executed by a single device. In addition, in the case where multiple processes are included in one step, the multiple processes included in one step may be shared and executed by multiple devices, or may be executed by a single device. In other words, the multiple processes included in one step may also be executed as processes in multiple steps. Conversely, the processes described as multiple steps may also be integrated into one step and executed.

[0232] Note that the program executed by the computer can be designed in such a way that the processing of the steps describing the program is performed in chronological order in the order described in this specification, or is performed separately in parallel or at necessary timing, such as when called. In other words, the processing of the individual steps can be performed in an order different from the above order as long as there is no inconsistency. In addition, these processing of the steps describing the program can be performed in parallel with the processing of other programs, or can be performed in combination with the processing of another program.

[0233] Note that, as long as there is no inconsistency, each of the multiple technologies described in this specification can be independently executed separately. Of course, any multiple technologies can also be executed simultaneously. For example, part or all of the technology described in any embodiment can be combined with part or all of the technology described in another embodiment. In addition, part or all of any of the technologies described above can also be executed simultaneously with other technologies not mentioned above.

[0234] <Exemplary Combinations of Configurations>

[0235] Note that the present technology may also be configured as follows. (1)

[0237] An image decoding device, comprising:

[0238] A decoding unit, which decodes an image of a bit stream consisting of access units, in which at least one or more NAL (Network Abstraction Layer) units are arranged, wherein

[0239] Reference image information indicating a reference image to be referred to by the current picture has been stored in a header area of ​​the access unit. (2)

[0241] The image decoding device according to (1) above, wherein:

[0242] The reference image information has been stored in an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (3)

[0244] The image decoding device according to (1) above, wherein:

[0245] The reference image information has been stored in a NAL unit for identification provided separately from an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (4)

[0247] The image decoding device according to any one of (1) to (3) above, wherein:

[0248] The reference image information includes long-term information indicating whether the reference image is designated as a long-term reference picture. (5)

[0250] The image decoding device according to (4) above, wherein:

[0251] The long-term information includes information specifying the reference image decided based on information from a client which is a receiving side of the bit stream. (6)

[0253] The image decoding device according to (5) above, wherein:

[0254] The information from the client is updated every frame. (7)

[0256] The image decoding device according to any one of (4) to (6) above, wherein:

[0257] The long-term information has been stored in an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (8)

[0259] The image decoding device according to any one of (4) to (6) above, wherein:

[0260] The long-term information has been stored in a NAL unit for identification provided separately from an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (9)

[0262] An image decoding method, comprising:

[0263] An image decoding device that performs image decoding processing decodes an image of a bit stream composed of access units in which at least one or more NAL (Network Abstraction Layer) units are arranged, wherein

[0264] Reference image information indicating a reference image to be referred to by the current picture has been stored in a header area of ​​the access unit. (10)

[0266] An image encoding device, comprising:

[0267] A coding unit for coding an image of a bit stream consisting of access units in which at least one or more NAL (Network Abstraction Layer) units are arranged, wherein

[0268] Reference image information indicating a reference image to be referred to by a current picture is stored in a header area of ​​the access unit. (11)

[0270] The image encoding device according to (10) above, wherein:

[0271] The reference image information is stored in an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (12)

[0273] The image encoding device according to (10) above, wherein:

[0274] The reference image information is stored in a NAL unit for identification provided separately from an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (13)

[0276] The image encoding device according to any one of (10) to (12) above, wherein:

[0277] The reference image information includes long-term information indicating whether the reference image is designated as a long-term reference picture. (14)

[0279] The image encoding device according to (13) above, wherein:

[0280] The long-term information includes information specifying a reference image decided based on information from a client which is a receiving side of the bit stream. (15)

[0282] The image encoding device according to (14) above, wherein:

[0283] The information from the client is updated every frame. (16)

[0285] The image encoding device according to any one of (13) to (15) above, wherein:

[0286] The long-term information is stored in an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (17)

[0288] The image encoding device according to any one of (13) to (15) above, wherein:

[0289] The long-term information is stored in a NAL unit for identification provided separately from an AUD (Access Unit Delimiter) NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit. (18)

[0291] An image encoding method, comprising:

[0292] An image encoding device that performs an image encoding process encodes an image of a bit stream composed of access units in which at least one or more NAL (Network Abstraction Layer) units are arranged, wherein:

[0293] Reference image information indicating a reference image to be referred to by a current picture is stored in a header area of ​​the access unit.

[0294] Note that the present embodiment is not limited to the above-described embodiment, and various modifications may be made without departing from the scope of the present disclosure. In addition, the effects described in this specification are only used as examples and should not be interpreted as being limiting. There may be other effects.

[0295] Reference numerals list

[0296] 10 image encoding device, 11 encoding unit, 12 determination unit, 13 VCL buffer, 14 non-VCL buffer, 15 file generation unit, 16 control unit, 20 storage unit, 30 image decoding device, 31 VCL buffer, 32 non-VCL buffer, 33 parameter memory, 34 decoding unit, 35 output buffer, 37 control unit.

Claims

1. An image decoding device, comprising: A decoding unit, which decodes an image of a bit stream consisting of access units, in which at least one or more network abstraction layer NAL units are arranged, wherein Reference picture information indicating a reference picture to be referred to by the current picture has been stored in the header area of ​​the access unit, and The reference image information is stored in a NAL unit for identification that is separately provided from an access unit delimiter AUD NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit.

2. The image decoding device according to claim 1, wherein: The reference image information includes long-term information indicating whether the reference image is designated as a long-term reference picture.

3. The image decoding device according to claim 2, wherein: The long-term information includes information specifying the reference image decided based on information from a client which is a receiving side of the bit stream.

4. The image decoding device according to claim 3, wherein: The information from the client is updated every frame.

5. An image decoding method, comprising: An image decoding device that performs image decoding processing decodes an image of a bit stream composed of access units, in which at least one or more network abstraction layer NAL units are arranged, wherein Reference picture information indicating a reference picture to be referred to by the current picture has been stored in the header area of ​​the access unit, and The reference image information is stored in a NAL unit for identification that is separately provided from an access unit delimiter AUD NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit.

6. An image encoding device, comprising: A coding unit for coding an image of a bit stream consisting of access units, in which at least one or more network abstraction layer NAL units are arranged, wherein: Reference picture information indicating a reference picture to be referred to by the current picture is stored in a header area of ​​the access unit, and The reference image information is stored in a NAL unit for identification that is separately provided from an access unit delimiter AUD NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit.

7. The image encoding device according to claim 6, wherein: The reference image information includes long-term information indicating whether the reference image is designated as a long-term reference picture.

8. The image encoding device according to claim 7, wherein: The long-term information includes information specifying the reference image decided based on information from a client which is a receiving side of the bit stream.

9. The image encoding device according to claim 8, wherein: The information from the client is updated every frame.

10. An image encoding method, comprising: An image encoding device that performs an image encoding process encodes an image of a bit stream composed of access units, in which at least one or more network abstraction layer NAL units are arranged, wherein: Reference picture information indicating a reference picture to be referred to by the current picture is stored in a header area of ​​the access unit, and The reference image information is stored in a NAL unit for identification that is separately provided from an access unit delimiter AUD NAL unit indicating a delimiter of the access unit, and is applied to all tile group NAL units in the access unit.

Images