Decoding apparatus, encoding apparatus, decoding method, and encoding method

Abstract

A decoding apparatus has a circuit that acquires first existence flag information indicating whether first video usability information is described in a bitstream from the bitstream, acquires the first video usability information from the bitstream in a case where the first existence flag information is true, acquires first extension flag information indicating whether second video usability information is described in the bitstream from the bitstream, acquires the second video usability information from the bitstream in a case where the first extension flag information is true, acquires second extension flag information indicating whether first extension information is described in the bitstream from the bitstream, the first extension information extending other information of a NAL (Network Abstraction Layer) unit in which the second video usability information is described, and acquires the first extension information from the bitstream in a case where the second extension flag information is true.

Description

Technical Field

[0001] This disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method. Background Technology

[0002] Patent Document 1 discloses an image processing system related to the background art. This image processing system includes an encoder (encoding device) and a decoder (decoding device). Various modalities of input images are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image parsing task based on the input features, thereby outputting a segmentation map.

[0003] However, in the background art, no research has been conducted on methods for extending the VUI (video usability information) parameter in encoding and decoding devices corresponding to AVC (H.264) or HEVC (H.265), which are existing video coding standards.

[0004] Prior art literature

[0005] Patent documents

[0006] Patent Document 1: U.S. Patent Application Publication No. 2024 / 0046453 Summary of the Invention

[0007] The purpose of this disclosure is to provide a decoding apparatus, encoding apparatus, decoding method, and encoding method that can appropriately extend VUI parameters within existing video coding standards.

[0008] One aspect of the decoding apparatus disclosed herein includes: a circuit; and a memory connected to the circuit, wherein the circuit obtains from a bitstream the first presence flag indicating whether first video availability information is recorded in the bitstream; if the first presence flag is true, it obtains the first video availability information from the bitstream; obtains from the bitstream the first extension flag indicating whether second video availability information is recorded in the bitstream; if the first extension flag is true, it obtains the second video availability information from the bitstream; obtains from the bitstream the second extension flag indicating whether the first extension information is recorded in the bitstream; the first extension information extends other information of the NAL (Network Abstraction Layer) cell that records the second video availability information; and if the second extension flag is true, it obtains the first extension information from the bitstream. Attached Figure Description

[0009] Figure 1 This is a simplified diagram illustrating the structure of the image processing system according to embodiments of the present disclosure.

[0010] Figure 2 This is a simplified diagram showing the structure of the circuitry of the encoding device.

[0011] Figure 3 This is a flowchart illustrating the processing performed by the circuitry of the encoding device.

[0012] Figure 4 This is a simplified diagram illustrating the structure of a bitstream.

[0013] Figure 5 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0014] Figure 6 This is a diagram illustrating an example of the correspondence between the value of vui_modality_type and the image category.

[0015] Figure 7 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0016] Figure 8 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0017] Figure 9 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0018] Figure 10 This is a diagram illustrating an example of the correspondence between index values ​​and wavelength information.

[0019] Figure 11 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0020] Figure 12 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0021] Figure 13 This is a diagram illustrating an example of the correspondence between the values ​​of the representation type information and the interpretation of each value.

[0022] Figure 14 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0023] Figure 15 This is a diagram illustrating an example of the correspondence between the values ​​of the interpolation information and their interpretations.

[0024] Figure 16 This is a diagram illustrating an example of the structure of a bitstream.

[0025] Figure 17This is a simplified diagram showing the structure of the circuitry of the decoding device.

[0026] Figure 18 This is a flowchart illustrating the processing performed by the circuitry of the decoding device.

[0027] Figure 19 This is a simplified illustration of an example of table information.

[0028] Figure 20 This is a simplified illustration of an example of table information.

[0029] Figure 21 This is a block diagram illustrating an example of the functional structure of the encoding section.

[0030] Figure 22 This is a diagram illustrating an example of the hierarchical structure of data in a stream.

[0031] Figure 23 This is a block diagram illustrating an example of the functional structure of the decoding unit.

[0032] Figure 24 This is a simplified illustration of the first example of the syntax for SPS in HEVC.

[0033] Figure 25 This is a simplified diagram illustrating an example of the syntax for the first video availability information in HEVC or AVC.

[0034] Figure 26 This is a simplified illustration of the second example of the SPS syntax in HEVC.

[0035] Figure 27 This is a simplified diagram illustrating the first example of the syntax for SPS in AVC.

[0036] Figure 28 This is a simplified diagram illustrating the first example of the syntax for SPS in AVC.

[0037] Figure 29 This is a simplified illustration of the second example of the SPS syntax in AVC.

[0038] Figure 30 This is a simplified diagram illustrating the first example of the syntax for the second video availability information in HEVC or AVC.

[0039] Figure 31 This is a simplified diagram illustrating the second example of the syntax for the second video availability information in HEVC or AVC.

[0040] Figure 32 This is a simplified diagram illustrating the third example of the syntax for the second video availability information in HEVC or AVC. Detailed Implementation

[0041] (The knowledge that forms the basis of this disclosure)

[0042] The background technology relates to image processing systems that include encoding and decoding units. Various modalities of input images are input to the encoding unit. The encoding unit extracts features from the input images and inputs the extracted features to the decoding unit. The decoding unit performs image parsing tasks based on the input features, thereby outputting a segmentation map.

[0043] The tasks performed by the decoding device include both human vision and machine tasks. Human vision refers to the visual or auditory confirmation of moving images by operators or users. Machine tasks include a wide variety of tasks that utilize AI models, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0044] In the prior art, image association information, such as modal information representing image categories, is not transmitted from the encoding device to the decoding device. Therefore, sometimes the optimal task processing corresponding to the image category of the input image is not selected, resulting in a decrease in the execution accuracy of the task processing. Therefore, in this disclosure, the encoding device includes image association information, such as modal information, in the bitstream and transmits it to the decoding device, which then performs optimal task processing based on the image category represented by the modal information, thereby improving execution accuracy.

[0045] The header region of a bitstream contains video usability information (VUI), which provides information related to image characteristics such as aspect ratio. With the increasing prevalence of video applications, VVC (H.266), as the next-generation video coding standard, has explored extending and including image-related information such as modal information in the VUI parameters. However, it is not possible to directly extend the VUI parameters in existing video coding standards such as AVC (H.264) or HEVC (H.265). Therefore, encoding and decoding devices corresponding to AVC or HEVC cannot properly handle image-related information such as modal information extended to the VUI parameters.

[0046] In order to solve the above-mentioned problems, the inventors of the present invention obtained the knowledge that by using extended flag information to appropriately hierarchically classify the syntax of SPS (Sequence Parameter Set), the above-mentioned problems can be solved, and thus the present disclosure was conceived.

[0047] Next, the various methods disclosed herein will be explained.

[0048] The decoding apparatus according to the first aspect of this disclosure includes: a circuit; and a memory connected to the circuit, wherein the circuit obtains from a bitstream a first presence flag indicating whether first video availability information is recorded in the bitstream; if the first presence flag is true, it obtains the first video availability information from the bitstream; obtains from the bitstream a first extension flag indicating whether second video availability information is recorded in the bitstream; if the first extension flag is true, it obtains the second video availability information from the bitstream; obtains from the bitstream a second extension flag indicating whether the first extension information is recorded in the bitstream; the first extension information extends other information of the NAL (Network Abstraction Layer) unit that records the second video availability information; if the second extension flag is true, it obtains the first extension information from the bitstream.

[0049] According to the first method, the video availability information can be appropriately extended using the first extended flag information and the second video availability information. Furthermore, other information of the NAL unit describing the second video availability information can be appropriately extended using the second extended flag information and the first extended information.

[0050] The decoding apparatus according to the second aspect of this disclosure, in the first aspect, may involve the circuit acquiring the first presence flag information, the first video availability information, the first extended flag information, the second video availability information, the second extended flag information, and the first extended information from the same NAL unit.

[0051] According to the second method, it can correspond to HEVC (H.265), which is the existing video coding standard.

[0052] The decoding apparatus according to the third aspect of this disclosure, in the second aspect, may be such that the circuit obtains the second video availability information from the NAL unit when the first extended flag information is true and the first existence flag information is true.

[0053] According to the third method, if the first existence flag information is false (that is, if the first video availability information is not included in the bitstream), the second video availability information is not included in the bitstream, thereby reducing the bit volume.

[0054] The decoding apparatus according to the fourth aspect of this disclosure, in the first aspect, may be such that the circuit obtains the first presence flag information and the first video availability information from the first NAL unit, and obtains the first extended flag information, the second video availability information, the second extended flag information, and the first extended information from the second NAL unit.

[0055] According to method 4, it can correspond to AVC (H.264), which is an existing video coding standard.

[0056] In the fourth embodiment, the decoding apparatus of the fifth embodiment of this disclosure may involve the circuit obtaining second presence flag information from the second NAL unit indicating whether the first video availability information is recorded in the bitstream, and obtaining the second video availability information from the second NAL unit when the first extended flag information is true and the second presence flag information is true.

[0057] According to the fifth method, if the second existence flag information is false (that is, if the first video availability information is not included in the bitstream), the second video availability information is not included in the bitstream, thereby reducing the bit volume.

[0058] In the first embodiment, the decoding apparatus according to the sixth aspect of this disclosure may involve the circuit obtaining image association information associated with an image contained in the bitstream from the second video availability information, and, if second extended information further extending the second video availability information is described in the second video availability information, obtaining size information representing the size of the second extended information from the second video availability information, and obtaining the second extended information from the second video availability information based on the size information.

[0059] According to the sixth method, the video availability information can be further expanded using the second extended information. Furthermore, the second extended information can be appropriately obtained from the second video availability information using size information.

[0060] In the first embodiment, the decoding apparatus according to the seventh aspect of this disclosure may involve the circuit obtaining image association information associated with an image contained in the bitstream from the second video availability information, obtaining a third extension flag information indicating whether a second extension information further extending the second video availability information is recorded in the second video availability information, and obtaining the second extension information from the second video availability information if the third extension flag information is true.

[0061] According to method 7, video availability information can be further extended through the third extended flag information and the second extended information.

[0062] In the first embodiment, the decoding apparatus involved in the eighth aspect of this disclosure may be a bit string in which the first extended information has a continuous bit string extending to the end of the NAL unit storing the first extended information.

[0063] According to the eighth method, the first extended information can be easily and appropriately obtained from the NAL unit where the first extended information is stored.

[0064] The decoding apparatus involved in the 9th aspect of this disclosure, in the 7th aspect, may be a bit string in which the second extended information has a continuous bit string extending to the end of the NAL unit storing the second extended information.

[0065] According to the ninth method, the second extended information can be easily and appropriately obtained from the NAL unit where the second extended information is stored.

[0066] In the first embodiment of the decoding apparatus according to the tenth aspect of this disclosure, the second video availability information may include image association information associated with an image contained in the bitstream. In the decoding apparatus where the image association information cannot be interpreted, if the first extended flag information is true, the circuit ignores the second video availability information, the second extended flag information, and the first extended information.

[0067] According to the 10th method, by ignoring the second video availability information, the second extended flag information, and the first extended information, it is possible to avoid the decoding device performing useless processing.

[0068] In the first embodiment of the decoding apparatus according to the 11th aspect of this disclosure, the second video availability information may include image association information associated with an image contained in the bitstream. In the decoding apparatus that can interpret the image association information but cannot interpret the first extended information, if the first extended flag information is true, the circuit obtains the second video availability information and the second extended flag information from the bitstream, and if the second extended flag information is true, ignores the first extended information.

[0069] According to the 11th method, by ignoring the first extended information, it is possible to avoid performing useless processing by the decoding device.

[0070] The encoding apparatus according to the 12th aspect of this disclosure includes: a circuit; and a memory connected to the circuit, wherein the circuit encodes a first presence flag indicating whether first video availability information is recorded in the bitstream, and encodes the first video availability information in the bitstream if the first presence flag is true; encodes a first extension flag indicating whether second video availability information is recorded in the bitstream, and encodes the second video availability information in the bitstream if the first extension flag is true; and encodes a second extension flag indicating whether the first extension information is recorded in the bitstream, wherein the first extension information extends other information of the NAL (Network Abstraction Layer) cell that records the second video availability information, and encodes the first extension information in the bitstream if the second extension flag is true.

[0071] According to the 12th method, the video availability information can be appropriately extended using the first extended flag information and the second video availability information. Furthermore, other information of the NAL unit describing the second video availability information can be appropriately extended using the second extended flag information and the first extended information.

[0072] In the 12th embodiment, the encoding apparatus of the 13th embodiment of this disclosure may be such that the circuit encodes the first presence flag information, the first video availability information, the first extended flag information, the second video availability information, the second extended flag information, and the first extended information in the same NAL unit.

[0073] According to method 13, it can correspond to HEVC (H.265), which is the existing video coding standard.

[0074] In the 13th embodiment, the encoding apparatus according to the 14th embodiment of this disclosure may be such that the circuit encodes the second video availability information in the NAL unit when the first extended flag information is true and the first existence flag information is true.

[0075] According to the 14th method, if the first existence flag information is false (that is, if the first video availability information is not included in the bitstream), the second video availability information is not included in the bitstream, thereby reducing the bit volume.

[0076] In the 12th embodiment, the encoding apparatus according to the 15th embodiment of this disclosure may be such that the circuit encodes the first presence flag information and the first video availability information in a 1NAL unit, and encodes the first extended flag information, the second video availability information, the second extended flag information, and the first extended information in a 2NAL unit.

[0077] According to method 15, it can correspond to AVC (H.264), which is an existing video coding standard.

[0078] In the 15th embodiment, the encoding apparatus according to the 16th embodiment of this disclosure may be such that the circuit encodes a second presence flag indicating whether the first video availability information is recorded in the bitstream in the 2NAL unit, and encodes the second video availability information in the 2NAL unit when the first extended flag information is true and the second presence flag information is true.

[0079] According to the 16th method, if the second existence flag information is false (that is, if the first video availability information is not included in the bitstream), the second video availability information is not included in the bitstream, thereby reducing the bit volume.

[0080] In the 12th embodiment, the encoding apparatus according to the 17th embodiment of this disclosure may be such that the circuit encodes image association information associated with an image contained in the bitstream into the second video availability information; and, in the case where second extended information further extending the second video availability information is described in the second video availability information, size information representing the size of the second extended information is encoded into the second video availability information, and the second extended information is encoded into the second video availability information based on the size information.

[0081] According to the 17th method, the second video availability information can be further extended using the second extended information. Furthermore, by using size information, the decoding device can appropriately obtain the second extended information from the second video availability information.

[0082] In the 12th embodiment, the encoding apparatus according to the 18th embodiment of this disclosure may include the circuit encoding image association information associated with an image contained in the bitstream into the second video availability information, encoding a third extension flag information indicating whether a second extension information further extending the second video availability information is recorded in the second video availability information, and encoding the second extension information into the second video availability information if the third extension flag information is true.

[0083] According to method 18, the second video availability information can be further extended through the third extended flag information and the second extended information.

[0084] In the 12th embodiment, the encoding device involved in the 19th embodiment of this disclosure may be a bit string in which the first extended information has a continuous bit string extending to the end bit of the NAL unit storing the first extended information.

[0085] According to the 19th method, the first extended information can be easily and appropriately encoded in the NAL unit where the first extended information is stored.

[0086] In the 18th embodiment, the encoding device involved in the 20th embodiment of this disclosure may be a bit string in which the second extended information has a continuous bit string extending to the end of the NAL unit where the second extended information is stored.

[0087] According to the 20th method, the second extended information can be easily and appropriately encoded in the NAL unit where the second extended information is stored.

[0088] In the 12th embodiment of the encoding apparatus according to the 21st embodiment of this disclosure, the second video availability information may include image association information associated with an image contained in the bitstream. In the encoding apparatus where the image association information cannot be interpreted, the circuit encodes the first extended flag information, which is false, in the bitstream, and does not encode the second video availability information, the second extended flag information, and the first extended information in the bitstream.

[0089] According to the 21st method, by not encoding the second video availability information, the second extended flag information, and the first extended information, it is possible to avoid the encoding device performing useless processing.

[0090] In the 12th aspect, the encoding apparatus of the 22nd aspect of this disclosure may include, in the case of the second video availability information comprising image association information associated with an image contained in the bitstream, the circuit in the encoding apparatus capable of interpreting the image association information but unable to interpret the first extended information encodes the first extended flag information which is true, the second extended flag information which is false, and the second video availability information in the bitstream, but does not encode the first extended information in the bitstream.

[0091] According to the 22nd method, by not encoding the first extended information, it is possible to avoid the encoding device performing useless processing.

[0092] In the decoding method according to the 23rd aspect of this disclosure, the decoding device obtains a first presence flag from the bitstream indicating whether the first video availability information is recorded in the bitstream. If the first presence flag is true, the device obtains the first video availability information from the bitstream. It also obtains a first extension flag from the bitstream indicating whether the second video availability information is recorded in the bitstream. If the first extension flag is true, the device obtains the second video availability information from the bitstream. Finally, it obtains a second extension flag from the bitstream indicating whether the first extension information is recorded in the bitstream. The first extension information extends other information of the NAL (Network Abstraction Layer) cell that records the second video availability information. If the second extension flag is true, the device obtains the first extension information from the bitstream.

[0093] According to the 23rd method, the video availability information can be appropriately extended using the first extended flag information and the second video availability information. Furthermore, other information of the NAL unit describing the second video availability information can be appropriately extended using the second extended flag information and the first extended information.

[0094] In the encoding method according to the 24th aspect of this disclosure, the encoding device encodes a first presence flag indicating whether first video availability information is recorded in the bitstream. If the first presence flag is true, the first video availability information is encoded in the bitstream. The device also encodes a first extension flag indicating whether second video availability information is recorded in the bitstream. If the first extension flag is true, the second video availability information is encoded in the bitstream. Furthermore, the device encodes a second extension flag indicating whether the first extension information is recorded in the bitstream. The first extension information extends other information of the NAL (Network Abstraction Layer) cell that records the second video availability information. If the second extension flag is true, the first extension information is encoded in the bitstream.

[0095] According to the 24th method, the video availability information can be appropriately extended using the first extended flag information and the second video availability information. Furthermore, other information of the NAL unit describing the second video availability information can be appropriately extended using the second extended flag information and the first extended information.

[0096] (Implementation of this disclosure)

[0097] Hereinafter, embodiments of the present disclosure will be described in detail using the accompanying drawings. Furthermore, elements labeled with the same reference numerals in different drawings are assumed to represent the same or corresponding elements.

[0098] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, constituent elements in the following embodiments that are not described in the independent claims representing the highest-level concept are described as arbitrary constituent elements. Furthermore, the contents can be substituted or combined throughout the embodiments. Additionally, these general or specific methods can be implemented by systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, or recording media.

[0099] Figure 1 This is a simplified diagram illustrating the structure of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, and a transmission path NW.

[0100] Image data D1 is input from an external device to the encoding device 1. The external device includes a camera that captures moving images, etc. The external device inputs the image data D1 of the captured moving images into the encoding device 1.

[0101] Encoding device 1 generates a bitstream BS based on image data D1. Here, a bitstream refers to a data string or stream of digital data. A bitstream (or simply a stream) can be a single stream or consist of multiple streams divided into multiple layers. Furthermore, a bitstream can be transmitted via a single transmission path through serial communication or via multiple transmission paths through packet communication. Encoding device 1 sends the generated bitstream BS to decoding device 2 via transmission path NW. Decoding device 2 receives the bitstream BS.

[0102] Decoding device 2 decodes image data D1 based on bitstream BS, and performs task processing based on the decoded image data D1. Task processing includes human vision and machine tasks. Human vision refers to the visual or auditory confirmation of moving images by operators or users. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using artificial intelligence (AI) models that are inference models completed through machine learning. The task processing unit performing human vision includes display devices such as liquid crystal displays or organic EL displays. The task processing unit performing machine tasks includes an inferrer that uses AI.

[0103] The transmission path NW can be the Internet, a WAN (wide area network), a LAN (local area network), or any combination thereof. The preferred transmission path NW is a private network, where secure communication is ensured through access restrictions.

[0104] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 is configured to include a processor such as a CPU. The memory 12 is configured to include any recording medium such as ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data processed by the circuit 11 or data processed during the process.

[0105] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 is configured to include a processor such as a CPU. The memory 22 is configured to include any recording medium such as ROM, RAM, HDD, SSD, or semiconductor memory. The memory 22 stores data processed by the circuit 21 or data processed during the process.

[0106] Figure 2 This diagram is a simplified representation of the structure of the circuit 11 included in the encoding device 1. The circuit 11 includes an acquisition unit 31, a setting unit 32, an encoding unit 33, and a transmission unit 34.

[0107] Next, the coding unit 33 involved in this embodiment will be described. Figure 21 This is a block diagram illustrating an example of the functional structure of the encoding unit 33 according to this embodiment. The encoding unit 33 encodes the image in units of blocks.

[0108] like Figure 21 As shown, the encoding unit 33 includes a segmentation unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Furthermore, the intra-frame prediction unit 124 and the inter-frame prediction unit 126 are configured as part of the prediction processing unit 125.

[0109] For example, Figure 21 The multiple components of the coding unit 33 shown are obtained through Figure 1 The circuit 11 and memory 12 shown are installed.

[0110] Circuit 11 is configured to include a processor such as a CPU. Circuit 11 can be a dedicated or general-purpose electronic circuit for encoding images, or it can be an assembly of multiple electronic circuits. Furthermore, for example, circuit 11 can also function as... Figure 21 The coding unit 33 shown has multiple components, including the components other than those used for storing information.

[0111] The memory 12 can be a dedicated or general-purpose electronic circuit for storing information, or it can be a collection of multiple electronic circuits. The memory 12 can be externally connected to the circuit 11, or it can be built into the circuit 11. Furthermore, the memory 12 can be a disk or optical disk, or it can be a storage device or recording medium. Additionally, the memory 12 can be non-volatile memory or volatile memory.

[0112] The memory 12 may store the encoded image or the stream corresponding to the encoded image. Furthermore, the memory 12 may also store a program for the processor to perform the image encoding process.

[0113] In addition, memory 12 can also play a role Figure 21 The encoding unit 33 shown has multiple components, among which the component for storing information functions. Specifically, the memory 12 can also function as... Figure 21 The block memory 118 and frame memory 122 are shown to have the following functions. More specifically, the reconstructed image (specifically, reconstructed blocks or reconstructed images, etc.) can also be stored in memory 12.

[0114] Additionally, in the encoding section 33, Figure 21 The installation of some of the multiple components shown can be omitted, as can the execution of some of the multiple processes performed by those components. Alternatively, Figure 21 A portion of the multiple components shown may also be mounted on other devices, and a portion of the multiple processes performed by the multiple components may also be performed by other devices.

[0115] Figure 3 This is a flowchart showing the process performed by the circuit 11 of the encoding device 1.

[0116] First, in step SP11, the acquisition unit 31 acquires image data D11 representing the image Q, which is the processing object input from an external device. Image data D11 corresponds to... Figure 1 The image data D1 shown is shown.

[0117] Next, in step SP12, the setting unit 32 sets parameter P in association with the image Q. Parameter P includes a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 contains modal information indicating the image category of image Q. The second parameter P2 contains wavelength information indicating the band of light radiation of image Q. The third parameter P3 contains color information of image Q. The fourth parameter P4 contains extended bit information corresponding to the prepared image category if the modal information indicates that the image category is an extended prepared image category. The setting unit 32 inputs data D12 containing the setting information of parameter P to the encoding unit 33.

[0118] The first parameter P1 contains modal information representing the image category of image Q. The image category may include at least one of visible light images, infrared images, and ultraviolet images. Visible light images include natural images or RGB images, used in providing detailed color information for human vision or machine tasks. Infrared images include images captured using an infrared camera, used in shooting in the dark. Ultraviolet images include images captured using an ultraviolet camera, used in detecting product damage. The modal information indicates that the image category is at least one of visible light images, infrared images, and ultraviolet images. Furthermore, the modal information may indicate at least one of the following: the image category is undefined, or the image category is a pre-defined image category for expansion. In other words, the modal information indicates that the image category is at least one of visible light images, infrared images, ultraviolet images, undefined, or a pre-defined image category for expansion. Additionally, the modal information may also include other image categories. The setting unit 32 can set the first parameter P1 based on image analysis based on image data D11, or it can set the first parameter P1 based on setting information input by the operator of the encoding device 1.

[0119] Next, in step SP13, the encoding unit 33 encodes the image Q represented by the image data D11 input from the acquisition unit 31 into the bit stream BS.

[0120] Next, in step SP14, the encoding unit 33 encodes the parameter P, represented by the data D12 input from the setting unit 32, into the bitstream BS. Encoding the parameter P into the bitstream BS can also be interpreted as storing the parameter P in the bitstream BS, or simply storing the parameter P in the bitstream BS. Furthermore, the execution order of steps SP13 and SP14 can also be... Figure 3 Conversely, the opposite can also be true.

[0121] Next, in step SP15, the transmitting unit 34 transmits the bit stream BS input from the encoding unit 33 to the decoding device 2 via the transmission path NW.

[0122] Figure 4 This is a simplified diagram illustrating the structure of the bitstream BS. The bitstream BS has a header region 41 and a payload region 42. The encoding unit 33 stores the encoded data of the image Q in the payload region 42 and stores the encoded data of the parameter P associated with the image Q in the header region 41.

[0123] The encoding unit 33 can also encode the encoded data of parameter P in a given region 43 within the header region 41. The given region 43 can also be VUI (video usability information) or SEI (Supplemental Enhancement Information). However, the given region 43 is not limited to VUI or SEI, and can also be VPS, SPS, PPS, PH, SH, APS, tile header, or system layer header, etc.

[0124] Figure 22 This is a diagram illustrating an example of the hierarchical structure of data in a stream. A stream, for example, contains video sequences. Video sequences are as follows... Figure 22 As shown in (A), it includes, for example, VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information), and multiple pictures.

[0125] In a dynamic image composed of multiple layers, a VPS includes encoding parameters shared by multiple layers, as well as encoding parameters associated with the multiple layers contained in the dynamic image or each layer.

[0126] The SPS contains parameters used for the sequence, that is, encoding parameters referenced by the decoding device 2 in order to decode the sequence. These encoding parameters may, for example, represent the width or height of the image. Furthermore, multiple SPSs may exist.

[0127] The PPS contains parameters used for the images; that is, the encoding parameters that the decoding device 2 refers to when decoding each image in the sequence. These encoding parameters may, for example, include a reference value for the quantization width used for image decoding and a flag indicating the application of weighted prediction. Furthermore, multiple PPSs may exist. SPS and PPS are sometimes simply referred to as parameter sets.

[0128] like Figure 22As shown in (B), the image includes an image header and one or more slices. The image header contains encoding parameters referenced by the decoding device 2 for decoding the one or more slices.

[0129] like Figure 22 As shown in (C), the stripe includes a stripe header and one or more bricks. The stripe header contains encoding parameters referenced by the decoding device 2 for decoding the one or more bricks.

[0130] like Figure 22 As shown in (D), the brick contains more than one CTU (Coding Tree Unit).

[0131] Alternatively, an image may not contain stripes, but instead contain groups of pieces. In this case, the group of pieces contains more than one piece. Furthermore, stripes can also be included within bricks.

[0132] CTU is also called a superblock or basic unit of partitioning. For example... Figure 22 As shown in (E), the CTU includes a CTU header and one or more CUs (Coding Units). The CTU header contains encoding parameters referenced by the decoding device 2 for decoding the one or more CUs.

[0133] A CU can also be divided into multiple smaller CUs. Furthermore, as... Figure 22 As shown in (F), the CU contains a CU header, prediction information, and residual coefficient information. Prediction information is used to predict the CU. Residual coefficient information represents the prediction residuals. Furthermore, the CU is essentially the same as a PU (Prediction Unit) or TU (Transform Unit), but can also contain multiple TUs smaller than the CU. Additionally, the CU can be processed into each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is, for example, a fixed unit that can be processed in one stage during pipelined processing in hardware.

[0134] Alternatively, a stream may not have... Figure 22 This is a subset of the multiple layers shown. Furthermore, the order of these layers can be interchanged, or any layer can be replaced with another layer.

[0135] The image that is the object of the processing performed by the encoding device 1 or decoding device 2 at the current point in time is called the current image. If the processing is encoding, the current image is synonymous with the image to be encoded; if the processing is decoding, the current image is synonymous with the image to be decoded. Furthermore, the block (CU or a block of the CU) that is the object of the processing performed by the encoding device 1 or decoding device 2 at the current point in time is called the current block. If the processing is encoding, the current block is synonymous with the block to be encoded; if the processing is decoding, the current block is synonymous with the block to be decoded.

[0136] Figure 5 This is a simplified diagram illustrating an example of the syntax for VUI parameters that include the first parameter P1 and the third parameter P3.

[0137] The first parameter, P1, contains a flag (vui_modality_info_present_flag) indicating whether modal information is recorded in the bitstream BS. A value of 1 for vui_modality_info_present_flag indicates the presence of modal information within the VUI parameter; a value of 0 indicates the absence of modal information within the VUI parameter. When vui_modality_info_present_flag indicates that modal information is recorded in the bitstream BS, the modal information is represented by the value of the identifier vui_modality_type contained in the VUI parameter.

[0138] Figure 6 This is an example diagram showing the correspondence between the values ​​of `vui_modality_type` and image categories. For example... Figure 6 As shown, a value of 0 for `vui_modality_type` indicates that the image category of image Q is undefined (i.e., undefined). A value of 1 for `vui_modality_type` indicates that the image category of image Q is a visible light image. A value of 2 for `vui_modality_type` indicates that the image category of image Q is an infrared image. A value of 3 for `vui_modality_type` indicates that the image category of image Q is an ultraviolet image. Other values ​​for `vui_modality_type` (4-255 in this example) represent reserved areas ensured for future extended use. Additionally, the value can be increased based on the image Q being processed. Figure 6The number of image categories defined in the code. Furthermore, the absence of `vui_modality_type` can also mean that the value of `vui_modality_type` is 0. Additionally, with `vui_modality_type` set to 0, wavelength information representing a wide band across multiple image categories, such as hyperspectral images, can be defined based on user settings, etc.

[0139] like Figure 5 As shown, parameter P3 contains a flag indicating whether color information is described in the bitstream BS (vui_colour_description_present_flag). A value of 1 for vui_colour_description_present_flag indicates the presence of color information within the VUI parameter; a value of 0 indicates the absence of color information within the VUI parameter. Although the illustration is omitted, the color information includes vui_colour_primaries, vui_trasfer_characteristics, and vui_matrix_coeffs. Furthermore, in the case of infrared or ultraviolet images, parameter P3 may not contain color information. In the encoding process, encoding device 1 sets the value of `vui_colour_description_present_flag` to 1 and records color information in the VUI parameters when the image category represented by the modal information is a visible light image. On the other hand, when the image category represented by the modal information is an infrared or ultraviolet image, it sets the value of `vui_colour_description_present_flag` to 0 and does not record color information in the VUI parameters. Similarly, in the decoding process, decoding device 2 obtains color information from the bitstream BS when the image category represented by the modal information is a visible light image. On the other hand, when the image category represented by the modal information is an infrared or ultraviolet image, it may not obtain color information from the bitstream BS, or it may ignore the value even if color information is obtained from the bitstream BS.

[0140] Figure 7 This is a simplified diagram illustrating an example of the syntax for VUI parameters that include the fourth parameter, P4.

[0141] The fourth parameter, P4, contains extension bit information corresponding to the prepared image category when the modal information indicates that the image category is an extension. The extension bit information includes information indicating the bit length of the prepared image category (vui_modality_type_extension_bits) and information indicating the content of the prepared image category (vui_reserved_modality_type_extension). A value of 0 for vui_modality_type_extension_bits indicates that vui_reserved_modality_type_extension is not recorded in the bitstream BS. Conversely, a value greater than 0 for vui_modality_type_extension_bits indicates that vui_reserved_modality_type_extension and its bit length are recorded in the bitstream BS. The maximum value of vui_modality_type_extension_bits can be arbitrarily set, for example, to 2048.

[0142] Figure 8 This is a simplified illustration of the first example of the syntax for VUI parameters, which includes the second parameter P2. The second parameter P2 contains wavelength information representing the band of light radiation from the image Q.

[0143] exist Figure 8 In the example shown, the second parameter P2 contains a flag (vui_spectrum_range_present_flag) indicating whether wavelength information is recorded in the bitstream BS. A value of 0 for vui_spectrum_range_present_flag indicates that the spectral band of the light radiation wavelength represented by the image of the CLVS (Coded Layer Video Sequence) does not exist within the VUI parameter. A value of 1 for vui_spectrum_range_present_flag indicates that the spectral band of the light radiation wavelength represented by the image of the CLVS exists within the VUI parameter.

[0144] When vui_spectrum_range_present_flag is set to 1, the wavelength information includes the minimum and maximum wavelengths (minimum wavelengths) representing the bands of image Q. Both the minimum and maximum wavelengths are measured in meters.

[0145] The minimum value of a band includes information representing the mantissa of the minimum value in an exponential representation using a given base (e.g., 10), and information representing the value obtained by adding 15 to the exponent of the minimum value (vui_min_wavelength_mantissa), and information representing the value obtained by adding 15 to the exponent of the minimum value (vui_min_wavelength_exponent_plus15). Here, if vui_min_wavelength_mantissa is 0 or does not exist, it means that it is ambiguous or uncertain and the minimum value is undefined, or defined by a method other than exponential representation. In this case, during the decoding process, the decoding device 2 may choose not to obtain vui_min_wavelength_exponent_plus15 from the bitstream BS, or it may ignore its value even if it obtains vui_min_wavelength_exponent_plus15 from the bitstream BS.

[0146] The information representing the maximum value of the band includes information about the mantissa of the maximum value (vui_max_wavelength_mantissa) in the exponential representation using a given base (e.g., 10), and information about the value obtained by adding 15 to the exponent of the maximum value (vui_max_wavelength_exponent_plus15). Here, if vui_max_wavelength_mantissa is 0 or does not exist, it means that the maximum value of the band is ambiguous or uncertain and is undefined, or defined by a method other than exponential representation. In this case, the decoding device 2 can obtain vui_max_wavelength_exponent_plus15 from the non-bitstream BS, or it can ignore its value even if it obtains vui_max_wavelength_exponent_plus15 from the bitstream BS.

[0147] Both `vui_min_wavelength_mantissa` and `vui_max_wavelength_mantissa` are decimal numbers with at least three significant digits. Figure 8 In the example shown, both `vui_min_wavelength_mantissa` and `vui_max_wavelength_mantissa` are binary numbers with a bit length of 10 bits, but they can also have any bit length of 11 bits or more. Similarly, in... Figure 8In the example shown, both vui_min_wavelength_exponent_plus15 and vui_max_wavelength_exponent_plus15 are binary numbers with a bit length of 5 bits, but they can also have any bit length of 6 bits or more.

[0148] Furthermore, in encoding device 1, the minimum value of the band represented by vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15 is set to be greater than or equal to the shortest wavelength corresponding to the image category represented by the modal information. Similarly, in encoding device 1, the maximum value of the band represented by vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15 is set to be less than or equal to the longest wavelength corresponding to the image category represented by the modal information.

[0149] Figure 9 This is a simplified diagram illustrating the second example of the syntax for VUI parameters that include the second parameter P2.

[0150] exist Figure 9 In the example shown, the second parameter P2 contains information about the index value representing the wavelength information (vui_spectrum_range).

[0151] Figure 10 This diagram illustrates an example of the correspondence between index values ​​and wavelength information. Minimum (minimum wavelength) and maximum (maximum wavelength) values ​​for each band are predefined and shared by encoding device 1 and decoding device 2, each corresponding to a specific index value. Figure 10 In the example shown, index values ​​2-6 correspond to the wavelength region of ultraviolet light, index value 7 corresponds to the wavelength region of visible light, and index values ​​8-10 correspond to the wavelength region of infrared light.

[0152] When the index value is, for example, 2, the minimum wavelength is defined as 1 * 10. -9 (m), the maximum wavelength is defined as 1*10 -7 (m), in the case of an index value of, for example, 3, the minimum wavelength is defined as 1 * 10m -7 (m), the maximum wavelength is defined as 19*10 -8 (m).

[0153] Furthermore, when the index value is a specific value (in) Figure 10 In the example shown, a value of 0 indicates that the minimum and maximum wavelengths are not defined.

[0154] Furthermore, the index value is another specific value (in Figure 10 In the example shown, case 1) means that the desired minimum and maximum wavelengths are defined according to user settings.

[0155] Additionally, the index values ​​for other values ​​(11-255 in this example) represent areas reserved for future extended use.

[0156] Furthermore, in encoding device 1, the index value is set according to the band corresponding to the image category represented by the modal information. For example, when vui_modality_type is 1 (visible light image), vui_spectrum_range is set to any one of 0, 1, or 7. Furthermore, when vui_modality_type is 2 (infrared image), vui_spectrum_range is set to any one of 0, 1, or 8 to 10. Furthermore, when vui_modality_type is 3 (ultraviolet image), vui_spectrum_range is set to any one of 0, 1, or 2 to 6.

[0157] Figure 11 This is a simplified diagram illustrating the third example of the syntax for VUI parameters that include the second parameter P2.

[0158] exist Figure 9 In the example shown, when the index value of vui_spectrum_range is set to 1 (user-defined), compared to... Figure 8 Similarly, the minimum value of the band can be defined using vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15, and the maximum value of the band can be defined using vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15.

[0159] Figure 12 This is a simplified diagram illustrating the first variation of the syntax for VUI parameters.

[0160] When the image category represented by the modal information is an infrared image, the encoding device 1 may further encode the representation type information (vui_infrared_radiance_representation_type) representing the representation type of infrared radiation intensity in the bit stream BS.

[0161] Figure 13This is a diagram illustrating an example of the correspondence between the values ​​of the representation type information and the interpretation of each value.

[0162] If vui_infrared_radiance_representation_type is 0 or does not exist, it means that the representation type of infrared radiation intensity is unclear or uncertain and the representation type is not defined, or it is defined by a method other than intensity value representation or temperature value representation.

[0163] When `vui_infrared_radiance_representation_type` is set to 1, it means that the infrared radiation intensity is represented by an intensity value. In the intensity value representation, each pixel value in image Q represents the intensity value of the infrared radiation.

[0164] When the value of `vui_infrared_radiance_representation_type` is 2, it means that the infrared radiation intensity is represented as a temperature value. In the temperature value representation, each pixel value of image Q represents a temperature value calculated based on the intensity value of the infrared radiation.

[0165] Other values ​​(3 in this example) of `vui_infrared_radiance_representation_type` indicate the prepared area to be used for future expansion.

[0166] In addition, such as Figure 12 As shown, when the value of vui_infrared_radiance_representation_type is 2 (that is, when it is a temperature value representation), the encoding device 1 records the temperature information in the VUI parameters. The temperature information includes the lowest and highest temperatures as floating decimal values.

[0167] The information representing the minimum temperature includes information in an exponential representation using a given base (e.g., 2) indicating the sign of the minimum temperature (vui_min_temperature_sign), the exponent of the minimum temperature (vui_min_temperature_exponent), the value obtained by subtracting 1 from the bit length of the minimum temperature's mantissa (vui_min_temperature_mantissa_len_minus1), and the mantissa of the minimum temperature (vui_min_temperature_mantissa). Alternatively, vui_min_temperature_mantissa_len_minus1 can be omitted by setting the bit length of the minimum temperature's mantissa to a fixed value.

[0168] When the value of vui_min_temperature_sign is 0, the sign of the lowest temperature is positive; when the value of vui_min_temperature_sign is 1, the sign of the lowest temperature is negative.

[0169] The information representing the maximum temperature includes: the sign of the maximum temperature (vui_max_temperature_sign) in an exponential representation using a given base (e.g., 2); the exponent of the maximum temperature (vui_max_temperature_exponent); the value obtained by subtracting 1 from the bit length of the mantissa of the maximum temperature (vui_max_temperature_mantissa_len_minus1); and the mantissa of the maximum temperature (vui_max_temperature_mantissa). Alternatively, the bit length of the mantissa of the maximum temperature can be set to a fixed value, and vui_max_temperature_mantissa_len_minus1 can be omitted.

[0170] When the value of vui_max_temperature_sign is 0, the sign of the highest temperature is positive; when the value of vui_max_temperature_sign is 1, the sign of the highest temperature is negative.

[0171] Figure 14 This is a simplified diagram illustrating the second variation of the syntax for VUI parameters.

[0172] The encoding device 1 may further encode the flag information (vui_false_colour_representation_present_flag) indicating whether the pixel values ​​of the image Q are represented using pseudo-color representation in the bitstream BS.

[0173] When vui_false_colour_representation_present_flag is 1, it means that the pixel values ​​of image Q are represented using pseudo-color. When vui_false_colour_representation_present_flag is 0, it means that the pixel values ​​of image Q are not represented using pseudo-color.

[0174] When the pixel values ​​of image Q are represented using pseudo-color, the generation information for generating a table that corresponds to the pixel values ​​of image Q and the color codes of the pseudo-color representation is recorded in the VUI parameters.

[0175] The generated information includes information representing the value obtained by subtracting 2 from the number of color codes encoded in the bitstream BS (vui_num_hex_colour_code_minus2), information representing the value obtained by subtracting 2 from the number of groups of pixel values ​​and color codes (vui_num_value_minus2), interpolation information used to interpolate the values ​​between these groups (vui_colour_interpolation_type), and information representing the hexadecimal color code of the i-th color (vui_hex_colour_code[i]). Each hexadecimal color code is 24 bits long, with the first 8 bits representing the intensity of red, the middle 8 bits representing the intensity of green, and the last 8 bits representing the intensity of blue. For example, vui_hex_colour_code 000000 represents black (RGB: 0, 0, 0), and vui_hex_colour_code FFFFFF represents white (RGB: 255, 255, 255).

[0176] If the value of vui_num_hex_colour_code_minus2 is 0, it means that the number of color codes encoded in the bitstream BS is 2.

[0177] If the value of vui_num_value_minus2 is, for example, 4, it means that there are 6 groups for pixel values ​​and color codes. Six groups mean that the range of color components can be divided into 5 equal intervals.

[0178] Figure 15 This is a diagram illustrating an example of the correspondence between the values ​​of the interpolation information and their interpretations.

[0179] If vui_colour_interpolation_type is 0 or does not exist, it means that the interpolation type is ambiguous or uncertain and the interpolation type is not defined, or it is defined by a method other than linear interpolation or bilinear interpolation.

[0180] When vui_colour_interpolation_type is 1, it means that the interpolation type is linear interpolation.

[0181] When vui_colour_interpolation_type is 2, it means that the interpolation type is bilinear interpolation.

[0182] Other values ​​for vui_colour_interpolation_type (3-15 in this example) indicate the prepared regions to be ensured for future extended use.

[0183] Figure 16 This is a diagram illustrating an example of the structure of a bitstream (BS).

[0184] Bitstream BS can also have multiple m-layer image layers L1 to L1 (where m is a natural number greater than 2). m A multi-layered structure. In Figure 16 In the diagram, only one access unit is shown. An access unit is the smallest unit of processing a temporal attribute, such as equivalent to one frame of a moving image. Furthermore, an access unit can be, for example, a collection of multiple NAL units equivalent to one frame of a moving image. A bitstream (BS) is constructed by containing multiple temporally consecutive access units.

[0185] In the payload area 42 of the image layer L1, which is the lowest layer, the image Q, which serves as the main image, is stored. L1 The encoded data. In the payload area 42 of the second image layer L2, the image Q, serving as an auxiliary image, is stored. L2 The encoded data. Similarly, in the image layer L at the m-th layer. m The effective payload region 42 stores the image Q, which serves as an auxiliary image. Lm Encoded data.

[0186] Image Q L1 ~Q Lm They can also have different image categories or bands. For example, in a multi-layered structure with m=3 and 3 layers, the image Q could also be... L1 For visible light images, image Q L2 This is an infrared image, image QL3 The image is an ultraviolet image. The encoding device 1 generates a bitstream BS with a multi-layer structure by encoding, for example, the output signal from a multispectral camera.

[0187] In image Q L1 ~Q Lm When there is a correlation between them, it can also be done in image layers L1 to L2. m Interstitial reference image. On the other hand, in image Q L1 ~Q Lm If there is no correlation between them, it is also possible not to use image layers L1 to L2. m Refer to the image in between.

[0188] In the header region 41 of image layer L1, the data related to image Q is stored. L1 The associated parameter P was established. L1 The encoded data. Parameter P L1 Includes image Q L1 Modal information and wavelength information, etc.

[0189] In the header region 41 of image layer L2, the data related to image Q is stored. L2 The associated parameter P was established. L2 The encoded data. Parameter P L2 Includes image Q L2 Modal information and wavelength information, etc.

[0190] In image layer L m The header area 41 stores the image Q. Lm The associated parameter P was established. Lm The encoded data. Parameter P Lm Includes image Q Lm Modal information and wavelength information, etc.

[0191] Figure 17 This diagram is a simplified representation of the structure of the circuit 21 included in the decoding device 2. The circuit 21 includes a receiving unit 51, a decoding unit 52, a switching unit 53, and multiple n (n being a natural number of 2 or more) task processing units 541 to 542. n Task Processing Department 541-54 n The tasks performed include both human vision and machine tasks. Machine tasks include a wide variety of tasks that utilize AI models, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0192] Next, the decoding unit 52 according to this embodiment will be described. Figure 23 This is a block diagram illustrating an example of the functional structure of the decoding unit 52 according to this embodiment. The decoding unit 52 decodes the stream of encoded images in blocks.

[0193] like Figure 23 As shown, the decoding unit 52 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter 212, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a segmentation determination unit 224. Furthermore, the intra-frame prediction unit 216 and the inter-frame prediction unit 218 are configured as part of the prediction processing unit 215.

[0194] For example, Figure 23 The decoding unit 52 shown has multiple components that are transmitted through Figure 1 The circuit 21 and memory 22 shown are installed.

[0195] Circuit 21 is configured to include a processor such as a CPU. Circuit 21 can be a dedicated or general-purpose electronic circuit for decoding streams, or it can be a collection of multiple electronic circuits. Furthermore, for example, circuit 21 can also function as… Figure 23 The decoding unit 52 shown has multiple components, including the components other than those used for storing information.

[0196] The memory 22 can be a dedicated or general-purpose electronic circuit for storing information, or it can be a collection of multiple electronic circuits. The memory 22 can be externally connected to the circuit 21, or it can be built into the circuit 21. Furthermore, the memory 22 can be a disk or optical disk, or it can be a storage device or recording medium. Additionally, the memory 22 can be non-volatile or volatile memory.

[0197] The memory 22 can store the decoded stream or the decoded image. Furthermore, the memory 22 can also store a program for the processor to perform the decoding process on the stream.

[0198] In addition, memory 22 can also play a role Figure 23 The decoding unit 52 shown has multiple components, among which the component for storing information functions. Specifically, the memory 22 can also function as... Figure 23 The block memory 210 and frame memory 214 are shown to have specific functions. More specifically, the memory 22 can also store reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.).

[0199] In addition, in the decoding unit 52, Figure 23 The installation of some of the multiple components shown can be omitted, as can the execution of some of the multiple processes performed by those components. Alternatively, Figure 23A portion of the multiple components shown may also be mounted on other devices, and a portion of the multiple processes performed by the multiple components may also be performed by other devices.

[0200] Figure 23 The decoding unit 52 shown includes an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, a prediction control unit 220, and a loop filter 212, which are respectively connected to... Figure 21 The encoding unit 33 shown includes the same inverse quantization unit 112, inverse transform unit 114, summation unit 116, block memory 118, frame memory 122, intra-frame prediction unit 124, inter-frame prediction unit 126, prediction control unit 128, and loop filter 120.

[0201] Figure 18 This is a flowchart showing the process performed by the circuit 21 of the decoding device 2.

[0202] First, in step SP21, the receiving unit 51 receives the bit stream BS sent by the encoding device 1 from the transmission path NW.

[0203] Next, in step SP22, the decoding unit 52 acquires the image Q by decoding the payload area 42 of the bitstream BS input from the receiving unit 51. Additionally, extraction may be included in the decoding process. The decoding unit 52 outputs image data D21 of the image Q. Image data D21 is equivalent to... Figure 2 The image data shown is D11.

[0204] like Figure 16 As shown, the bitstream BS has multiple image layers L1 to L2. m In the case of a multi-layer structure, the decoding unit 52 can also decode from multiple image layers L1 to L2. m Acquire multiple images of different image categories or bands Q L1 ~Q Lm .

[0205] Next, in step SP23, the decoding unit 52 obtains parameter P by decoding the header region 41 (or a given region 43 within the header region 41) of the bit stream BS input from the receiving unit 51. As described above, parameter P includes a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 contains modal information indicating the image category of image Q. The second parameter P2 contains wavelength information indicating the band of light radiation of image Q. The third parameter P3 contains color information of image Q. The fourth parameter P4 contains extended bit information corresponding to the prepared image category when the modal information indicates that the image category is a prepared image category for expansion. Furthermore, the execution order of steps SP22 and SP23 can also be... Figure 18 Conversely, the opposite can also be true.

[0206] exist Figure 12 When the VUI parameters shown are included in parameter P, and the modal information indicates that the image category is an infrared image (that is, when the value of vui_modality_type is 2), the decoding unit 52 further obtains the aforementioned performance type information indicating the performance type of infrared radiation intensity from the bitstream BS. Furthermore, when the performance type information indicates that the pixel value represents a temperature value (that is, when it represents a temperature value), the decoding unit 52 further obtains temperature information including the lowest and highest temperatures from the bitstream BS.

[0207] exist Figure 14 When the VUI parameters shown are included in parameter P, and the flag information indicates that pixel values ​​are represented using pseudo-color representation (that is, when the value of vui_false_colour_representation_present_flag is 1), the decoding unit 52 further obtains from the bitstream BS the aforementioned generation information used to generate a table that establishes a correspondence between the pixel values ​​of image Q and the color codes of the pseudo-color representation. Furthermore, the decoding unit 52 generates the table information based on the obtained generation information.

[0208] Figure 19 , 20 This is a simplified illustration of an example of table information.

[0209] exist Figure 19 , 20 In the example shown, there are 6 groups of pixel values ​​and color codes (meaning the value of vui_num_value_minus2 is 4), and the image category is infrared image. The interpolation type can be linear interpolation or bilinear interpolation.

[0210] exist Figure 19In the example shown, the table information represents the correspondence between hexadecimal color codes (Hex color codes), RGB color codes, and gray density.

[0211] exist Figure 20 In the example shown, the table information represents the correspondence between Hex color codes, gray intensity, and temperature, with gray intensity mapped to temperature. For example, black represents a temperature of 300K, gray represents a temperature of 330K, and white represents a temperature of 350K.

[0212] Next, in step SP24, the switching unit 53, based on the modal information input from the decoding unit 52, adjusts the settings of the task processing units 541 to 542. n Switching is performed. Switching unit 53 maintains preset multiple image categories and multiple task processing units 541-54. n The table information showing the correspondence between the tasks is omitted. The switching unit 53 refers to this table information to switch between multiple task processing units 541-54. n Select a task processing unit 541-54 corresponding to the image category represented by modal information. n .

[0213] Reference Figure 18 Next, in step SP25, one of the task processing units 541 to 54 selected in step SP24... n Task processing is performed based on the image data D21 input from the decoding unit 52 via the switching unit 53.

[0214] According to the encoding device 1 and decoding device 2 of this embodiment, modal information representing the image category of image Q and wavelength information representing the band of light radiation of image Q can be included in the bit stream BS and transmitted from the encoding device 1 to the decoding device 2, thereby improving the execution accuracy of the decoding device 2 in task processing.

[0215] Furthermore, according to the encoding apparatus 1 and decoding apparatus 2 of this embodiment, the modal information indicates that the image category is at least one of the following: visible light image, infrared image, ultraviolet image, undefined, and a prepared image category for expansion. Therefore, since the modal information includes image category information indicating that the image category is undefined or that the image category is a prepared image category for expansion, it is also possible to address situations where the image characteristics in the encoding apparatus 1 are unclear and the image category cannot be determined, or future expansion of the image category.

[0216] (The development form of this disclosure)

[0217] The encoding device 1 and decoding device 2 described in the above embodiments are devices corresponding to VVC (H.266), which is a next-generation video coding standard. In VVC, it has been studied to extend and include image-related information such as modal information in VUI parameters. However, in existing video coding standards such as AVC (H.264) or HEVC (H.265), VUI parameters cannot be directly extended. Therefore, when encoding device 1 and decoding device 2 are devices corresponding to AVC or HEVC, image-related information such as modal information extended to VUI parameters cannot be properly processed. Therefore, in this development, by appropriately layering the syntax of SPS (Sequence Parameter Set) using extended flag information, image-related information such as modal information can be properly processed even if encoding device 1 and decoding device 2 cannot directly extend VUI parameters. Furthermore, it is not limited to SPS; it can also be VUI, VPS, SPS, PPS, PH, SH, APS, SEI, slice header, or system layer header, etc.

[0218] and Figure 1 Similarly, the encoding device 1 of this development includes a circuit 11 and a memory 12 connected to the circuit 11. The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21.

[0219] Figure 24 This is a simplified illustration of the first example of the syntax for SPS in HEVC.

[0220] The NAL (Network Abstraction Layer) unit of SPS contains the first presence flag (vui_parameters_present_flag), the first video availability flag (vui_parameters()), the first extension flag (sps_vui_extension_flag), the second video availability flag (sps_vui_extension()), the second extension flag (sps_extension_3bits), and the first extension flag (sps_extension_data_flag).

[0221] The first presence flag indicates whether the first video availability information is recorded in the bitstream BS. When the first presence flag is true (1), it means that the first video availability information is recorded in the bitstream BS; when the first presence flag is false (0), it means that the first video availability information is not recorded in the bitstream BS.

[0222] Figure 25This is a simplified diagram illustrating an example of the syntax for the first video availability information in HEVC or AVC. The first video availability information consists of VUI parameters that contain information related to the aspect ratio of the image contained in the bitstream BS (sar_width, sar_height), as well as a portion of the image's color information (colour_primaries, transfer_characteristics, matrix_coeffs), etc.

[0223] Reference Figure 24 The first extended flag indicates whether the second video availability information, which extends the video availability information, is recorded in the bitstream BS. When the first extended flag is true (1), it means that the second video availability information is recorded in the bitstream BS; when the first extended flag is false (0), it means that the second video availability information is not recorded in the bitstream BS.

[0224] The second video availability information includes modal information and other related information (image association information) relating to the images contained in the BS in the bitstream. Details of the second video availability information will be described later.

[0225] The second extension flag indicates a reserved area secured for future extended use. The second extension flag indicates whether the first extension information, which extends other information in the NAL unit describing the second video availability information, is described in the bitstream BS. If the second extension flag is true (1), it indicates that the first extension information is described in the bitstream BS; if the second extension flag is false (0) or does not exist, it indicates that the first extension information is not described in the bitstream BS. If the first extension information is included at the very end of the NAL unit storing the first extension information, the first extension information has a bit string that continues to the last bit of the NAL unit.

[0226] In the case where the encoding device 1 corresponds to HEVC or AVC but not to the second video availability information, the encoding unit 33 encodes the first extended flag information, which is a false value (0), into the bitstream BS, and does not encode the second video availability information, the second extended flag information, and the first extended information into the bitstream BS.

[0227] Furthermore, if the decoding device 2 corresponds to HEVC or AVC but not to the second video availability information, and the first extended flag information is true (1), the decoding unit 52 ignores the second video availability information, the second extended flag information, and the first extended information during the decoding process. Ignoring these information during the decoding process includes not decoding or discarding the decoding result.

[0228] If the encoding device 1 corresponds to the second video availability information but cannot interpret the first extended information, the encoding unit 33 encodes the first extended flag information as a true value (1), the second extended flag information as a false value (0), and the second video availability information in the bitstream BS, but does not encode the first extended information in the bitstream BS.

[0229] Furthermore, if the decoding device 2 corresponds to the second video availability information but cannot interpret the first extended information, and the first extended flag information is true (1), the decoding unit 52 decodes the second video availability information and the second extended flag information from the bit stream BS. If the second extended flag information is true (1), the first extended information is ignored in the decoding process.

[0230] like Figure 2 , 3 As shown, the setting unit 32 sets parameters P such as SPS in step SP12, and the encoding unit 33 encodes the parameters P in the bitstream BS in step SP14. Furthermore, as... Figure 17 , 18 As shown, the decoding unit 52 decodes parameter P from the bit stream BS in step SP23.

[0231] Here, the encoding unit 33 encodes the first presence flag information in the bitstream BS. When the first presence flag information is true (1), the encoding unit 33 encodes the first video availability information in the bitstream BS; when the first presence flag information is false (0), it does not encode the first video availability information in the bitstream BS. Furthermore, the decoding unit 52 decodes the first presence flag information from the bitstream BS. When the first presence flag information is true (1), the decoding unit 52 decodes the first video availability information from the bitstream BS; when the first presence flag information is false (0), it does not decode the first video availability information from the bitstream BS.

[0232] Furthermore, the encoding unit 33 encodes the first extended flag information in the bitstream BS. When the first extended flag information is true (1), the encoding unit 33 encodes the second video availability information in the bitstream BS; when the first extended flag information is false (0), it does not encode the second video availability information in the bitstream BS. The decoding unit 52 decodes the first extended flag information from the bitstream BS. When the first extended flag information is true (1), the decoding unit 52 decodes the second video availability information from the bitstream BS; when the first extended flag information is false (0), it does not decode the second video availability information from the bitstream BS.

[0233] Furthermore, the encoding unit 33 encodes the second extended flag information in the bitstream BS. When the second extended flag information is true (1), the encoding unit 33 encodes the first extended information in the bitstream BS; when the second extended flag information is false (0), it does not encode the first extended information in the bitstream BS. The decoding unit 52 decodes the second extended flag information from the bitstream BS. When the second extended flag information is true (1), the decoding unit 52 decodes the first extended information from the bitstream BS; when the second extended flag information is false (0), it does not decode the first extended information from the bitstream BS.

[0234] Figure 26 This is a simplified illustration of the second example of the SPS syntax in HEVC.

[0235] When the first extended flag information is true (1) and the first existence flag information is true (1), the encoding unit 33 encodes the second video availability information in the bitstream BS. When the first extended flag information is true (1) and the first existence flag information is true (1), the decoding unit 52 decodes the second video availability information from the bitstream BS. Thus, contradictory processing such as encoding and decoding only the second video availability information without encoding and decoding the first video availability information can be prevented.

[0236] Figure 27 as well as Figure 28 This is a simplified diagram illustrating the first example of the syntax for SPS in AVC.

[0237] like Figure 27 As shown, the first NAL unit of SPS contains the first presence flag information (vui_parameters_present_flag) and the first video availability information (vui_parameters()). For example... Figure 28 As shown, the 2NAL unit of SPS contains the 1st extension flag information (additional_extension_flag), the 2nd video availability information (sps_vui_extension()), the 2nd extension flag information (additional_extension2_flag), and the 1st extension information (additional_extension2_data_flag).

[0238] The encoding unit 33 encodes the first presence flag information and the first video availability information in the first NAL unit. Furthermore, the encoding unit 33 encodes the first extended flag information, the second video availability information, the second extended flag information, and the first extended information in the second NAL unit. The decoding unit 52 decodes the first presence flag information and the first video availability information from the first NAL unit. Furthermore, the decoding unit 52 decodes the first extended flag information, the second video availability information, the second extended flag information, and the first extended information from the second NAL unit.

[0239] Figure 29 This is a simplified diagram illustrating the syntax of SPS in AVC, shown as the second example. The syntax of the 1NAL unit is... Figure 27 Similarly. Figure 29 As shown, the 2NAL unit, in addition to containing... Figure 28 In addition to the same first extended flag information, second video availability information, second extended flag information, and first extended information, it also includes second presence flag information (vui_parameters_present_flag). Like the first presence flag information, the second presence flag information indicates whether the first video availability information is recorded in the bitstream BS. When the second presence flag information is true (1), it indicates that the first video availability information is recorded in the bitstream BS; when the second presence flag information is false (0), it indicates that the first video availability information is not recorded in the bitstream BS.

[0240] When both the second extended flag information and the first existence flag information are true (1), the encoding unit 33 encodes the second video availability information in the bitstream BS. When both the second extended flag information and the first existence flag information are true (1), the decoding unit 52 decodes the second video availability information from the bitstream BS. This prevents contradictory processing where only the second video availability information is encoded and decoded without encoding or decoding the first video availability information.

[0241] Figure 30 This is a simplified diagram illustrating the first example of the syntax for the second video availability information in HEVC or AVC.

[0242] The second video availability information includes image association information, first size information (modality_type_extension_bits), extended modality information (reserved_modality_type_extension), second size information (vui_extension_bits), and second extension information (vui_reserved_extension).

[0243] Image association information includes modality information (modality_type) and wavelength information (min_wavelength_mantissa, max_wavelength_mantissa, min_wavelength_exponent, max_wavelength_exponent), etc. However, image association information is not limited to modality information and wavelength information; it can also be any information associated with the image contained in the bitstream (BS). For example, image association information can also be phase information communicated from the encoder to the decoder in Regression Propagation (RPR) coding techniques that scale down an image on the encoder side and enlarge it on the decoder side to avoid display position shifts. Phase information can also be set separately for the horizontal and vertical directions of the image.

[0244] The first size information represents the size of the extended modal information (in bits in this example). The extended modal information expands upon the modal information. The second size information represents the size of the second extended information (in bits in this example). The second extended information further expands upon the second video availability information.

[0245] A first-order size information bit greater than 0 indicates the presence of extended modality information and the number of bits of extended modality information within the second video availability information. A first-order size information value of 0 indicates the absence of extended modality information within the second video availability information. The value of the first-order size information ranges from 0 to, for example, 2048.

[0246] Similarly, a second size information bit greater than 0 indicates the second extended information and the number of bits of the second extended information present in the second video availability information. A second size information value of 0 indicates that the second extended information is not present in the second video availability information. The value of the second size information ranges from 0 to, for example, 2048.

[0247] The encoding unit 33 encodes the association information related to the images contained in the bitstream BS into the second video availability information.

[0248] Furthermore, when extended modal information is set, the encoding unit 33 encodes first size information representing the size of the extended modal information into the second video availability information, and encodes the extended modal information into the second video availability information based on the first size information. The decoding unit 52 decodes the first size information from the second video availability information, and decodes the extended modal information from the second video availability information based on the first size information.

[0249] On the other hand, if extended modal information is not set, or if extended modal information cannot be interpreted, the encoding unit 33 encodes 0 bits of the first size information in the second video availability information, and does not encode the extended modal information in the second video availability information. The decoding unit 52, when decoding 0 bits of the first size information from the second video availability information, does not decode the extended modal information from the second video availability information.

[0250] Furthermore, when the second extended information is set, the encoding unit 33 encodes the second size information, representing the size of the second extended information, into the second video availability information, and encodes the second extended information into the second video availability information based on the second size information. The decoding unit 52 decodes the second size information from the second video availability information, and decodes the second extended information from the second video availability information based on the second size information.

[0251] On the other hand, if the second extended information is not set, or if the second extended information cannot be interpreted, the encoding unit 33 encodes 0 bits of the second size information in the second video availability information, and does not encode the second extended information in the second video availability information. The decoding unit 52, when decoding 0 bits of the second size information from the second video availability information, does not decode the second extended information from the second video availability information. Therefore, if information is further added to the second video availability information due to future standard updates, modality-related information is recorded in the bitstream as extended modality information, and other video availability-related information is recorded in the bitstream as second extended information, thereby enabling appropriate encoding and decoding. On the other hand, even if information is further added to the second video availability information due to future standard updates, existing decoding devices that do not correspond to the aforementioned future standard updates can decode the bitstream without contradiction by ignoring this information, even if they record both the extended modality information and the second extended information in the bitstream.

[0252] Figure 31 This is a simplified diagram illustrating the second example of the syntax for the second video availability information in HEVC or AVC.

[0253] Second video availability information replaces Figure 30 The second size information shown includes the third extension flag information (vui_extension_flag). If the third extension flag information is true (1), it means that the second extension information is recorded in the bitstream BS. If the third extension flag information is false (0) or there is no third extension flag information, it means that the second extension information is not recorded in the bitstream BS.

[0254] Figure 32 This is a simplified diagram illustrating the third example of the syntax for the second video availability information in HEVC or AVC.

[0255] and Figure 31 Similarly, in the second example shown, the second video availability information includes the third extension flag information (vui_extension_flag). If the third extension flag information is true (1), it indicates that the second extension information is recorded in the bitstream BS; if the third extension flag information is false (0) or does not exist, it indicates that the second extension information is not recorded in the bitstream BS. If the second extension information is included at the very end of the NAL unit where it is stored, the second extension information has a bit string that continues to the last bit of the NAL unit.

[0256] According to the encoding apparatus 1 and decoding apparatus 2 of this development, the video availability information can be appropriately extended using the first extended flag information and the second video availability information. Furthermore, the extended flag information can be appropriately extended using the second extended flag information and the first extended information.

[0257] Industrial availability

[0258] This disclosure is particularly useful for image processing systems having an encoding device for transmitting an image in a bitstream and a decoding device for decoding an image from a received bitstream.

Claims

1. A decoding device, comprising: Circuit; and The memory is connected to the circuit. The circuit performs the following processing: Obtain a first presence flag from the bitstream indicating whether the first video availability information is recorded in the bitstream. If the first presence flag is true, the first video availability information is obtained from the bitstream. Obtain from the bitstream a first extended flag indicating whether the second video availability information is recorded in the bitstream. If the first extended flag information is true, the second video availability information is obtained from the bitstream. A second extension flag, indicating whether the first extension information is recorded in the bitstream, is obtained from the bitstream. The first extension information extends other information of the Network Abstraction Layer (NAL) unit that records the second video availability information. If the second extended flag information is true, the first extended information is obtained from the bit stream.

2. The decoding device according to claim 1, wherein, The circuit obtains the first presence flag information, the first video availability information, the first extended flag information, the second video availability information, the second extended flag information, and the first extended information from the same NAL unit.

3. The decoding device according to claim 2, wherein, The circuit obtains the second video availability information from the NAL unit when the first extended flag information is true and the first existence flag information is true.

4. The decoding device according to claim 1, wherein, The circuit performs the following processing: The first presence flag information and the first video availability information are obtained from the first NAL unit. The first extended flag information, the second video availability information, the second extended flag information, and the first extended information are obtained from the second NAL unit.

5. The decoding device according to claim 4, wherein, The circuit performs the following processing: Obtain a second presence flag indicating whether the first video availability information is recorded in the bitstream from the second NAL unit. If the first extended flag information is true and the second existence flag information is true, the second video availability information is obtained from the second NAL unit.

6. The decoding apparatus according to claim 1, wherein, The circuit performs the following processing: Image association information associated with the images contained in the bitstream is obtained from the second video availability information. When a second extended information, which further expands the second video availability information, is recorded in the second video availability information, size information representing the size of the second extended information is obtained from the second video availability information. Based on the size information, the second extended information is obtained from the second video availability information.

7. The decoding device according to claim 1, wherein, The circuit performs the following processing: Image association information associated with the images contained in the bitstream is obtained from the second video availability information. Obtain from the second video availability information a third extension flag indicating whether second extended information, which further expands the second video availability information, is recorded in the second video availability information. If the third extended flag information is true, the second extended information is obtained from the second video availability information.

8. The decoding apparatus according to claim 1, wherein, The first extended information has a bit string that continues to the last bit of the NAL unit where the first extended information is stored.

9. The decoding apparatus according to claim 7, wherein, The second extended information has a bit string that continues to the last bit of the NAL unit where the second extended information is stored.

10. The decoding apparatus according to claim 1, wherein, The second video availability information includes image association information associated with the images contained in the bitstream. In the decoding device that cannot interpret the image association information, if the first extended flag information is true, the circuit ignores the second video availability information, the second extended flag information, and the first extended information.

11. The decoding apparatus according to claim 1, wherein, The second video availability information includes image association information associated with the images contained in the bitstream. In the decoding device that can interpret the image association information but cannot interpret the first extended information, when the first extended flag information is true, the circuit performs the following processing: The second video availability information and the second extended flag information are obtained from the bitstream. If the second extended flag information is true, the first extended information is ignored.

12. An encoding device comprising: Circuit; and The memory is connected to the circuit. The circuit performs the following processing: The first presence flag, indicating whether the first video availability information is recorded in the bitstream, is encoded in the bitstream. If the first presence flag is true, the first video availability information is encoded in the bitstream. The first extended flag information, indicating whether the second video availability information is recorded in the bitstream, is encoded in the bitstream. If the first extended flag information is true, the second video availability information is encoded in the bitstream. A second extension flag, indicating whether the first extension information is recorded in the bitstream, is encoded in the bitstream. The first extension information extends other information of the Network Abstraction Layer (NAL) unit that records the second video availability information. If the second extended flag information is true, the first extended information is encoded in the bitstream.

13. The encoding device according to claim 12, wherein, The circuit encodes the first presence flag information, the first video availability information, the first extended flag information, the second video availability information, the second extended flag information, and the first extended information in the same NAL unit.

14. The encoding device according to claim 13, wherein, The circuit encodes the second video availability information in the NAL unit when the first extended flag information is true and the first existence flag information is true.

15. The encoding device according to claim 12, wherein, The circuit performs the following processing: The first presence flag information and the first video availability information are encoded in the first NAL unit. The first extended flag information, the second video availability information, the second extended flag information, and the first extended information are encoded in the second NAL unit.

16. The encoding device according to claim 15, wherein, The circuit performs the following processing: The second presence flag, indicating whether the first video availability information is recorded in the bitstream, is encoded in the second NAL unit. If the first extended flag information is true and the second existence flag information is true, the second video availability information is encoded in the second NAL unit.

17. The encoding device according to claim 12, wherein, The circuit performs the following processing: Image association information associated with the images contained in the bitstream is encoded in the second video availability information. When a second extended information, which further expands the second video availability information, is described in the second video availability information, the size information representing the size of the second extended information is encoded in the second video availability information. Based on the size information, the second extended information is encoded in the second video availability information.

18. The encoding device according to claim 12, wherein, The circuit performs the following processing: Image association information associated with the images contained in the bitstream is encoded in the second video availability information. The third extension flag, indicating whether the second extended information, which further expands the second video availability information, is recorded in the second video availability information, is encoded in the second video availability information. If the third extended flag information is true, the second extended information is encoded in the second video availability information.

19. The encoding device according to claim 12, wherein, The first extended information has a bit string that continues to the last bit of the NAL unit where the first extended information is stored.

20. The encoding device according to claim 18, wherein, The second extended information has a bit string that continues to the last bit of the NAL unit where the second extended information is stored.

21. The encoding device according to claim 12, wherein, The second video availability information includes image association information associated with the images contained in the bitstream. In the encoding device that cannot interpret the image association information, the circuit performs the following processing: The first extended flag information that is false is encoded in the bitstream. The second video availability information, the second extended flag information, and the first extended information are not encoded in the bitstream.

22. The encoding device according to claim 12, wherein, The second video availability information includes image association information associated with the images contained in the bitstream. In the encoding device that can interpret the image association information but cannot interpret the first extended information, the circuit performs the following processing: The first extended flag information that is true, the second extended flag information that is false, and the second video availability information are encoded in the bitstream. The first extended information is not encoded in the bitstream.

23. A decoding method, The decoding device performs the following processing: Obtain a first presence flag from the bitstream indicating whether the first video availability information is recorded in the bitstream. If the first presence flag is true, the first video availability information is obtained from the bitstream. Obtain from the bitstream a first extended flag indicating whether the second video availability information is recorded in the bitstream. If the first extended flag information is true, the second video availability information is obtained from the bitstream. A second extension flag, indicating whether the first extension information is recorded in the bitstream, is obtained from the bitstream. The first extension information extends other information of the Network Abstraction Layer (NAL) unit that records the second video availability information. If the second extended flag information is true, the first extended information is obtained from the bit stream.

24. An encoding device, The encoding device performs the following processing: The first presence flag, indicating whether the first video availability information is recorded in the bitstream, is encoded in the bitstream. If the first presence flag is true, the first video availability information is encoded in the bitstream. The first extended flag information, indicating whether the second video availability information is recorded in the bitstream, is encoded in the bitstream. If the first extended flag information is true, the second video availability information is encoded in the bitstream. A second extension flag, indicating whether the first extension information is recorded in the bitstream, is encoded in the bitstream. The first extension information extends other information of the Network Abstraction Layer (NAL) unit that records the second video availability information. If the second extended flag information is true, the first extended information is encoded in the bitstream.

Images