Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device includes a circuit and a memory connected to the circuit. The circuit acquires, from a bit stream, an image and a first parameter and a second parameter related to the image. The first parameter includes modality information indicating the image type of the image, and the second parameter includes wavelength information indicating the wavelength band of optical radiation on the image.

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method.

[0002] Patent Literature 1 discloses an image processing system according to the background art. The image processing system includes an encoder (encoding device) and a decoder (decoding device). Input images of various modalities are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image analysis task based on the input features, thereby outputting a segmentation map.

[0003] However, the background art does not consider at all the transmission of image modality information and wavelength information from an encoding device to a decoding device.

[0004] US Patent Application Publication No. 2024 / 0046453

[0005] The present disclosure aims to provide a decoding device, an encoding device, a decoding method, and an encoding method that can transmit image modality information and wavelength information from an encoding device to a decoding device, thereby improving the accuracy of task processing execution by the decoding device.

[0006] A decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit acquires an image and first and second parameters associated with the image from a bitstream, the first parameters including modality information indicating an image type of the image, and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

[0007] FIG. 1 is a simplified diagram showing the configuration of an image processing system according to an embodiment of the present disclosure. FIG. 2 is a simplified diagram showing the configuration of a circuit included in an encoding device. FIG. 3 is a flowchart showing processing executed by a circuit included in an encoding device. FIG. 4 is a simplified diagram showing the configuration of a bitstream. FIG. 5 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 6 is a diagram showing an example of the correspondence between values ​​of vui_modality_type and image types. FIG. 7 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 8 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 9 is a diagram showing an example of the correspondence between index values ​​and wavelength information. FIG. 10 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 11 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 12 is a diagram showing an example of the correspondence between values ​​of representation type information and interpretations of each value. FIG. 13 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 14 is a diagram showing an example of the correspondence between values ​​of interpolation information and interpretations of each value. FIG. 15 is a diagram showing an example of the configuration of a bitstream. FIG. 1 is a simplified diagram showing the configuration of a circuit included in a decoding device; FIG. 2 is a flowchart showing processing executed by a circuit included in a decoding device; FIG. 3 is a simplified diagram showing an example of table information; FIG. 4 is a simplified diagram showing an example of table information; FIG. 5 is a block diagram showing an example of the functional configuration of an encoding unit; FIG. 6 is a diagram showing an example of a hierarchical structure of data in a stream; and FIG. 7 is a block diagram showing an example of the functional configuration of a decoding unit.

[0008] (Findings underlying the present disclosure) An image processing system according to the background art includes an encoding device and a decoding device. Input images of various modalities are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image analysis task based on the input features, thereby outputting a segmentation map.

[0009] The task processing performed by the decoding device includes human vision and machine tasks. Human vision is the viewing or viewing of video images by a human, such as an operator or user. Machine tasks include various types of task processing using an AI model, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0010] In the background art, modality information and wavelength information are not transmitted from the encoding device to the decoding device, which may result in the selection of an optimal task process according to the image type of the input image, resulting in a decrease in the accuracy of the task process execution.

[0011] In order to solve this problem, the inventor discovered that the above problem can be solved by including modality information indicating the image type of an image and wavelength information indicating the wavelength band of the light radiation of the image in a bit stream and transmitting it from an encoding device to a decoding device, and the decoding device then performing task processing based on the modality information and wavelength information, and thus came up with the present disclosure.

[0012] Next, each aspect of the present disclosure will be described.

[0013] A decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit acquires an image and first and second parameters associated with the image from a bitstream, the first parameters including modality information indicating an image type of the image, and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

[0014] According to the first aspect, modality information indicating the image type of the image and wavelength information indicating the wavelength band of the optical radiation of the image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby making it possible to improve the accuracy of task processing execution by the decoding device.

[0015] In a decoding device according to a second aspect of the present disclosure, in the first aspect, the second parameter includes flag information indicating whether the wavelength information is described in the bit stream, and if the flag information indicates that the wavelength information is described in the bit stream, the circuit may obtain information indicating the minimum and maximum values ​​of the wavelength band from the bit stream.

[0016] According to the second aspect, the circuit can appropriately identify the wavelength band of the image by the acquired minimum and maximum values.

[0017] In the decoding device according to the third aspect of the present disclosure, in the second aspect, when the flag information indicates that the wavelength information is not described in the bit stream, the minimum value and the maximum value may be undefined.

[0018] According to the third aspect, the circuit can easily determine, by means of flag information, that the minimum and maximum values ​​of the wavelength band are not defined in the bit stream.

[0019] In the decoding device according to the fourth aspect of the present disclosure, in the first aspect, the second parameter may include an index value, and a minimum value and a maximum value of the wavelength band corresponding to the index value may be defined.

[0020] According to the fourth aspect, the circuit can appropriately identify the wavelength band of the image based on the acquired index value, and the amount of data transmitted from the encoding device to the decoding device can be reduced compared to when the minimum and maximum values ​​of the wavelength band are written in the bitstream.

[0021] A decoding device according to a fifth aspect of the present disclosure is the fourth aspect, wherein when the index value is a specific value, the minimum value and the maximum value are undefined.

[0022] According to the fifth aspect, it is possible to cope with cases where the image characteristics are unknown in the encoding device and the wavelength band of the image cannot be identified, or cases where the decoding device does not acquire wavelength information from the bitstream.

[0023] A decoding device according to a sixth aspect of the present disclosure is such that, in the fourth aspect, when the index value is a specific value, the minimum value and the maximum value are described in the bit stream, and the circuit obtains information indicating the minimum value and the maximum value from the bit stream.

[0024] According to the sixth aspect, it is possible to cope with cases where the minimum and maximum values ​​of the wavelength band are set by the user in the encoding device.

[0025] In the decoding device according to the seventh aspect of the present disclosure, in the second or sixth aspect, the information indicating the minimum value may include information indicating a mantissa of the minimum value and information indicating an exponent in an exponential representation using a predetermined base, and the information indicating the maximum value may include information indicating a mantissa of the maximum value and information indicating an exponent in an exponential representation using a predetermined base.

[0026] According to the seventh aspect, the minimum and maximum values ​​of the wavelength band can be appropriately described in exponential notation in the bit stream.

[0027] In the decoding device according to the eighth aspect of the present disclosure, in the seventh aspect, the information indicating the mantissa of the minimum value has three or more significant digits in decimal notation, and the information indicating the mantissa of the maximum value has three or more significant digits in decimal notation.

[0028] According to the eighth aspect, the minimum and maximum values ​​of the wavelength band can be described in the bit stream with sufficient significant digits.

[0029] In a decoding device according to a ninth aspect of the present disclosure, in the seventh aspect, when the information indicating the mantissa of the minimum value indicates "0", the minimum value is undefined, and when the information indicating the mantissa of the maximum value indicates "0", the maximum value is undefined.

[0030] According to the ninth aspect, it is also possible to deal with cases where the image characteristics are unknown in the encoding device and the minimum or maximum value of the wavelength band cannot be identified, or where the decoding device does not obtain the minimum or maximum value of the wavelength band from the bit stream.

[0031] In the decoding device of the tenth aspect of the present disclosure, in the second or fourth aspect, the minimum value is set to be equal to or greater than the shortest wavelength corresponding to the image type indicated in the modality information, and the maximum value is set to be equal to or less than the longest wavelength corresponding to the image type indicated in the modality information.

[0032] According to the tenth aspect, it is possible to avoid a discrepancy between the wavelength band corresponding to the image type and the wavelength band described in the bit stream.

[0033] In the decoding device of the 11th aspect of the present disclosure, in any one of the 1st to 10th aspects, the circuit, in acquiring the first parameter and the second parameter, may acquire the first parameter and the second parameter from a predetermined header area of ​​the bitstream, and the predetermined header area may include a VUI or an SEI.

[0034] According to the eleventh aspect, the decoding device can easily obtain the first parameter and the second parameter from a predetermined header area of ​​the bitstream.

[0035] In a decoding device according to a twelfth aspect of the present disclosure, in any one of the first to eleventh aspects, the bit stream has a multi-layer structure including multiple image layers, and the circuit, in acquiring the images, may acquire multiple images having different wavelength bands from the multiple image layers.

[0036] According to the twelfth aspect, the encoding device can transmit a plurality of images with different wavelength bands to the decoding device, and the decoding device can appropriately acquire the plurality of images from the bitstream.

[0037] A decoding device according to a thirteenth aspect of the present disclosure is the twelfth aspect, wherein the bitstream having the multi-layer structure is a bitstream obtained by encoding an output signal from a multispectral camera.

[0038] According to the thirteenth aspect, the encoding device can transmit a plurality of images of a plurality of wavelength bands included in an output signal from a multispectral camera to the decoding device, and the decoding device can appropriately acquire the plurality of images from the bitstream.

[0039] A coding device according to a fourteenth aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit encodes an image and first and second parameters associated with the image into a bitstream, the first parameters including modality information indicating an image type of the image, and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

[0040] According to the fourteenth aspect, modality information indicating the image type of the image and wavelength information indicating the wavelength band of the optical radiation of the image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby making it possible to improve the accuracy of task processing execution by the decoding device.

[0041] In the encoding device according to the fifteenth aspect of the present disclosure, in the fourteenth aspect, the second parameter includes flag information indicating whether the wavelength information is described in the bit stream, and the circuit encodes information indicating the minimum and maximum values ​​of the wavelength band into the bit stream, and in that case, sets the flag information indicating that the wavelength information is described in the bit stream.

[0042] According to the fifteenth aspect, the decoding device can appropriately identify the wavelength band of the image by the minimum and maximum values ​​by acquiring the wavelength information from the bitstream.

[0043] In the encoding device according to the 16th aspect of the present disclosure, in the 15th aspect, the circuit does not define the minimum value and the maximum value, and in that case, it is preferable to set the flag information indicating that the wavelength information is not described in the bit stream.

[0044] According to the sixteenth aspect, the decoding device can easily determine, by using flag information, that the minimum and maximum values ​​of the wavelength band are not defined in the bit stream.

[0045] A coding device according to a seventeenth aspect of the present disclosure is the fourteenth aspect, wherein the second parameter includes an index value, and a minimum value and a maximum value of the wavelength band corresponding to the index value are defined.

[0046] According to the seventeenth aspect, the decoding device can appropriately identify the wavelength band of an image by the index value by acquiring wavelength information from the bit stream. Furthermore, compared to the case where the minimum and maximum values ​​of the wavelength band are described in the bit stream, the amount of data transmitted from the encoding device to the decoding device can be reduced.

[0047] An encoding device according to an eighteenth aspect of the present disclosure is the seventeenth aspect, wherein the circuit does not define the minimum value and the maximum value, and in that case, the index value is set to a specific value.

[0048] According to the eighteenth aspect, it is possible to cope with cases where the image characteristics are unknown in the encoding device and the wavelength band of the image cannot be identified, or cases where the decoding device does not acquire wavelength information from the bitstream.

[0049] In the encoding device according to the 19th aspect of the present disclosure, in the 17th aspect, the circuit encodes information indicating the minimum value and the maximum value into the bit stream, and in that case, it is preferable that the index value be set to a specific value.

[0050] According to the nineteenth aspect, it is possible to cope with cases where the minimum and maximum values ​​of the wavelength band are set by the user in the encoding device.

[0051] In the encoding device according to the twentieth aspect of the present disclosure, in the fifteenth or nineteenth aspect, the information indicating the minimum value may include information indicating a mantissa of the minimum value and information indicating an exponent in exponential representation using a predetermined base, and the information indicating the maximum value may include information indicating a mantissa of the maximum value and information indicating an exponent in exponential representation using a predetermined base.

[0052] According to the twentieth aspect, the minimum and maximum values ​​of the wavelength band can be appropriately described in exponential notation in the bit stream.

[0053] In the encoding device according to the 21st aspect of the present disclosure, in the 20th aspect, it is preferable that the information indicating the mantissa of the minimum value has three or more significant digits in decimal notation, and the information indicating the mantissa of the maximum value has three or more significant digits in decimal notation.

[0054] According to the twenty-first aspect, the minimum and maximum values ​​of the wavelength band can be described in the bit stream with sufficient significant digits.

[0055] In the encoding device according to the 22nd aspect of the present disclosure, in the 20th aspect, the circuit may not define the minimum value, in which case the information indicating the mantissa of the minimum value is set to "0", and may not define the maximum value, in which case the information indicating the mantissa of the maximum value is set to "0".

[0056] According to the 22nd aspect, it is also possible to deal with cases where the image characteristics are unknown in the encoding device and the minimum or maximum value of the wavelength band cannot be identified, or where the decoding device does not obtain the minimum or maximum value of the wavelength band from the bit stream.

[0057] In the encoding device of the 23rd aspect of the present disclosure, in the 15th or 17th aspect, the circuit may set the minimum value to be equal to or greater than the shortest wavelength corresponding to the image type indicated in the modality information, and set the maximum value to be equal to or less than the longest wavelength corresponding to the image type indicated in the modality information.

[0058] According to the twenty-third aspect, it is possible to avoid a discrepancy between the wavelength band corresponding to the image type and the wavelength band described in the bit stream.

[0059] In the encoding device according to the 24th aspect of the present disclosure, in any one of the 14th to 23rd aspects, the circuit, in encoding the first parameter and the second parameter, may encode the first parameter and the second parameter in a predetermined header area of ​​the bitstream, and the predetermined header area may include a VUI or an SEI.

[0060] According to the twenty-fourth aspect, the decoding device can easily obtain the first parameter and the second parameter from a predetermined header area of ​​the bitstream.

[0061] In the encoding device according to the 25th aspect of the present disclosure, in any one of the 14th to 24th aspects, the bit stream has a multi-layer structure including multiple image layers, and the circuit, in encoding the images, may encode multiple images having different wavelength bands into the multiple image layers.

[0062] According to the twenty-fifth aspect, the encoding device can transmit a plurality of images with different wavelength bands to the decoding device, and the decoding device can appropriately acquire the plurality of images from the bitstream.

[0063] In the encoding device according to a 26th aspect of the present disclosure, in the 25th aspect, the circuit may generate the bitstream having the multi-layer structure by encoding an output signal from a multispectral camera.

[0064] According to the 26th aspect, the encoding device can transmit a plurality of images of a plurality of wavelength bands included in the output signal from the multispectral camera to the decoding device, and the decoding device can appropriately acquire the plurality of images from the bitstream.

[0065] A decoding method according to a 27th aspect of the present disclosure includes a decoding device that acquires an image and first and second parameters associated with the image from a bitstream, the first parameters including modality information indicating an image type of the image, and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

[0066] According to the 27th aspect, modality information indicating the image type of the image and wavelength information indicating the wavelength band of the optical radiation of the image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby making it possible to improve the accuracy of task processing execution by the decoding device.

[0067] An encoding method according to a 28th aspect of the present disclosure includes an encoding device encoding an image and first and second parameters related to the image into a bitstream, the first parameters including modality information indicating an image type of the image, and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

[0068] According to the 28th aspect, modality information indicating the image type of the image and wavelength information indicating the wavelength band of the optical radiation of the image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby making it possible to improve the accuracy of task processing execution by the decoding device.

[0069] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0070] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts are described as optional components. Furthermore, in all embodiments, the respective contents can be substituted or combined. Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or by any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0071] 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, and a transmission path NW.

[0072] Image data D1 is input from an external device to the encoding device 1. The external device may include a camera or the like that captures moving images. The external device inputs image data D1 of the captured moving images to the encoding device 1.

[0073] The encoding device 1 generates a bit stream BS based on the image data D1. Here, a bit stream refers to a data string or flow of digital data. The bit stream (or simply stream) may be a single stream, or may be composed of multiple streams divided into multiple layers. Furthermore, the bit stream may be transmitted by serial communication over a single transmission path, or by packet communication over multiple transmission paths. The encoding device 1 transmits the generated bit stream BS to the decoding device 2 via the transmission path NW. The decoding device 2 receives the bit stream BS.

[0074] The decoding device 2 decodes image data D1 from the bitstream BS and performs task processing based on the decoded image data D1. The task processing includes human vision and machine tasks. Human vision is the viewing or viewing of moving images by a human being such as an operator or a user. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using an artificial intelligence (AI) model, which is a machine-learned estimation model. The task processing unit that performs human vision includes a display device such as a liquid crystal display or an organic EL display. The task processing unit that performs machine tasks includes an inference device using AI.

[0075] The transmission path NW may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. The transmission path NW is preferably a private network or the like in which secure communication is ensured by access restrictions.

[0076] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 includes a processor such as a CPU. The memory 12 includes any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data to be processed by the circuit 11 or data in the middle of processing.

[0077] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 includes a processor such as a CPU. The memory 22 includes any recording medium such as a ROM, a RAM, a HDD, an SSD, or a semiconductor memory. The memory 22 stores data to be processed by the circuit 21 or data in the middle of processing.

[0078] 2 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 includes an acquisition unit 31, a setting unit 32, an encoding unit 33, and a transmission unit .

[0079] Next, the encoding unit 33 according to this embodiment will be described. Fig. 21 is a block diagram showing an example of the functional configuration of the encoding unit 33 according to this embodiment. The encoding unit 33 encodes an image in units of blocks.

[0080] 21 , the encoding unit 33 includes a dividing unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Note that the intra prediction unit 124 and the inter prediction unit 126 are configured as part of a prediction processing unit 125.

[0081] For example, the components of the encoding unit 33 shown in FIG. 21 are implemented by the circuit 11 and memory 12 shown in FIG.

[0082] The circuit 11 is configured to include a processor such as a CPU. The circuit 11 may be a dedicated or general-purpose electronic circuit for encoding images, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 11 may fulfill the roles of multiple components of the encoding unit 33 shown in FIG. 21 , excluding the component for storing information.

[0083] The memory 12 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 12 may be externally connected to the circuit 11 or may be built into the circuit 11. The memory 12 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 12 may be a non-volatile memory or a volatile memory.

[0084] The memory 12 may store an image to be encoded, or a stream corresponding to the encoded image, or may store a program for the processor to execute the image encoding process.

[0085] Furthermore, the memory 12 may serve as a component for storing information among the multiple components included in the encoding unit 33 shown in Fig. 21. Specifically, the memory 12 may serve as the block memory 118 and the frame memory 122 shown in Fig. 21. More specifically, the memory 12 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0086] 21 may be omitted from the encoding unit 33, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 21 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0087] FIG. 3 is a flowchart showing the process executed by the circuit 11 included in the encoding device 1.

[0088] First, in step SP11, the acquisition unit 31 acquires image data D11 representing the image Q to be processed, which is input from an external device. The image data D11 corresponds to the image data D1 shown in FIG.

[0089] Next, in step SP12, the setting unit 32 sets parameters P in association with the image Q. The parameters P include a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 includes modality information indicating the image type of the image Q. The second parameter P2 includes wavelength information indicating the wavelength band of the optical radiation of the image Q. The third parameter P3 includes color information of the image Q. The fourth parameter P4 includes extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension. The setting unit 32 inputs data D12 including the setting information of the parameters P to the encoding unit 33.

[0090] The first parameter P1 includes modality information indicating the image type of the image Q. The image type includes, for example, at least one of a visible light image, an infrared image, and an ultraviolet image. The visible light image includes a natural image or an RGB image, and is used to provide detailed color information in human vision or machine tasks. The infrared image includes an image captured using an infrared camera, and is used to capture images in the dark. The ultraviolet image includes an image captured using an ultraviolet camera, and is used to detect scratches on products. The modality information indicates that the image type is at least one of a visible light image, an infrared image, and an ultraviolet image. The modality information also indicates that the image type is at least one of an undefined image type and a spare image type for extension. In other words, the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, an undefined image type, and a spare image type for extension. Note that the modality information may include other image types. The setting unit 32 may set the first parameter P1 by image analysis based on the image data D11, or may set the first parameter P1 based on setting information input by an operator of the encoding device 1.

[0091] Next, in step SP13, the encoding unit 33 encodes the image Q represented by the image data D11 input from the acquisition unit 31 into a bit stream BS.

[0092] Next, in step SP14, the encoding unit 33 encodes the parameters P indicated by the data D12 input from the setting unit 32 into a bit stream BS. Here, encoding the parameters P into the bit stream BS may be rephrased as saving the parameters P in the bit stream BS or storing the parameters P in the bit stream BS. Note that the order of execution of steps SP13 and SP14 may be reversed from the example in FIG. 3, or steps SP13 and SP14 may be executed simultaneously.

[0093] Next, in step SP15, the transmitting unit 34 transmits the bit stream BS input from the encoding unit 33 to the decoding device 2 via the transmission path NW.

[0094] 4 is a simplified diagram showing the structure of a bitstream BS. The bitstream BS has a header area 41 and a payload area 42. The encoding unit 33 stores encoded data of an image Q in the payload area 42, and stores encoded data of a parameter P associated with the image Q in the header area 41.

[0095] The encoding unit 33 may encode the encoded data of the parameter P in a predetermined area 43 in the header area 41. The predetermined area 43 may be VUI (video usability information) or SEI (supplemental enhancement information). Although an example using VUI parameters will be described in this specification, when describing in SEI, the VUI parameters may be read as SEI parameters. However, the predetermined area 43 is not limited to VUI or SEI, and may also be VPS, SPS, PPS, PH, SH, APS, a tile header, a system layer header, or the like.

[0096] 22 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. As shown in (A) of FIG. 22, the video sequence includes, for example, a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), supplemental enhancement information (SEI), and multiple pictures.

[0097] In a video composed of multiple layers, the VPS includes coding parameters common to multiple layers and coding parameters related to multiple layers or individual layers included in the video.

[0098] The SPS includes parameters used for a sequence, i.e., encoding parameters that the decoding device 2 refers to in order to decode the sequence. The encoding parameters may indicate, for example, the width or height of a picture. Note that there may be multiple SPSs.

[0099] The PPS includes parameters used for a picture, i.e., encoding parameters that the decoding device 2 references to decode each picture in a sequence. The encoding parameters may include, for example, a reference value of the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Note that there may be multiple PPSs. The SPS and PPS may also be simply referred to as parameter sets.

[0100] As shown in (B) of Fig. 22, a picture includes a picture header and one or more slices. The picture header includes coding parameters that are referenced by the decoding device 2 to decode the one or more slices.

[0101] As shown in Fig. 22C, a slice includes a slice header and one or more bricks. The slice header includes coding parameters that are referenced by the decoding device 2 to decode the one or more bricks.

[0102] As shown in FIG. 22(D), a brick includes one or more CTUs (Coding Tree Units).

[0103] Note that a picture may not contain slices, but may contain tile groups instead, where each tile group contains one or more tiles. Also, a brick may contain slices.

[0104] A CTU is also called a superblock or a basic division unit. As shown in (E) of Fig. 22, a CTU includes a CTU header and one or more coding units (CUs). The CTU header includes coding parameters that the decoding device 2 references to decode the one or more CUs.

[0105] A CU may be divided into multiple smaller CUs. Furthermore, as shown in (F) of FIG. 22 , a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting a CU. The residual coefficient information is information indicating a prediction residual. A CU is basically the same as a PU (Prediction Unit) or a TU (Transform Unit), but may include multiple TUs smaller than a CU. A CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is a fixed unit that can be processed in one stage, for example, when performing pipeline processing in hardware.

[0106] Note that a stream may not have some of the layers among the layers shown in Fig. 22. Furthermore, the order of these layers may be changed, and any layer may be replaced with another layer.

[0107] A picture currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current picture. If the processing is encoding, the current picture is synonymous with a picture to be encoded, and if the processing is decoding, the current picture is synonymous with a picture to be decoded. Furthermore, a block (CU or a block of a CU) currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current block. If the processing is encoding, the current block is synonymous with a block to be encoded, and if the processing is decoding, the current block is synonymous with a block to be decoded.

[0108] FIG. 5 is a simplified diagram showing an example of the syntax of VUI parameters including a first parameter P1 and a third parameter P3.

[0109] The first parameter P1 includes flag information (vui_modality_info_present_flag) indicating whether modality information is described in the bitstream BS. When the value of vui_modality_info_present_flag is 1, it indicates that modality information is present in the VUI parameters, and when the value of vui_modality_info_present_flag is 0, it indicates that modality information is not present in the VUI parameters. When vui_modality_info_present_flag indicates that modality information is described in the bitstream BS, the modality information is represented as the value of the identifier vui_modality_type included in the VUI parameters.

[0110] FIG. 6 is a diagram showing an example of the correspondence between the value of vui_modality_type and the image type. As shown in FIG. 6, a value of 0 for vui_modality_type indicates that the image type of image Q is not defined (i.e., undefined). A value of 1 for vui_modality_type indicates that the image type of image Q is a visible light image. A value of 2 for vui_modality_type indicates that the image type of image Q is an infrared image. A value of 3 for vui_modality_type indicates that the image type of image Q is an ultraviolet image. Other values ​​of vui_modality_type (4-255 in this example) indicate a spare area reserved for future extended use. Note that the number of image types defined in FIG. 6 may be increased depending on the image Q to be processed. Furthermore, the absence of vui_modality_type may mean that the value of vui_modality_type is 0. Furthermore, when the value of vui_modality_type is 0, wavelength information indicating a wide wavelength band spanning multiple image types, such as a hyperspectral image, may be defined by user settings or the like.

[0111] As shown in Fig. 5, the third parameter P3 includes flag information (vui_colour_description_present_flag) indicating whether color information is described in the bitstream BS. A value of 1 in vui_colour_description_present_flag indicates that color information is present in the VUI parameters, and a value of 0 in vui_colour_description_present_flag indicates that color information is not present in the VUI parameters. Although not shown, the color information includes vui_colour_primaries, vui_transfer_characteristics, vui_matrix_coeffs, etc. Note that if the image type is an infrared image or an ultraviolet image, the third parameter P3 does not need to include color information. In the encoding process, if the image type indicated by the modality information is a visible light image, the encoding device 1 sets the value of vui_colour_description_present_flag to 1 and describes color information in the VUI parameters, whereas if the image type indicated by the modality information is an infrared image or an ultraviolet image, the encoding device 1 sets the value of vui_colour_description_present_flag to 0 and does not describe color information in the VUI parameters. Furthermore, in the decoding process, if the image type indicated by the modality information is a visible light image, the decoding device 2 acquires color information from the bitstream BS, whereas if the image type indicated by the modality information is an infrared image or an ultraviolet image, it is not necessary to acquire color information from the bitstream BS, or even if color information is acquired from the bitstream BS, the value of the color information may be ignored.

[0112] FIG. 7 is a simplified diagram showing an example of the syntax of VUI parameters including the fourth parameter P4.

[0113] The fourth parameter P4 includes extension bit information corresponding to the reserved image type when the modality information indicates that the image type is a reserved image type for extension. The extension bit information includes information (vui_modality_type_extension_bits) indicating the bit length of additional information required for the reserved image type and additional information (vui_reserved_modality_type_extension) having the bit length. A value of vui_modality_type_extension_bits of 0 indicates that vui_reserved_modality_type_extension is not described in the bitstream BS. On the other hand, a value of vui_modality_type_extension_bits greater than 0 indicates that vui_reserved_modality_type_extension is described in the bitstream BS and its bit length. The maximum value of vui_modality_type_extension_bits can be set arbitrarily, for example, to 2048.

[0114] 8 is a simplified diagram showing a first example of the syntax of VUI parameters including a second parameter P2, which includes wavelength information indicating the wavelength band of the light radiation of the image Q.

[0115] 8, the second parameter P2 includes flag information (vui_spectrum_range_present_flag) indicating whether wavelength information is described in the bitstream BS. When the value of vui_spectrum_range_present_flag is 0, it indicates that the spectral band of the optical radiation wavelength represented by the CLVS (Coded Layer Video Sequence) image does not exist in the VUI parameters, and when the value of vui_spectrum_range_present_flag is 1, it indicates that the spectral band of the optical radiation wavelength represented by the CLVS image exists in the VUI parameters.

[0116] When the value of vui_spectrum_range_present_flag is 1, the wavelength information includes information indicating the minimum value (minimum wavelength) and maximum value (maximum wavelength) of the wavelength band of image Q. Both the minimum wavelength and the maximum wavelength are wavelengths in meters.

[0117] The information indicating the minimum value of the wavelength band includes information indicating the mantissa of the minimum value in exponential notation using a predetermined base (for example, 10) (vui_min_wavelength_mantissa) and information indicating the value obtained by adding 15 to the exponent of the minimum value (vui_min_wavelength_exponent_plus15). The minimum value Min_Wavelength of the wavelength band is expressed as Min_Wavelength = vui_min_wavelength_mantissa × 10 vui_min_wavelength_exponent_plus15 - 15. As a result, in the example shown in FIG. 8, 10 －１５ ~10 ＋１６ A minimum value can be specified within the range of (m). Here, if vui_min_wavelength_mantissa is 0 or does not exist, it means that the minimum value of the wavelength band is unknown or unspecified and is not defined, or that it is defined using a method other than exponential notation. In this case, the decoding device 2 does not need to acquire vui_min_wavelength_exponent_plus15 from the bitstream BS during the decoding process, and even if it acquires vui_min_wavelength_exponent_plus15 from the bitstream BS, it may ignore the value.

[0118] The information indicating the maximum value of the wavelength band includes information indicating the mantissa of the maximum value in exponential notation using a predetermined base (for example, 10) (vui_max_wavelength_mantissa) and information indicating the value obtained by adding 15 to the exponent of the maximum value (vui_max_wavelength_exponent_plus15). The maximum value of the wavelength band, Max_Wavelength, is expressed as Max_Wavelength = vui_max_wavelength_mantissa × 10 vui_max_wavelength_exponent_plus15 - 15. As a result, in the example shown in FIG. 8, 10 －１５ ~10 ＋１６ The maximum value can be specified in the range of (m). Here, if vui_max_wavelength_mantissa is 0 or does not exist, it means that the maximum value of the wavelength band is unknown or unspecified and therefore not defined, or that it is defined using a method other than exponential notation. In this case, the decoding device 2 does not need to acquire vui_max_wavelength_exponent_plus15 from the bitstream BS, or even if it acquires vui_max_wavelength_exponent_plus15 from the bitstream BS, it may ignore the value.

[0119] Both vui_min_wavelength_mantissa and vui_max_wavelength_mantissa have three or more significant decimal digits. In the example shown in FIG. 8, both vui_min_wavelength_mantissa and vui_max_wavelength_mantissa have a binary bit length of 10 bits, but may have any bit length of 11 bits or more. Similarly, in the example shown in FIG. 8, both vui_min_wavelength_exponent_plus15 and vui_max_wavelength_exponent_plus15 have a binary bit length of 5 bits, but may have any bit length of 6 bits or more. Furthermore, the value added to the exponent is not limited to "+15", and a syntax for adding another fixed value may be used.

[0120] In the encoding device 1, the minimum value of the wavelength band indicated by vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15 is set to be equal to or greater than the shortest wavelength corresponding to the image type indicated by the modality information. Similarly, in the encoding device 1, the maximum value of the wavelength band indicated by vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15 is set to be equal to or less than the longest wavelength corresponding to the image type indicated by the modality information.

[0121] FIG. 9 is a simplified diagram showing a second example of the syntax of the VUI parameters including the second parameter P2.

[0122] In the example shown in FIG. 9, the second parameter P2 includes information (vui_spectrum_range) indicating an index value of the wavelength information.

[0123] 10 is a diagram showing an example of the correspondence between index values ​​and wavelength information. A minimum value (minimum wavelength) and a maximum value (maximum wavelength) of a wavelength band are predefined corresponding to each of a plurality of index values, and are shared by the encoding device 1 and the decoding device 2. In the example shown in FIG. 10, index values ​​2 to 6 correspond to the ultraviolet wavelength region, index value 7 corresponds to the visible light wavelength region, and index values ​​8 to 10 correspond to the infrared wavelength region.

[0124] For example, if the index value is 2, the minimum wavelength is 1*10 －９ (m), the maximum wavelength is 1*10 －７ (m), and if the index value is, for example, 3, the minimum wavelength is 1*10 －７ (m), the maximum wavelength is 19*10 －８ (m) is defined as:

[0125] Furthermore, when the index value is a certain value (0 in the example shown in FIG. 10), it means that the minimum wavelength and the maximum wavelength are not defined.

[0126] If the index value is another specific value (1 in the example shown in FIG. 10), this means that the desired minimum and maximum wavelengths are defined by user settings or the like.

[0127] Note that other index values ​​(11-255 in this example) indicate spare areas reserved for future extended use.

[0128] In the encoding device 1, the index value is set according to the wavelength band corresponding to the image type indicated by the modality information. For example, if vui_modality_type is 1 (visible light image), vui_spectrum_range is set to 0, 1, or 7. If vui_modality_type is 2 (infrared image), vui_spectrum_range is set to 0, 1, or any of 8 to 10. If vui_modality_type is 3 (ultraviolet image), vui_spectrum_range is set to 0, 1, or any of 2 to 6. In other words, the encoding device 1 sets an appropriate index value according to the image type, allowing the decoding device 2 to identify an appropriate wavelength band based on the index value and decode the image.

[0129] FIG. 11 is a simplified diagram showing a third example of the syntax of VUI parameters including the second parameter P2.

[0130] In the example shown in Figure 9, if the index value of vui_spectrum_range is set to 1 (user-defined), the minimum value of the wavelength band may be defined by vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15, as in the example shown in Figure 8, and the maximum value of the wavelength band may be defined by vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15.

[0131] FIG. 12 is a simplified diagram showing a first modified example of the syntax of VUI parameters.

[0132] If the image type indicated by the modality information is an infrared image, the encoding device 1 may further encode representation type information (vui_infrared_radiance_representation_type) indicating the representation type of infrared radiance into the bitstream BS.

[0133] FIG. 13 is a diagram showing an example of the correspondence between values ​​of the expression type information and interpretations of each value.

[0134] If vui_infrared_radiance_representation_type is 0 or does not exist, it means that the representation type of the infrared radiation intensity is unknown or unspecified and the representation type is not defined, or is defined by a method other than intensity value representation or temperature value representation.

[0135] If the value of vui_infrared_radiance_representation_type is 1, this means that the representation type of infrared radiation intensity is intensity value representation. In the intensity value representation, each pixel value of image Q indicates an intensity value of infrared radiation intensity.

[0136] If the value of vui_infrared_radiance_representation_type is 2, this means that the representation type of the infrared radiation intensity is temperature value representation. In the temperature value representation, each pixel value of image Q indicates a temperature value converted from the intensity value of the infrared radiation intensity.

[0137] Any other value of vui_infrared_radiance_representation_type (3 in this example) indicates a spare area reserved for future expansion.

[0138] 12, when the value of vui_infrared_radiance_representation_type is 2 (i.e., temperature value representation), the encoding device 1 describes temperature information in the VUI parameters. The temperature information includes the minimum and maximum temperatures of the temperature value, which is a floating-point value.

[0139] The information indicating the minimum temperature includes information (vui_min_temperature_sign) indicating the positive or negative sign of the minimum temperature in exponential notation using a predetermined base (for example, 2), information indicating the exponent of the minimum temperature (vui_min_temperature_exponent), information indicating the value obtained by subtracting 1 from the bit length of the mantissa of the minimum temperature (vui_min_temperature_mantissa_len_minus1), and information indicating the mantissa of the minimum temperature (vui_min_temperature_mantissa). Note that vui_min_temperature_mantissa_len_minus1 may be omitted by setting the bit length of the mantissa of the minimum temperature to a fixed value.

[0140] If the value of vui_min_temperature_sign is 0, it indicates that the sign of the minimum temperature is positive, and if the value of vui_min_temperature_sign is 1, it indicates that the sign of the minimum temperature is negative.

[0141] The information indicating the maximum temperature includes information (vui_max_temperature_sign) indicating the sign of the maximum temperature in exponential notation using a predetermined base (for example, 2), information (vui_max_temperature_exponent) indicating the exponent of the maximum temperature, information (vui_max_temperature_mantissa_len_minus1) indicating the value obtained by subtracting 1 from the bit length of the mantissa of the maximum temperature, and information (vui_max_temperature_mantissa) indicating the mantissa of the maximum temperature. Note that vui_max_temperature_mantissa_len_minus1 may be omitted by setting the bit length of the mantissa of the maximum temperature to a fixed value.

[0142] If the value of vui_max_temperature_sign is 0, it indicates that the sign of the maximum temperature is positive, and if the value of vui_max_temperature_sign is 1, it indicates that the sign of the maximum temperature is negative.

[0143] FIG. 14 is a simplified diagram showing a second modified example of the syntax of the VUI parameters.

[0144] The encoding device 1 may further encode flag information (vui_false_colour_representation_present_flag) indicating whether or not pixel values ​​of the image Q are represented using a false colour representation into the bitstream BS.

[0145] If the value of vui_false_colour_representation_present_flag is 1, it indicates that the pixel values ​​of image Q are represented using pseudo-colour representation, and if the value of vui_false_colour_representation_present_flag is 0, it indicates that the pixel values ​​of image Q are not represented using pseudo-colour representation.

[0146] When the pixel values ​​of image Q are expressed using pseudo-color representation, generation information for generating table information that associates each pixel value of image Q with a color code of the pseudo-color representation is described in the VUI parameters.

[0147] The generation information includes information (vui_num_hex_colour_code_minus2) indicating the value obtained by subtracting 2 from the number of color codes encoded in the bitstream BS, information (vui_num_value_minus2) indicating the value obtained by subtracting 2 from the number of pairs of pixel values ​​and color codes, interpolation information (vui_colour_interpolation_type) for interpolating values ​​between the pairs, and information (vui_hex_colour_code[i]) indicating the i-th hexadecimal color code. Each hexadecimal color code is 24 bits long, with the first 8 bits representing the depth of red, the middle 8 bits representing the depth of green, and the last 8 bits representing the depth of blue. For example, if vui_hex_colour_code is 000000, it indicates black (RGB: 0,0,0), and if vui_hex_colour_code is FFFFFF, it indicates white (RGB: 255,255,255).

[0148] For example, if the value of vui_num_hex_colour_code_minus2 is 0, it means that the number of colour codes coded in the bitstream BS is 2.

[0149] For example, if the value of vui_num_value_minus2 is 4, this means that the number of pairs of pixel values ​​and color codes is 6. If the number of color codes coded in the bitstream is 2 and the number of pairs is 6, this means that the range of color components is divided into 5 equal sections based on the values ​​of the two color codes, and the values ​​of the middle 4 pairs of color codes are generated by interpolation, thereby generating table information containing a total of 6 pairs of combinations of pixel values ​​and color codes.

[0150] FIG. 15 is a diagram showing an example of the correspondence between the values ​​of the interpolation information and the interpretation of each value.

[0151] If vui_colour_interpolation_type is 0 or not present, it means that the interpolation type is unknown or unspecified and not defined, or is defined by a method other than linear or bilinear interpolation.

[0152] If vui_colour_interpolation_type is 1, it means that the interpolation type is linear.

[0153] If vui_colour_interpolation_type is 2, it means that the interpolation type is bilinear interpolation.

[0154] Other values ​​of vui_colour_interpolation_type (3-15 in this example) indicate a spare area reserved for future expansion.

[0155] FIG. 16 is a diagram showing an example of the structure of the bit stream BS.

[0156] The bitstream BS is a multi-layer image layer L １ ~L ｍ16 shows only one access unit. An access unit is the minimum processing unit of a time attribute, and corresponds to, for example, one frame of a video image. A bit stream BS is configured to include multiple access units that are consecutive in time.

[0157] The first image layer L, which is the lowest layer １ The payload area 42 of the image Q Ｌ１ The coded data of the second image layer L is stored. ２ The payload area 42 of the image Q Ｌ２ Similarly, the encoded data of the m-th image layer L ｍ The payload area 42 of the image Q Ｌｍ The encoded data is stored.

[0158] Image Q Ｌ１ ~Q Ｌｍ may have different image types or wavelength bands. For example, in a three-layer multi-layer structure with m=3, image Q Ｌ１ is the visible light image, and image Q Ｌ２ is the infrared image, and image Q Ｌ３ For example, when encoding an output signal from a multispectral camera, the encoding device 1 divides the spectral range into m regions and generates an image Q for each spectral range. Ｌ１ ~Q Ｌｍ may be coded as a multi-layered bitstream BS.

[0159] Image Q Ｌ１ ~Q Ｌｍ If there is a correlation between the image layers L １ ~L ｍ Images may be referenced between images. Ｌ１ ~Q Ｌｍ If there is no correlation between the image layers L １ ~L ｍ Images do not need to be referenced between them.

[0160] Image layer L １ The header area 41 of the image Q Ｌ１The parameter P associated with Ｌ１ The encoded data of the parameter P is stored. Ｌ１ is image Q Ｌ１ This includes modality information and wavelength information.

[0161] Image layer L ２ The header area 41 of the image Q Ｌ２ The parameter P associated with Ｌ２ The encoded data of the parameter P is stored. Ｌ２ is image Q Ｌ２ This includes modality information and wavelength information.

[0162] Image layer L ｍ The header area 41 of the image Q Ｌｍ The parameter P associated with Ｌｍ The encoded data of the parameter P is stored. Ｌｍ is image Q Ｌｍ This includes modality information and wavelength information.

[0163] 17 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 includes a receiving unit 51, a decoding unit 52, a switching unit 53, and a plurality of n task processing units 54 (n is a natural number of 2 or more). １ ~54 ｎ The task processing unit 54 １ ~54 ｎ The tasks performed by include human vision and machine tasks, including various types of tasks using AI models, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0164] Next, the decoding unit 52 according to this embodiment will be described. Fig. 23 is a block diagram showing an example of the functional configuration of the decoding unit 52 according to this embodiment. The decoding unit 52 decodes a stream, which is an encoded image, in units of blocks.

[0165] 23 , the decoding unit 52 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a partition determination unit 224. Note that the intra prediction unit 216 and the inter prediction unit 218 are configured as part of a prediction processing unit 215.

[0166] For example, the components of the decoding unit 52 shown in FIG. 23 are implemented by the circuit 21 and memory 22 shown in FIG.

[0167] The circuit 21 is configured to include a processor such as a CPU. The circuit 21 may be a dedicated or general-purpose electronic circuit for decoding streams, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 21 may fulfill the roles of multiple components of the decoding unit 52 shown in FIG. 23 , excluding the component for storing information.

[0168] The memory 22 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 22 may be externally connected to the circuit 21 or may be built into the circuit 21. The memory 22 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 22 may be a non-volatile memory or a volatile memory.

[0169] The memory 22 may store a stream to be decoded or a decoded image, or may store a program for the processor to execute the decoding process of the stream.

[0170] Furthermore, the memory 22 may serve as a component for storing information among the multiple components included in the decoding unit 52 shown in Fig. 23. Specifically, the memory 22 may serve as the block memory 210 and the frame memory 214 shown in Fig. 23. More specifically, the memory 22 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0171] 23 may be omitted from the decoding unit 52, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 23 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0172] The inverse quantization unit 204, inverse transform unit 206, addition unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter 212 included in the decoding unit 52 shown in Figure 23 perform processing similar to that of the inverse quantization unit 112, inverse transform unit 114, addition unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter 120 included in the encoding unit 33 shown in Figure 21.

[0173] FIG. 18 is a flowchart showing the processing executed by the circuit 21 included in the decoding device 2.

[0174] First, in step SP21, the receiving unit 51 receives the bit stream BS transmitted by the encoding device 1 from the transmission path NW.

[0175] Next, in step SP22, the decoding unit 52 obtains image Q by decoding it from the payload area 42 of the bitstream BS input from the receiving unit 51. Note that the decoding may include extraction. The decoding unit 52 outputs image data D21 of image Q. The image data D21 corresponds to the image data D11 shown in FIG. 2.

[0176] As shown in FIG. 16, the bitstream BS includes multiple image layers L １ ~L ｍ In the case where the image has a multi-layer structure including a plurality of images Q having different image types or wavelength bands, the decoding unit 52 Ｌ１ ~Q Ｌｍ multiple image layers L １ ~L ｍ You can also get it from.

[0177] Next, in step SP23, the decoding unit 52 obtains the parameters P by decoding them from the header region 41 (or a predetermined region 43 within the header region 41) of the bit stream BS input from the receiving unit 51. As described above, the parameters P include a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 includes modality information indicating the image type of the image Q. The second parameter P2 includes wavelength information indicating the wavelength band of the optical radiation of the image Q. The third parameter P3 includes color information of the image Q. The fourth parameter P4 includes extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension. Note that the order of execution of steps SP22 and SP23 may be reversed from the example in FIG. 18, or they may be performed simultaneously.

[0178] 12 is included in the parameter P, and if the modality information indicates that the image type is an infrared image (i.e., the value of vui_modality_type is 2), the decoding unit 52 further acquires the above-mentioned expression type information indicating the expression type of infrared radiation intensity from the bitstream BS. Also, if the expression type information indicates that the pixel value indicates a temperature value (i.e., the expression is a temperature value), the decoding unit 52 further acquires temperature information including the minimum and maximum temperatures of the temperature values ​​from the bitstream BS.

[0179] 14 is included in the parameter P, and if the flag information indicates that the pixel values ​​are represented using pseudo-color representation (i.e., if the value of vui_false_colour_representation_present_flag is 1), the decoding unit 52 further acquires from the bitstream BS the generation information for generating table information that associates each pixel value of the image Q with a color code of the pseudo-color representation. The decoding unit 52 also generates the table information based on the acquired generation information.

[0180] 19 and 20 are simplified diagrams showing examples of table information.

[0181] 19 and 20, the number of pairs of pixel values ​​and color codes is 6 (i.e., the value of vui_num_value_minus2 is 4), the image type is an infrared image, and the interpolation type may be linear or bilinear.

[0182] In the example shown in FIG. 19, the table information indicates the correspondence between hexadecimal color codes, RGB color codes, and gray shades.

[0183] 20, the table information shows the correspondence between hex color codes, shades of gray, and temperatures, with shades of gray being mapped to temperatures. For example, black represents a temperature of 300K, gray represents a temperature of 330K, and white represents a temperature of 350K.

[0184] Next, in step SP24, the switching unit 53 switches the task processing unit 54 based on the modality information input from the decoding unit 52. １ ~54 ｎ The switching unit 53 switches between a plurality of image types and a plurality of task processing units 54. １ ~54 ｎ The switching unit 53 holds table information (not shown) in which a correspondence relationship between the task and the task is preset. １ ~54 ｎ Among these, one task processing unit 54 corresponding to the image type indicated by the modality information is selected.１ ~54 ｎ Select .

[0185] Referring to FIG. 18, next, in step SP25, the one task processing unit 54 selected in step SP24 １ ~54 ｎ executes task processing based on image data D21 input from the decoding unit 52 via the switching unit 53.

[0186] According to the encoding device 1 and decoding device 2 of this embodiment, modality information indicating the image type of image Q and wavelength information indicating the wavelength band of the optical radiation of image Q can be included in the bit stream BS and transmitted from the encoding device 1 to the decoding device 2, thereby making it possible to improve the accuracy of task processing execution by the decoding device 2.

[0187] Furthermore, according to the encoding device 1 and the decoding device 2 of this embodiment, the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension. Therefore, since the modality information includes modality information indicating that the image type is undefined or that the image type is a spare image type for extension, it is possible to deal with cases where the image characteristics are unknown in the encoding device 1 and the image type cannot be identified, or with future extensions of the image type, etc.

[0188] The present disclosure is particularly useful when applied to an image processing system that includes an encoding device that encodes an image into a bitstream and transmits the bitstream, and a decoding device that decodes an image from the received bitstream.

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, the circuit obtaining from a bitstream an image and first and second parameters associated with the image, the first parameters including modality information indicating an image type of the image, and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

2. The decoding device of claim 1, wherein the second parameter includes flag information indicating whether or not the wavelength information is described in the bit stream, and if the flag information indicates that the wavelength information is described in the bit stream, the circuit obtains information indicating the minimum and maximum values ​​of the wavelength band from the bit stream.

3. The decoding device according to claim 2, wherein when the flag information indicates that the wavelength information is not described in the bit stream, the minimum value and the maximum value are not defined.

4. The decoding device according to claim 1, wherein the second parameter includes an index value, and a minimum value and a maximum value of the wavelength band corresponding to the index value are defined.

5. The decoding device according to claim 4, wherein when the index value is a specific value, the minimum value and the maximum value are undefined.

6. The decoding device according to claim 4, wherein when the index value is a specific value, the minimum value and the maximum value are described in the bit stream, and the circuit obtains information indicating the minimum value and the maximum value from the bit stream.

7. A decoding device as described in claim 2 or 6, wherein the information indicating the minimum value includes information indicating a mantissa of the minimum value and information indicating an exponent in exponential representation using a predetermined base, and the information indicating the maximum value includes information indicating a mantissa of the maximum value and information indicating an exponent in exponential representation using a predetermined base.

8. The decoding device according to claim 7, wherein the information indicating the mantissa of the minimum value has three or more significant digits in decimal notation, and the information indicating the mantissa of the maximum value has three or more significant digits in decimal notation.

9. The decoding device according to claim 7, wherein when the information indicating the mantissa of the minimum value indicates "0", the minimum value is undefined, and when the information indicating the mantissa of the maximum value indicates "0", the maximum value is undefined.

10. A decoding device as described in claim 2 or 4, wherein the minimum value is set to be equal to or greater than the shortest wavelength corresponding to the image type indicated in the modality information, and the maximum value is set to be equal to or less than the longest wavelength corresponding to the image type indicated in the modality information.

11. The decoding device of claim 1, wherein the circuit, in obtaining the first parameter and the second parameter, obtains the first parameter and the second parameter from a predetermined header area of ​​the bitstream, and the predetermined header area includes a VUI or a SEI.

12. The decoding device according to claim 1, wherein the bit stream has a multi-layer structure including a plurality of image layers, and the circuit, in acquiring the image, acquires a plurality of images having different wavelength bands from the plurality of image layers.

13. The decoding device according to claim 12, wherein the bit stream having the multi-layer structure is a bit stream in which an output signal from a multispectral camera is encoded.

14. An encoding device comprising: a circuit; and a memory connected to the circuit, the circuit encoding an image, a first parameter and a second parameter associated with the image into a bit stream, the first parameter including modality information indicating an image type of the image, and the second parameter including wavelength information indicating a wavelength band of optical radiation of the image.

15. The encoding device described in claim 14, wherein the second parameter includes flag information indicating whether or not the wavelength information is described in the bit stream, and the circuit encodes information indicating the minimum and maximum values ​​of the wavelength band into the bit stream, and in that case sets the flag information indicating that the wavelength information is described in the bit stream.

16. The encoding device according to claim 15, wherein the circuit does not define the minimum value and the maximum value, and in that case sets the flag information indicating that the wavelength information is not described in the bit stream.

17. The encoding device according to claim 14, wherein the second parameter includes an index value, and a minimum value and a maximum value of the wavelength band corresponding to the index value are defined.

18. The encoding device of claim 17, wherein the circuit does not define the minimum and maximum values, and in that case sets the index values ​​to specific values.

19. The encoding device of claim 17, wherein said circuitry encodes information indicative of said minimum and maximum values ​​into said bitstream, in which case said index values ​​are set to particular values.

20. An encoding device as described in claim 15 or 19, wherein the information indicating the minimum value includes information indicating a mantissa of the minimum value and information indicating an exponent in exponential representation using a predetermined base, and the information indicating the maximum value includes information indicating a mantissa of the maximum value and information indicating an exponent in exponential representation using a predetermined base.

21. The encoding device according to claim 20, wherein the information indicating the mantissa of the minimum value has three or more significant digits in decimal notation, and the information indicating the mantissa of the maximum value has three or more significant digits in decimal notation.

22. The encoding device of claim 20, wherein the circuit: does not define the minimum value, in which case sets information indicating the mantissa of the minimum value to "0"; and does not define the maximum value, in which case sets information indicating the mantissa of the maximum value to "0." 23. The encoding device of claim 15 or 17, wherein the circuit sets the minimum value to be equal to or greater than the shortest wavelength corresponding to the image type indicated in the modality information, and sets the maximum value to be equal to or less than the longest wavelength corresponding to the image type indicated in the modality information.

24. The encoding device of claim 14, wherein, in encoding the first parameter and the second parameter, the circuit encodes the first parameter and the second parameter in a predetermined header area of ​​the bitstream, and the predetermined header area includes a VUI or a SEI.

25. The encoding device according to claim 14, wherein the bit stream has a multi-layer structure including a plurality of image layers, and the circuit, in encoding the image, encodes a plurality of images having different wavelength bands into the plurality of image layers.

26. The encoding device according to claim 25, wherein the circuit generates the bitstream having the multi-layer structure by encoding an output signal from a multispectral camera.

27. A decoding method, comprising: a decoding device obtaining an image and first and second parameters associated with the image from a bitstream; the first parameters including modality information indicating an image type of the image; and the second parameters including wavelength information indicating a wavelength band of optical radiation of the image.

28. An encoding method, comprising: an encoding device encoding an image, a first parameter and a second parameter associated with the image into a bit stream; the first parameter including modality information indicating an image type of the image; and the second parameter including wavelength information indicating a wavelength band of optical radiation of the image.

Images