Decoding device, encoding device, decoding method, and encoding method

Abstract

Provided is a decoding device comprising a circuit and a memory connected to the circuit. The circuit acquires, from a bit stream, an image and a first parameter associated with the image. The first parameter includes modality information indicating an image type of the image. The modality information indicates that the image type is at least any one of a visible light image, an infrared image, an ultraviolet image, undefined, and a preliminary image type for extension.

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method.

[0002] Patent Literature 1 discloses an image processing system according to the background art. The image processing system includes an encoder (encoding device) and a decoder (decoding device). Input images of various modalities are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image analysis task based on the input features, thereby outputting a segmentation map.

[0003] However, the background art does not consider at all the transmission of image modality information from an encoding device to a decoding device.

[0004] US Patent Application Publication No. 2024 / 0046453

[0005] The present disclosure aims to provide a decoding device, an encoding device, a decoding method, and an encoding method that can transmit image modality information from an encoding device to a decoding device, thereby improving the accuracy of task processing execution by the decoding device.

[0006] A decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit acquires an image and first parameters associated with the image from a bitstream, the first parameters including modality information indicating an image type of the image, and the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

[0007] FIG. 1 is a simplified diagram showing the configuration of an image processing system according to an embodiment of the present disclosure. FIG. 2 is a simplified diagram showing the configuration of a circuit included in an encoding device. FIG. 3 is a flowchart showing processing executed by a circuit included in an encoding device. FIG. 4 is a simplified diagram showing the configuration of a bitstream. FIG. 5 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 6 is a diagram showing an example of the correspondence between values ​​of vui_modality_type and image types. FIG. 7 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 8 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 9 is a diagram showing an example of the correspondence between index values ​​and wavelength information. FIG. 10 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 11 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 12 is a diagram showing an example of the correspondence between values ​​of representation type information and interpretations of each value. FIG. 13 is a simplified diagram showing an example of the syntax of VUI parameters. FIG. 14 is a diagram showing an example of the correspondence between values ​​of interpolation information and interpretations of each value. FIG. 15 is a diagram showing an example of the configuration of a bitstream. FIG. 1 is a simplified diagram showing the configuration of a circuit included in a decoding device; FIG. 2 is a flowchart showing processing executed by a circuit included in a decoding device; FIG. 3 is a simplified diagram showing an example of table information; FIG. 4 is a simplified diagram showing an example of table information; FIG. 5 is a block diagram showing an example of the functional configuration of an encoding unit; FIG. 6 is a diagram showing an example of a hierarchical structure of data in a stream; and FIG. 7 is a block diagram showing an example of the functional configuration of a decoding unit.

[0008] (Findings underlying the present disclosure) An image processing system according to the background art includes an encoding device and a decoding device. Input images of various modalities are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image analysis task based on the input features, thereby outputting a segmentation map.

[0009] The task processing performed by the decoding device includes human vision and machine tasks. Human vision is the viewing or viewing of video images by a human, such as an operator or user. Machine tasks include various types of task processing using an AI model, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0010] In the background art, modality information is not transmitted from the encoding device to the decoding device, which may result in the selection of an optimal task process depending on the image type of the input image, resulting in a decrease in the accuracy of the task process execution.

[0011] In order to solve this problem, the inventor discovered that the above problem can be solved by including modality information indicating the image type of the image in a bitstream and transmitting it from the encoding device to the decoding device, and the decoding device then performing task processing based on the modality information, and thus came up with the present disclosure.

[0012] Next, each aspect of the present disclosure will be described.

[0013] A decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit acquires an image and first parameters associated with the image from a bitstream, the first parameters including modality information indicating an image type of the image, and the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

[0014] According to the first aspect, modality information indicating the image type of an image can be included in a bitstream and transmitted from an encoding device to a decoding device, thereby improving the accuracy of task processing by the decoding device. In addition, since the modality information indicating that the image type is undefined or that the image type is a spare image type for extension is included, it is possible to deal with cases where the image characteristics are unknown in the encoding device and the image type cannot be identified, or with future extensions of image types, etc.

[0015] In the decoding device according to the second aspect of the present disclosure, in the first aspect, the circuit acquires a second parameter from the bitstream, the second parameter including wavelength information indicating a wavelength band of optical radiation of the image, and when the modality information indicates that the image type is not defined, the wavelength information may be capable of indicating a wavelength band spanning multiple image types.

[0016] According to the second aspect, wavelength information indicating the wavelength band of the optical radiation of an image can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby further improving the accuracy of task processing by the decoding device. Furthermore, by not defining the image type in the modality information, it is possible to deal with cases where the wavelength band of an image spans multiple image types.

[0017] In the decoding device according to the third aspect of the present disclosure, in the first or second aspect, the circuit acquires a third parameter from the bitstream, the third parameter including flag information indicating whether color information of the image is included or not and the color information, and when the modality information indicates that the image type is an infrared image or an ultraviolet image, the circuit (1) acquires the flag information indicating that the color information is not included, or (2) ignores the color information even if it acquires the flag information indicating that the color information is included.

[0018] According to the third aspect, when the modality information indicates that the image type is an infrared image or an ultraviolet image, the circuit does not acquire color information, thereby making it possible to avoid the execution of unnecessary processing.

[0019] In a decoding device according to a fourth aspect of the present disclosure, in any one of the first to third aspects, the circuit may acquire a fourth parameter from the bit stream, and the fourth parameter may include extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension.

[0020] According to the fourth aspect, extension bit information corresponding to the spare image type for extension can be transmitted from the encoding device to the decoding device, thereby allowing the decoding device to appropriately execute task processing corresponding to the spare image type.

[0021] In the decoding device according to the fifth aspect of the present disclosure, in the fourth aspect, the extended bit information may include information indicating the bit length of additional information related to the auxiliary image type and the additional information having the bit length.

[0022] According to the fifth aspect, the extension bit information can be accurately transmitted from the encoding device to the decoding device.

[0023] In a decoding device according to a sixth aspect of the present disclosure, in any one of the first to fifth aspects, when the modality information indicates that the image type is an infrared image, the circuit obtains representation type information indicating a representation type of infrared radiation intensity from the bit stream, and the representation type information may indicate at least one of the following: that the pixel values ​​of the image indicate intensity values ​​of infrared radiation intensity; and that the pixel values ​​of the image indicate temperature values ​​converted from the intensity values ​​of infrared radiation intensity.

[0024] According to the sixth aspect, the representation type information indicating the representation type of the infrared radiation intensity can be transmitted from the encoding device to the decoding device, thereby enabling the decoding device to appropriately perform task processing using the infrared image.

[0025] In a decoding device according to a seventh aspect of the present disclosure, in the sixth aspect, when the representation type information indicates that the pixel value indicates the temperature value, the circuit may obtain temperature information from the bitstream, the temperature information including the minimum and maximum temperatures of the temperature value.

[0026] According to the seventh aspect, the temperature information includes the minimum temperature and the maximum temperature, so that the temperature information can be accurately transmitted from the encoding device to the decoding device.

[0027] In the decoding device according to the eighth aspect of the present disclosure, in the seventh aspect, the temperature information may include, for each of the minimum temperature and the maximum temperature, information indicating a positive or negative sign, information indicating a mantissa in an exponential representation using a predetermined base, and information indicating an exponent.

[0028] According to the eighth aspect, the minimum temperature and the maximum temperature can be appropriately described in the bit stream using exponential notation.

[0029] A decoding device according to a ninth aspect of the present disclosure, in any one of the first to fifth aspects, may be configured such that the circuit acquires flag information from the bitstream indicating whether or not pixel values ​​of the image are represented using pseudo-color representation, and if the flag information indicates that the pixel values ​​are represented using pseudo-color representation, the circuit acquires generation information from the bitstream for generating table information that associates the pixel values ​​with color codes of the pseudo-color representation, and generates the table information based on the generation information.

[0030] According to the ninth aspect, the decoding device can appropriately generate table information that associates pixel values ​​with color codes in pseudo-color representation, based on generation information acquired from the bitstream.

[0031] A decoding device according to a tenth aspect of the present disclosure is the ninth aspect, wherein the generation information includes a plurality of pairs of the pixel value and the color code, and interpolation information for interpolating values ​​between the plurality of pairs.

[0032] According to the tenth aspect, the decoding device can generate table information appropriately and with high accuracy based on a plurality of pairs of pixel values ​​and color codes and interpolation information for interpolating values ​​between the plurality of pairs.

[0033] In the decoding device of the 11th aspect of the present disclosure, in any one of the 1st to 10th aspects, the circuit, in acquiring the first parameter, may acquire the first parameter from a predetermined header area of ​​the bitstream, and the predetermined header area may include a VUI or an SEI.

[0034] According to the eleventh aspect, the decoding device can easily obtain the first parameter from a predetermined header area of ​​the bitstream.

[0035] A coding device according to a twelfth aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit encodes an image and first parameters associated with the image into a bitstream, the first parameters including modality information indicating an image type of the image, and the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

[0036] According to the twelfth aspect, modality information indicating the image type of an image can be included in a bitstream and transmitted from an encoding device to a decoding device, thereby improving the accuracy of task processing by the decoding device. In addition, since the modality information includes information indicating that the image type is undefined or that the image type is a spare image type for extension, it is possible to deal with cases where the image characteristics are unknown in the encoding device and the image type cannot be identified, or with future extensions of image types, etc.

[0037] In the encoding device according to the thirteenth aspect of the present disclosure, in the twelfth aspect, the circuit encodes a second parameter into the bit stream, the second parameter including wavelength information indicating a wavelength band of optical radiation of the image, and when the modality information indicates that the image type is not defined, the wavelength information may be capable of indicating a wavelength band spanning multiple image types.

[0038] According to the thirteenth aspect, wavelength information indicating the wavelength band of the optical radiation of an image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby further improving the accuracy of task processing by the decoding device. Furthermore, by not defining the image type in the modality information, it is possible to deal with cases where the wavelength band of an image spans multiple image types.

[0039] In the encoding device according to the fourteenth aspect of the present disclosure, in the twelfth or thirteenth aspect, the circuit encodes a third parameter into the bitstream, the third parameter including flag information indicating whether color information of the image is included or not and the color information, and if the image is an infrared image or an ultraviolet image, the circuit preferably includes the flag information indicating that the color information is not included in the third parameter.

[0040] According to the fourteenth aspect, when the image is an infrared image or an ultraviolet image, flag information indicating that no color information is included is included in the third parameter, thereby preventing the decoding device from performing unnecessary processing.

[0041] In the encoding device according to the 15th aspect of the present disclosure, in any one of the 12th to 14th aspects, the circuit may encode a fourth parameter into the bit stream, and the fourth parameter may include extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension.

[0042] According to the fifteenth aspect, extension bit information corresponding to the spare image type for extension can be transmitted from the encoding device to the decoding device, thereby allowing the decoding device to appropriately execute task processing corresponding to the spare image type.

[0043] In the encoding device according to the 16th aspect of the present disclosure, in the 15th aspect, the extension bit information may include information indicating the bit length of additional information related to the auxiliary image type and the additional information having that bit length.

[0044] According to the sixteenth aspect, the extension bit information can be accurately transmitted from the encoding device to the decoding device.

[0045] In the encoding device according to the seventeenth aspect of the present disclosure, in any one of the twelfth to sixteenth aspects, when the image is an infrared image, the circuit encodes representation type information indicating a representation type of infrared radiation intensity into the bit stream, and the representation type information may indicate at least one of the following: that pixel values ​​of the image indicate intensity values ​​of infrared radiation intensity; and that pixel values ​​of the image indicate temperature values ​​converted from intensity values ​​of infrared radiation intensity.

[0046] According to the seventeenth aspect, the representation type information indicating the representation type of the infrared radiation intensity can be transmitted from the encoding device to the decoding device, thereby enabling the decoding device to appropriately perform task processing using the infrared image.

[0047] In the encoding device of the 18th aspect of the present disclosure, in the 17th aspect, when the representation type information indicates that the pixel value indicates the temperature value, the circuit may encode temperature information including the minimum and maximum temperatures of the temperature value into the bitstream.

[0048] According to the eighteenth aspect, the temperature information includes the minimum temperature and the maximum temperature, so that the temperature information can be accurately transmitted from the encoding device to the decoding device.

[0049] In the encoding device according to the 19th aspect of the present disclosure, in the 18th aspect, the temperature information may include, for each of the minimum temperature and the maximum temperature, information indicating a positive or negative sign, information indicating a mantissa in an exponential representation using a predetermined base, and information indicating an exponent.

[0050] According to the nineteenth aspect, the minimum temperature and the maximum temperature can be appropriately described in the bit stream using exponential notation.

[0051] In the encoding device according to the 20th aspect of the present disclosure, in any one of the 12th to 16th aspects, the circuit encodes flag information indicating whether or not pixel values ​​of the image are represented using pseudo-color representation into the bit stream, and if the flag information indicates that the pixel values ​​are represented using pseudo-color representation, the circuit encodes generation information for generating table information that associates the pixel values ​​with color codes of the pseudo-color representation into the bit stream.

[0052] According to the twentieth aspect, the encoding device encodes the generated information into a bit stream, and the decoding device can appropriately generate table information that associates pixel values ​​with color codes of pseudo-color representation based on the generated information obtained from the bit stream.

[0053] In the encoding device according to the 21st aspect of the present disclosure, in the 20th aspect, the generation information may include multiple pairs of the pixel value and the color code, and interpolation information for interpolating values ​​between the multiple pairs.

[0054] According to the 21st aspect, since the generation information includes multiple pairs of pixel values ​​and color codes, and interpolation information for interpolating values ​​between the multiple pairs, the decoding device can generate table information appropriately and with high accuracy based on this information obtained from the bitstream.

[0055] In the encoding device according to the 22nd aspect of the present disclosure, in any one of the 12th to 19th aspects, the circuit, in encoding the first parameter, may encode the first parameter in a predetermined header area of ​​the bitstream, and the predetermined header area may include a VUI or an SEI.

[0056] According to the twenty-second aspect, the decoding device can easily obtain the first parameter from a predetermined header area of ​​the bitstream.

[0057] A decoding method according to a 23rd aspect of the present disclosure includes a decoding device acquiring an image and a first parameter associated with the image from a bitstream, the first parameter including modality information indicating an image type of the image, the modality information indicating that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

[0058] According to the 23rd aspect, modality information indicating the image type of an image can be included in a bitstream and transmitted from an encoding device to a decoding device, thereby improving the accuracy of task processing performed by the decoding device. Furthermore, since the modality information includes information indicating that the image type is undefined or that the image type is a backup image type for extension, it is possible to deal with cases where the image characteristics are unknown in the encoding device and the image type cannot be identified, or with future extensions of image types, etc.

[0059] An encoding method according to a 24th aspect of the present disclosure includes an encoding device encoding an image and a first parameter associated with the image into a bitstream, the first parameter including modality information indicating an image type of the image, the modality information indicating that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

[0060] According to the 24th aspect, modality information indicating the image type of an image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby improving the accuracy of task processing by the decoding device. Furthermore, since the modality information includes information indicating that the image type is undefined or that the image type is a spare image type for extension, it is possible to deal with cases where the image characteristics are unknown in the encoding device and the image type cannot be identified, or with future extensions of image types, etc.

[0061] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0062] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts are described as optional components. Furthermore, in all embodiments, the respective contents can be substituted or combined. Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or by any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0063] 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, and a transmission path NW.

[0064] Image data D1 is input from an external device to the encoding device 1. The external device may include a camera or the like that captures moving images. The external device inputs image data D1 of the captured moving images to the encoding device 1.

[0065] The encoding device 1 generates a bit stream BS based on the image data D1. Here, a bit stream refers to a data string or flow of digital data. The bit stream (or simply stream) may be a single stream, or may be composed of multiple streams divided into multiple layers. Furthermore, the bit stream may be transmitted by serial communication over a single transmission path, or by packet communication over multiple transmission paths. The encoding device 1 transmits the generated bit stream BS to the decoding device 2 via the transmission path NW. The decoding device 2 receives the bit stream BS.

[0066] The decoding device 2 decodes image data D1 from the bitstream BS and performs task processing based on the decoded image data D1. The task processing includes human vision and machine tasks. Human vision is the viewing or viewing of moving images by a human being such as an operator or a user. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using an artificial intelligence (AI) model, which is a machine-learned estimation model. The task processing unit that performs human vision includes a display device such as a liquid crystal display or an organic EL display. The task processing unit that performs machine tasks includes an inference device using AI.

[0067] The transmission path NW may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. The transmission path NW is preferably a private network or the like in which secure communication is ensured by access restrictions.

[0068] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 includes a processor such as a CPU. The memory 12 includes any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data to be processed by the circuit 11 or data in the middle of processing.

[0069] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 includes a processor such as a CPU. The memory 22 includes any recording medium such as a ROM, a RAM, a HDD, an SSD, or a semiconductor memory. The memory 22 stores data to be processed by the circuit 21 or data in the middle of processing.

[0070] 2 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 includes an acquisition unit 31, a setting unit 32, an encoding unit 33, and a transmission unit .

[0071] Next, the encoding unit 33 according to this embodiment will be described. Fig. 21 is a block diagram showing an example of the functional configuration of the encoding unit 33 according to this embodiment. The encoding unit 33 encodes an image in units of blocks.

[0072] 21 , the encoding unit 33 includes a dividing unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Note that the intra prediction unit 124 and the inter prediction unit 126 are configured as part of a prediction processing unit 125.

[0073] For example, the components of the encoding unit 33 shown in FIG. 21 are implemented by the circuit 11 and memory 12 shown in FIG.

[0074] The circuit 11 is configured to include a processor such as a CPU. The circuit 11 may be a dedicated or general-purpose electronic circuit for encoding images, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 11 may fulfill the roles of multiple components of the encoding unit 33 shown in FIG. 21 , excluding the component for storing information.

[0075] The memory 12 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 12 may be externally connected to the circuit 11 or may be built into the circuit 11. The memory 12 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 12 may be a non-volatile memory or a volatile memory.

[0076] The memory 12 may store an image to be encoded, or a stream corresponding to the encoded image, or may store a program for the processor to execute the image encoding process.

[0077] Furthermore, the memory 12 may serve as a component for storing information among the multiple components included in the encoding unit 33 shown in Fig. 21. Specifically, the memory 12 may serve as the block memory 118 and the frame memory 122 shown in Fig. 21. More specifically, the memory 12 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0078] 21 may be omitted from the encoding unit 33, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 21 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0079] FIG. 3 is a flowchart showing the process executed by the circuit 11 included in the encoding device 1.

[0080] First, in step SP11, the acquisition unit 31 acquires image data D11 representing the image Q to be processed, which is input from an external device. The image data D11 corresponds to the image data D1 shown in FIG.

[0081] Next, in step SP12, the setting unit 32 sets parameters P in association with the image Q. The parameters P include a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 includes modality information indicating the image type of the image Q. The second parameter P2 includes wavelength information indicating the wavelength band of the optical radiation of the image Q. The third parameter P3 includes color information of the image Q. The fourth parameter P4 includes extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension. The setting unit 32 inputs data D12 including the setting information of the parameters P to the encoding unit 33.

[0082] The first parameter P1 includes modality information indicating the image type of the image Q. The image type includes, for example, at least one of a visible light image, an infrared image, and an ultraviolet image. The visible light image includes a natural image or an RGB image, and is used to provide detailed color information in human vision or machine tasks. The infrared image includes an image captured using an infrared camera, and is used to capture images in the dark. The ultraviolet image includes an image captured using an ultraviolet camera, and is used to detect scratches on products. The modality information indicates that the image type is at least one of a visible light image, an infrared image, and an ultraviolet image. The modality information also indicates that the image type is at least one of an undefined image type and a spare image type for extension. In other words, the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, an undefined image type, and a spare image type for extension. Note that the modality information may include other image types. The setting unit 32 may set the first parameter P1 by image analysis based on the image data D11, or may set the first parameter P1 based on setting information input by an operator of the encoding device 1.

[0083] Next, in step SP13, the encoding unit 33 encodes the image Q represented by the image data D11 input from the acquisition unit 31 into a bit stream BS.

[0084] Next, in step SP14, the encoding unit 33 encodes the parameters P indicated by the data D12 input from the setting unit 32 into a bit stream BS. Here, encoding the parameters P into the bit stream BS may be rephrased as saving the parameters P in the bit stream BS or storing the parameters P in the bit stream BS. Note that the order of execution of steps SP13 and SP14 may be reversed from the example in FIG. 3, or steps SP13 and SP14 may be executed simultaneously.

[0085] Next, in step SP15, the transmitting unit 34 transmits the bit stream BS input from the encoding unit 33 to the decoding device 2 via the transmission path NW.

[0086] 4 is a simplified diagram showing the structure of a bitstream BS. The bitstream BS has a header area 41 and a payload area 42. The encoding unit 33 stores encoded data of an image Q in the payload area 42, and stores encoded data of a parameter P associated with the image Q in the header area 41.

[0087] The encoding unit 33 may encode the encoded data of the parameter P in a predetermined area 43 in the header area 41. The predetermined area 43 may be VUI (video usability information) or SEI (supplemental enhancement information). Although an example using VUI parameters will be described in this specification, when describing in SEI, the VUI parameters may be read as SEI parameters. However, the predetermined area 43 is not limited to VUI or SEI, and may also be VPS, SPS, PPS, PH, SH, APS, a tile header, a system layer header, or the like.

[0088] 22 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. As shown in (A) of FIG. 22, the video sequence includes, for example, a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), supplemental enhancement information (SEI), and multiple pictures.

[0089] In a video composed of multiple layers, the VPS includes coding parameters common to multiple layers and coding parameters related to multiple layers or individual layers included in the video.

[0090] The SPS includes parameters used for a sequence, i.e., encoding parameters that the decoding device 2 refers to in order to decode the sequence. The encoding parameters may indicate, for example, the width or height of a picture. Note that there may be multiple SPSs.

[0091] The PPS includes parameters used for a picture, i.e., encoding parameters that the decoding device 2 references to decode each picture in a sequence. The encoding parameters may include, for example, a reference value of the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Note that there may be multiple PPSs. The SPS and PPS may also be simply referred to as parameter sets.

[0092] As shown in (B) of Fig. 22, a picture includes a picture header and one or more slices. The picture header includes coding parameters that are referenced by the decoding device 2 to decode the one or more slices.

[0093] As shown in Fig. 22C, a slice includes a slice header and one or more bricks. The slice header includes coding parameters that are referenced by the decoding device 2 to decode the one or more bricks.

[0094] As shown in FIG. 22(D), a brick includes one or more CTUs (Coding Tree Units).

[0095] Note that a picture may not contain slices, but may contain tile groups instead, where each tile group contains one or more tiles. Also, a brick may contain slices.

[0096] A CTU is also called a superblock or a basic division unit. As shown in (E) of Fig. 22, a CTU includes a CTU header and one or more coding units (CUs). The CTU header includes coding parameters that the decoding device 2 references to decode the one or more CUs.

[0097] A CU may be divided into multiple smaller CUs. Furthermore, as shown in (F) of FIG. 22 , a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting a CU. The residual coefficient information is information indicating a prediction residual. A CU is basically the same as a PU (Prediction Unit) or a TU (Transform Unit), but may include multiple TUs smaller than a CU. A CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is a fixed unit that can be processed in one stage, for example, when performing pipeline processing in hardware.

[0098] Note that a stream may not have some of the layers among the layers shown in Fig. 22. Furthermore, the order of these layers may be changed, and any layer may be replaced with another layer.

[0099] A picture currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current picture. If the processing is encoding, the current picture is synonymous with a picture to be encoded, and if the processing is decoding, the current picture is synonymous with a picture to be decoded. Furthermore, a block (CU or a block of a CU) currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current block. If the processing is encoding, the current block is synonymous with a block to be encoded, and if the processing is decoding, the current block is synonymous with a block to be decoded.

[0100] FIG. 5 is a simplified diagram showing an example of the syntax of VUI parameters including a first parameter P1 and a third parameter P3.

[0101] The first parameter P1 includes flag information (vui_modality_info_present_flag) indicating whether modality information is described in the bitstream BS. When the value of vui_modality_info_present_flag is 1, it indicates that modality information is present in the VUI parameters, and when the value of vui_modality_info_present_flag is 0, it indicates that modality information is not present in the VUI parameters. When vui_modality_info_present_flag indicates that modality information is described in the bitstream BS, the modality information is represented as the value of the identifier vui_modality_type included in the VUI parameters.

[0102] FIG. 6 is a diagram showing an example of the correspondence between the value of vui_modality_type and the image type. As shown in FIG. 6, a value of 0 for vui_modality_type indicates that the image type of image Q is not defined (i.e., undefined). A value of 1 for vui_modality_type indicates that the image type of image Q is a visible light image. A value of 2 for vui_modality_type indicates that the image type of image Q is an infrared image. A value of 3 for vui_modality_type indicates that the image type of image Q is an ultraviolet image. Other values ​​of vui_modality_type (4-255 in this example) indicate a spare area reserved for future extended use. Note that the number of image types defined in FIG. 6 may be increased depending on the image Q to be processed. Furthermore, the absence of vui_modality_type may mean that the value of vui_modality_type is 0. Furthermore, when the value of vui_modality_type is 0, wavelength information indicating a wide wavelength band spanning multiple image types, such as a hyperspectral image, may be defined by user settings or the like.

[0103] As shown in Fig. 5, the third parameter P3 includes flag information (vui_colour_description_present_flag) indicating whether color information is described in the bitstream BS. A value of 1 in vui_colour_description_present_flag indicates that color information is present in the VUI parameters, and a value of 0 in vui_colour_description_present_flag indicates that color information is not present in the VUI parameters. Although not shown, the color information includes vui_colour_primaries, vui_transfer_characteristics, vui_matrix_coeffs, etc. Note that if the image type is an infrared image or an ultraviolet image, the third parameter P3 does not need to include color information. In the encoding process, if the image type indicated by the modality information is a visible light image, the encoding device 1 sets the value of vui_colour_description_present_flag to 1 and describes color information in the VUI parameters, whereas if the image type indicated by the modality information is an infrared image or an ultraviolet image, the encoding device 1 sets the value of vui_colour_description_present_flag to 0 and does not describe color information in the VUI parameters. Furthermore, in the decoding process, if the image type indicated by the modality information is a visible light image, the decoding device 2 acquires color information from the bitstream BS, whereas if the image type indicated by the modality information is an infrared image or an ultraviolet image, it is not necessary to acquire color information from the bitstream BS, or even if color information is acquired from the bitstream BS, the value of the color information may be ignored.

[0104] FIG. 7 is a simplified diagram showing an example of the syntax of VUI parameters including the fourth parameter P4.

[0105] The fourth parameter P4 includes extension bit information corresponding to the reserved image type when the modality information indicates that the image type is a reserved image type for extension. The extension bit information includes information (vui_modality_type_extension_bits) indicating the bit length of additional information required for the reserved image type and additional information (vui_reserved_modality_type_extension) having the bit length. A value of vui_modality_type_extension_bits of 0 indicates that vui_reserved_modality_type_extension is not described in the bitstream BS. On the other hand, a value of vui_modality_type_extension_bits greater than 0 indicates that vui_reserved_modality_type_extension is described in the bitstream BS and its bit length. The maximum value of vui_modality_type_extension_bits can be set arbitrarily, for example, to 2048.

[0106] 8 is a simplified diagram showing a first example of the syntax of VUI parameters including a second parameter P2, which includes wavelength information indicating the wavelength band of the light radiation of the image Q.

[0107] 8, the second parameter P2 includes flag information (vui_spectrum_range_present_flag) indicating whether wavelength information is described in the bitstream BS. When the value of vui_spectrum_range_present_flag is 0, it indicates that the spectral band of the optical radiation wavelength represented by the CLVS (Coded Layer Video Sequence) image does not exist in the VUI parameters, and when the value of vui_spectrum_range_present_flag is 1, it indicates that the spectral band of the optical radiation wavelength represented by the CLVS image exists in the VUI parameters.

[0108] When the value of vui_spectrum_range_present_flag is 1, the wavelength information includes information indicating the minimum value (minimum wavelength) and maximum value (maximum wavelength) of the wavelength band of image Q. Both the minimum wavelength and the maximum wavelength are wavelengths in meters.

[0109] The information indicating the minimum value of the wavelength band includes information indicating the mantissa of the minimum value in exponential notation using a predetermined base (for example, 10) (vui_min_wavelength_mantissa) and information indicating the value obtained by adding 15 to the exponent of the minimum value (vui_min_wavelength_exponent_plus15). The minimum value Min_Wavelength of the wavelength band is expressed as Min_Wavelength = vui_min_wavelength_mantissa × 10 vui_min_wavelength_exponent_plus15 - 15. As a result, in the example shown in FIG. 8, 10 －１５ ~10 ＋１６ A minimum value can be specified within the range of (m). Here, if vui_min_wavelength_mantissa is 0 or does not exist, it means that the minimum value of the wavelength band is unknown or unspecified and is not defined, or that it is defined using a method other than exponential notation. In this case, the decoding device 2 does not need to acquire vui_min_wavelength_exponent_plus15 from the bitstream BS during the decoding process, and even if it acquires vui_min_wavelength_exponent_plus15 from the bitstream BS, it may ignore the value.

[0110] The information indicating the maximum value of the wavelength band includes information indicating the mantissa of the maximum value in exponential notation using a predetermined base (for example, 10) (vui_max_wavelength_mantissa) and information indicating the value obtained by adding 15 to the exponent of the maximum value (vui_max_wavelength_exponent_plus15). The maximum value of the wavelength band, Max_Wavelength, is expressed as Max_Wavelength = vui_max_wavelength_mantissa × 10 vui_max_wavelength_exponent_plus15 - 15. As a result, in the example shown in FIG. 8, 10 －１５ ~10 ＋１６ The maximum value can be specified in the range of (m). Here, if vui_max_wavelength_mantissa is 0 or does not exist, it means that the maximum value of the wavelength band is unknown or unspecified and therefore not defined, or that it is defined using a method other than exponential notation. In this case, the decoding device 2 does not need to acquire vui_max_wavelength_exponent_plus15 from the bitstream BS, or even if it acquires vui_max_wavelength_exponent_plus15 from the bitstream BS, it may ignore the value.

[0111] Both vui_min_wavelength_mantissa and vui_max_wavelength_mantissa have three or more significant decimal digits. In the example shown in FIG. 8, both vui_min_wavelength_mantissa and vui_max_wavelength_mantissa have a binary bit length of 10 bits, but may have any bit length of 11 bits or more. Similarly, in the example shown in FIG. 8, both vui_min_wavelength_exponent_plus15 and vui_max_wavelength_exponent_plus15 have a binary bit length of 5 bits, but may have any bit length of 6 bits or more. Furthermore, the value added to the exponent is not limited to "+15", and a syntax for adding another fixed value may be used.

[0112] In the encoding device 1, the minimum value of the wavelength band indicated by vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15 is set to be equal to or greater than the shortest wavelength corresponding to the image type indicated by the modality information. Similarly, in the encoding device 1, the maximum value of the wavelength band indicated by vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15 is set to be equal to or less than the longest wavelength corresponding to the image type indicated by the modality information.

[0113] FIG. 9 is a simplified diagram showing a second example of the syntax of the VUI parameters including the second parameter P2.

[0114] In the example shown in FIG. 9, the second parameter P2 includes information (vui_spectrum_range) indicating an index value of the wavelength information.

[0115] 10 is a diagram showing an example of the correspondence between index values ​​and wavelength information. A minimum value (minimum wavelength) and a maximum value (maximum wavelength) of a wavelength band are predefined corresponding to each of a plurality of index values, and are shared by the encoding device 1 and the decoding device 2. In the example shown in FIG. 10, index values ​​2 to 6 correspond to the ultraviolet wavelength region, index value 7 corresponds to the visible light wavelength region, and index values ​​8 to 10 correspond to the infrared wavelength region.

[0116] For example, if the index value is 2, the minimum wavelength is 1*10 －９ (m), the maximum wavelength is 1*10 －７ (m), and if the index value is, for example, 3, the minimum wavelength is 1*10 －７ (m), the maximum wavelength is 19*10 －８ (m) is defined as:

[0117] Furthermore, when the index value is a certain value (0 in the example shown in FIG. 10), it means that the minimum wavelength and the maximum wavelength are not defined.

[0118] If the index value is another specific value (1 in the example shown in FIG. 10), this means that the desired minimum and maximum wavelengths are defined by user settings or the like.

[0119] Note that other index values ​​(11-255 in this example) indicate spare areas reserved for future extended use.

[0120] In the encoding device 1, the index value is set according to the wavelength band corresponding to the image type indicated by the modality information. For example, if vui_modality_type is 1 (visible light image), vui_spectrum_range is set to 0, 1, or 7. If vui_modality_type is 2 (infrared image), vui_spectrum_range is set to 0, 1, or any of 8 to 10. If vui_modality_type is 3 (ultraviolet image), vui_spectrum_range is set to 0, 1, or any of 2 to 6. In other words, the encoding device 1 sets an appropriate index value according to the image type, allowing the decoding device 2 to identify an appropriate wavelength band based on the index value and decode the image.

[0121] FIG. 11 is a simplified diagram showing a third example of the syntax of VUI parameters including the second parameter P2.

[0122] In the example shown in Figure 9, if the index value of vui_spectrum_range is set to 1 (user-defined), the minimum value of the wavelength band may be defined by vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15, as in the example shown in Figure 8, and the maximum value of the wavelength band may be defined by vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15.

[0123] FIG. 12 is a simplified diagram showing a first modified example of the syntax of VUI parameters.

[0124] If the image type indicated by the modality information is an infrared image, the encoding device 1 may further encode representation type information (vui_infrared_radiance_representation_type) indicating the representation type of infrared radiance into the bitstream BS.

[0125] FIG. 13 is a diagram showing an example of the correspondence between values ​​of the expression type information and interpretations of each value.

[0126] If vui_infrared_radiance_representation_type is 0 or does not exist, it means that the representation type of the infrared radiation intensity is unknown or unspecified and the representation type is not defined, or is defined by a method other than intensity value representation or temperature value representation.

[0127] If the value of vui_infrared_radiance_representation_type is 1, this means that the representation type of infrared radiation intensity is intensity value representation. In the intensity value representation, each pixel value of image Q indicates an intensity value of infrared radiation intensity.

[0128] If the value of vui_infrared_radiance_representation_type is 2, this means that the representation type of the infrared radiation intensity is temperature value representation. In the temperature value representation, each pixel value of image Q indicates a temperature value converted from the intensity value of the infrared radiation intensity.

[0129] Any other value of vui_infrared_radiance_representation_type (3 in this example) indicates a spare area reserved for future expansion.

[0130] 12, when the value of vui_infrared_radiance_representation_type is 2 (i.e., temperature value representation), the encoding device 1 describes temperature information in the VUI parameters. The temperature information includes the minimum and maximum temperatures of the temperature value, which is a floating-point value.

[0131] The information indicating the minimum temperature includes information (vui_min_temperature_sign) indicating the positive or negative sign of the minimum temperature in exponential notation using a predetermined base (for example, 2), information indicating the exponent of the minimum temperature (vui_min_temperature_exponent), information indicating the value obtained by subtracting 1 from the bit length of the mantissa of the minimum temperature (vui_min_temperature_mantissa_len_minus1), and information indicating the mantissa of the minimum temperature (vui_min_temperature_mantissa). Note that vui_min_temperature_mantissa_len_minus1 may be omitted by setting the bit length of the mantissa of the minimum temperature to a fixed value.

[0132] If the value of vui_min_temperature_sign is 0, it indicates that the sign of the minimum temperature is positive, and if the value of vui_min_temperature_sign is 1, it indicates that the sign of the minimum temperature is negative.

[0133] The information indicating the maximum temperature includes information (vui_max_temperature_sign) indicating the sign of the maximum temperature in exponential notation using a predetermined base (for example, 2), information (vui_max_temperature_exponent) indicating the exponent of the maximum temperature, information (vui_max_temperature_mantissa_len_minus1) indicating the value obtained by subtracting 1 from the bit length of the mantissa of the maximum temperature, and information (vui_max_temperature_mantissa) indicating the mantissa of the maximum temperature. Note that vui_max_temperature_mantissa_len_minus1 may be omitted by setting the bit length of the mantissa of the maximum temperature to a fixed value.

[0134] If the value of vui_max_temperature_sign is 0, it indicates that the sign of the maximum temperature is positive, and if the value of vui_max_temperature_sign is 1, it indicates that the sign of the maximum temperature is negative.

[0135] FIG. 14 is a simplified diagram showing a second modified example of the syntax of the VUI parameters.

[0136] The encoding device 1 may further encode flag information (vui_false_colour_representation_present_flag) indicating whether or not pixel values ​​of the image Q are represented using a false colour representation into the bitstream BS.

[0137] If the value of vui_false_colour_representation_present_flag is 1, it indicates that the pixel values ​​of image Q are represented using pseudo-colour representation, and if the value of vui_false_colour_representation_present_flag is 0, it indicates that the pixel values ​​of image Q are not represented using pseudo-colour representation.

[0138] When the pixel values ​​of image Q are expressed using pseudo-color representation, generation information for generating table information that associates each pixel value of image Q with a color code of the pseudo-color representation is described in the VUI parameters.

[0139] The generation information includes information (vui_num_hex_colour_code_minus2) indicating the value obtained by subtracting 2 from the number of color codes encoded in the bitstream BS, information (vui_num_value_minus2) indicating the value obtained by subtracting 2 from the number of pairs of pixel values ​​and color codes, interpolation information (vui_colour_interpolation_type) for interpolating values ​​between the pairs, and information (vui_hex_colour_code[i]) indicating the i-th hexadecimal color code. Each hexadecimal color code is 24 bits long, with the first 8 bits representing the depth of red, the middle 8 bits representing the depth of green, and the last 8 bits representing the depth of blue. For example, if vui_hex_colour_code is 000000, it indicates black (RGB: 0,0,0), and if vui_hex_colour_code is FFFFFF, it indicates white (RGB: 255,255,255).

[0140] For example, if the value of vui_num_hex_colour_code_minus2 is 0, it means that the number of colour codes coded in the bitstream BS is 2.

[0141] For example, if the value of vui_num_value_minus2 is 4, this means that the number of pairs of pixel values ​​and color codes is 6. If the number of color codes coded in the bitstream is 2 and the number of pairs is 6, this means that the range of color components is divided into 5 equal sections based on the values ​​of the two color codes, and the values ​​of the middle 4 pairs of color codes are generated by interpolation, thereby generating table information containing a total of 6 pairs of combinations of pixel values ​​and color codes.

[0142] FIG. 15 is a diagram showing an example of the correspondence between the values ​​of the interpolation information and the interpretation of each value.

[0143] If vui_colour_interpolation_type is 0 or not present, it means that the interpolation type is unknown or unspecified and not defined, or is defined by a method other than linear or bilinear interpolation.

[0144] If vui_colour_interpolation_type is 1, it means that the interpolation type is linear.

[0145] If vui_colour_interpolation_type is 2, it means that the interpolation type is bilinear interpolation.

[0146] Other values ​​of vui_colour_interpolation_type (3-15 in this example) indicate a spare area reserved for future expansion.

[0147] FIG. 16 is a diagram showing an example of the structure of the bit stream BS.

[0148] The bitstream BS is a multi-layer image layer L １ ~L ｍ16 shows only one access unit. An access unit is the minimum processing unit of a time attribute, and corresponds to, for example, one frame of a video image. A bit stream BS is configured to include multiple access units that are consecutive in time.

[0149] The first image layer L, which is the lowest layer １ The payload area 42 of the image Q Ｌ１ The coded data of the second image layer L is stored. ２ The payload area 42 of the image Q Ｌ２ Similarly, the encoded data of the m-th image layer L ｍ The payload area 42 of the image Q Ｌｍ The encoded data is stored.

[0150] Image Q Ｌ１ ~Q Ｌｍ may have different image types or wavelength bands. For example, in a three-layer multi-layer structure with m=3, image Q Ｌ１ is the visible light image, and image Q Ｌ２ is the infrared image, and image Q Ｌ３ For example, when encoding an output signal from a multispectral camera, the encoding device 1 divides the spectral range into m regions and generates an image Q for each spectral range. Ｌ１ ~Q Ｌｍ may be coded as a multi-layered bitstream BS.

[0151] Image Q Ｌ１ ~Q Ｌｍ If there is a correlation between the image layers L １ ~L ｍ Images may be referenced between images. Ｌ１ ~Q Ｌｍ If there is no correlation between the image layers L １ ~L ｍ Images do not need to be referenced between them.

[0152] Image layer L １ The header area 41 of the image Q Ｌ１The parameter P associated with Ｌ１ The encoded data of the parameter P is stored. Ｌ１ is image Q Ｌ１ This includes modality information and wavelength information.

[0153] Image layer L ２ The header area 41 of the image Q Ｌ２ The parameter P associated with Ｌ２ The encoded data of the parameter P is stored. Ｌ２ is image Q Ｌ２ This includes modality information and wavelength information.

[0154] Image layer L ｍ The header area 41 of the image Q Ｌｍ The parameter P associated with Ｌｍ The encoded data of the parameter P is stored. Ｌｍ is image Q Ｌｍ This includes modality information and wavelength information.

[0155] 17 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 includes a receiving unit 51, a decoding unit 52, a switching unit 53, and a plurality of n task processing units 54 (n is a natural number of 2 or more). １ ~54 ｎ The task processing unit 54 １ ~54 ｎ The tasks performed by include human vision and machine tasks, including various types of tasks using AI models, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0156] Next, the decoding unit 52 according to this embodiment will be described. Fig. 23 is a block diagram showing an example of the functional configuration of the decoding unit 52 according to this embodiment. The decoding unit 52 decodes a stream, which is an encoded image, in units of blocks.

[0157] 23 , the decoding unit 52 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a partition determination unit 224. Note that the intra prediction unit 216 and the inter prediction unit 218 are configured as part of a prediction processing unit 215.

[0158] For example, the components of the decoding unit 52 shown in FIG. 23 are implemented by the circuit 21 and memory 22 shown in FIG.

[0159] The circuit 21 is configured to include a processor such as a CPU. The circuit 21 may be a dedicated or general-purpose electronic circuit for decoding streams, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 21 may fulfill the roles of multiple components of the decoding unit 52 shown in FIG. 23 , excluding the component for storing information.

[0160] The memory 22 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 22 may be externally connected to the circuit 21 or may be built into the circuit 21. The memory 22 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 22 may be a non-volatile memory or a volatile memory.

[0161] The memory 22 may store a stream to be decoded or a decoded image, or may store a program for the processor to execute the decoding process of the stream.

[0162] Furthermore, the memory 22 may serve as a component for storing information among the multiple components included in the decoding unit 52 shown in Fig. 23. Specifically, the memory 22 may serve as the block memory 210 and the frame memory 214 shown in Fig. 23. More specifically, the memory 22 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0163] 23 may be omitted from the decoding unit 52, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 23 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0164] The inverse quantization unit 204, inverse transform unit 206, addition unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter 212 included in the decoding unit 52 shown in Figure 23 perform processing similar to that of the inverse quantization unit 112, inverse transform unit 114, addition unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter 120 included in the encoding unit 33 shown in Figure 21.

[0165] FIG. 18 is a flowchart showing the processing executed by the circuit 21 included in the decoding device 2.

[0166] First, in step SP21, the receiving unit 51 receives the bit stream BS transmitted by the encoding device 1 from the transmission path NW.

[0167] Next, in step SP22, the decoding unit 52 obtains image Q by decoding it from the payload area 42 of the bitstream BS input from the receiving unit 51. Note that the decoding may include extraction. The decoding unit 52 outputs image data D21 of image Q. The image data D21 corresponds to the image data D11 shown in FIG. 2.

[0168] As shown in FIG. 16, the bitstream BS includes multiple image layers L １ ~L ｍ In the case where the image has a multi-layer structure including a plurality of images Q having different image types or wavelength bands, the decoding unit 52 Ｌ１ ~Q Ｌｍ multiple image layers L １ ~L ｍ You can also get it from.

[0169] Next, in step SP23, the decoding unit 52 obtains the parameters P by decoding them from the header region 41 (or a predetermined region 43 within the header region 41) of the bit stream BS input from the receiving unit 51. As described above, the parameters P include a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 includes modality information indicating the image type of the image Q. The second parameter P2 includes wavelength information indicating the wavelength band of the optical radiation of the image Q. The third parameter P3 includes color information of the image Q. The fourth parameter P4 includes extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension. Note that the order of execution of steps SP22 and SP23 may be reversed from the example in FIG. 18, or they may be performed simultaneously.

[0170] 12 is included in the parameter P, and if the modality information indicates that the image type is an infrared image (i.e., the value of vui_modality_type is 2), the decoding unit 52 further acquires the above-mentioned expression type information indicating the expression type of infrared radiation intensity from the bitstream BS. Also, if the expression type information indicates that the pixel value indicates a temperature value (i.e., the expression is a temperature value), the decoding unit 52 further acquires temperature information including the minimum and maximum temperatures of the temperature values ​​from the bitstream BS.

[0171] 14 is included in the parameter P, and if the flag information indicates that the pixel values ​​are represented using pseudo-color representation (i.e., if the value of vui_false_colour_representation_present_flag is 1), the decoding unit 52 further acquires from the bitstream BS the generation information for generating table information that associates each pixel value of the image Q with a color code of the pseudo-color representation. The decoding unit 52 also generates the table information based on the acquired generation information.

[0172] 19 and 20 are simplified diagrams showing examples of table information.

[0173] 19 and 20, the number of pairs of pixel values ​​and color codes is 6 (i.e., the value of vui_num_value_minus2 is 4), the image type is an infrared image, and the interpolation type may be linear or bilinear.

[0174] In the example shown in FIG. 19, the table information indicates the correspondence between hexadecimal color codes, RGB color codes, and gray shades.

[0175] 20, the table information shows the correspondence between hex color codes, shades of gray, and temperatures, with shades of gray being mapped to temperatures. For example, black represents a temperature of 300K, gray represents a temperature of 330K, and white represents a temperature of 350K.

[0176] Next, in step SP24, the switching unit 53 switches the task processing unit 54 based on the modality information input from the decoding unit 52. １ ~54 ｎ The switching unit 53 switches between a plurality of image types and a plurality of task processing units 54. １ ~54 ｎ The switching unit 53 holds table information (not shown) in which a correspondence relationship between the task and the task is preset. １ ~54 ｎ Among these, one task processing unit 54 corresponding to the image type indicated by the modality information is selected.１ ~54 ｎ Select .

[0177] Referring to FIG. 18, next, in step SP25, the one task processing unit 54 selected in step SP24 １ ~54 ｎ executes task processing based on image data D21 input from the decoding unit 52 via the switching unit 53.

[0178] According to the encoding device 1 and decoding device 2 of this embodiment, modality information indicating the image type of image Q and wavelength information indicating the wavelength band of the optical radiation of image Q can be included in the bit stream BS and transmitted from the encoding device 1 to the decoding device 2, thereby making it possible to improve the accuracy of task processing execution by the decoding device 2.

[0179] Furthermore, according to the encoding device 1 and the decoding device 2 of this embodiment, the modality information indicates that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension. Therefore, since the modality information includes modality information indicating that the image type is undefined or that the image type is a spare image type for extension, it is possible to deal with cases where the image characteristics are unknown in the encoding device 1 and the image type cannot be identified, or with future extensions of the image type, etc.

[0180] The present disclosure is particularly useful when applied to an image processing system that includes an encoding device that encodes an image into a bitstream and transmits the bitstream, and a decoding device that decodes an image from the received bitstream.

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, the circuit obtaining from a bitstream an image and first parameters associated with the image, the first parameters including modality information indicating an image type of the image, the modality information indicating the image type being at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

2. The decoding device of claim 1, wherein the circuitry obtains second parameters from the bit stream, the second parameters including wavelength information indicating a wavelength band of optical radiation of the image, and when the modality information indicates that the image type is undefined, the wavelength information is capable of indicating a wavelength band spanning multiple image types.

3. The decoding device of claim 1, wherein the circuit acquires a third parameter from the bit stream, the third parameter including flag information indicating whether or not color information of the image is included and the color information, and when the modality information indicates that the image type is an infrared image or an ultraviolet image, the circuit (1) acquires the flag information indicating that the color information is not included, or (2) ignores the color information even if it acquires the flag information indicating that the color information is included.

4. The decoding device of claim 1, wherein the circuit acquires a fourth parameter from the bit stream, and the fourth parameter includes extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension.

5. The decoding device according to claim 4, wherein the extended bit information includes information indicating a bit length of additional information relating to the spare image type, and the additional information having the bit length.

6. The decoding device of claim 1, wherein when the modality information indicates that the image type is an infrared image, the circuit obtains representation type information indicating a representation type of infrared radiation intensity from the bit stream, and the representation type information indicates at least one of: that the pixel values ​​of the image indicate intensity values ​​of infrared radiation intensity; and that the pixel values ​​of the image indicate temperature values ​​converted from the intensity values ​​of infrared radiation intensity.

7. The decoding device of claim 6, wherein, if the representation type information indicates that the pixel value indicates the temperature value, the circuitry obtains temperature information from the bitstream, the temperature information including a minimum temperature and a maximum temperature of the temperature values.

8. The decoding device according to claim 7, wherein the temperature information includes, for each of the minimum temperature and the maximum temperature, information indicating a positive or negative sign, information indicating a mantissa in an exponential representation using a predetermined base, and information indicating an exponent.

9. The decoding device of claim 1, wherein the circuit obtains flag information from the bit stream indicating whether or not pixel values ​​of the image are represented using pseudo-color representation, and if the flag information indicates that the pixel values ​​are represented using pseudo-color representation, the circuit obtains generation information from the bit stream for generating table information that associates the pixel values ​​with color codes of the pseudo-color representation, and generates the table information based on the generation information.

10. The decoding device according to claim 9, wherein the generation information includes a plurality of pairs of the pixel values ​​and the color codes, and interpolation information for interpolating values ​​between the plurality of pairs.

11. The decoding device of claim 1, wherein, in obtaining the first parameter, the circuit obtains the first parameter from a predetermined header area of ​​the bitstream, and the predetermined header area includes a VUI or a SEI.

12. An encoding device comprising: a circuit; and a memory connected to the circuit, the circuit encoding an image and first parameters associated with the image into a bitstream, the first parameters including modality information indicating an image type of the image, the modality information indicating the image type being at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

13. The encoding device of claim 12, wherein the circuit encodes a second parameter into the bit stream, the second parameter including wavelength information indicating a wavelength band of optical radiation of the image, and when the modality information indicates that the image type is undefined, the wavelength information is capable of indicating a wavelength band spanning multiple image types.

14. The encoding device of claim 12, wherein the circuit encodes a third parameter into the bit stream, the third parameter including flag information indicating whether color information of the image is included or not and the color information, and when the image is an infrared image or an ultraviolet image, the circuit includes in the third parameter the flag information indicating that the color information is not included.

15. The encoding device of claim 12, wherein the circuit encodes a fourth parameter into the bit stream, the fourth parameter including extension bit information corresponding to the spare image type when the modality information indicates that the image type is a spare image type for extension.

16. The encoding device according to claim 15, wherein the extension bit information includes: information indicating a bit length of additional information relating to the spare image type; and the additional information having the bit length.

17. The encoding device of claim 12, wherein, when the image is an infrared image, the circuit encodes representation type information indicating a representation type of infrared radiation intensity into the bit stream, the representation type information indicating at least one of: that pixel values ​​of the image indicate intensity values ​​of infrared radiation intensity; and that pixel values ​​of the image indicate temperature values ​​converted from intensity values ​​of infrared radiation intensity.

18. The encoding device of claim 17, wherein if the representation type information indicates that the pixel value represents the temperature value, the circuit encodes temperature information including a minimum temperature and a maximum temperature of the temperature values ​​into the bitstream.

19. The encoding device according to claim 18, wherein the temperature information includes, for each of the minimum temperature and the maximum temperature, information indicating a positive or negative sign, information indicating a mantissa and information indicating an exponent in an exponential representation using a predetermined base.

20. The encoding device of claim 12, wherein the circuit encodes, into the bit stream, flag information indicating whether or not pixel values ​​of the image are represented using a pseudo-color representation, and if the flag information indicates that the pixel values ​​are represented using a pseudo-color representation, the circuit encodes, into the bit stream, generation information for generating table information that associates the pixel values ​​with color codes of the pseudo-color representation.

21. The encoding device according to claim 20, wherein the generation information includes a plurality of pairs of the pixel value and the color code, and interpolation information for interpolating values ​​between the plurality of pairs.

22. The encoding device according to claim 12, wherein, in encoding the first parameter, the circuit encodes the first parameter in a predetermined header area of ​​the bitstream, the predetermined header area including a VUI or a SEI.

23. A decoding method, comprising: a decoding device obtaining an image and first parameters associated with the image from a bitstream; the first parameters including modality information indicating an image type of the image; and the modality information indicating that the image type is at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

24. An encoding method, in which an encoding device encodes an image and first parameters associated with the image into a bitstream, the first parameters including modality information indicating an image type of the image, the modality information indicating the image type being at least one of a visible light image, an infrared image, an ultraviolet image, undefined, and a spare image type for extension.

Images