Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device comprises a circuit and a memory connected to the circuit. The circuit acquires, from a bit stream, an image and a parameter associated with the image, and the parameter includes modality information indicating the image class of the image.

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method.

[0002] Patent Literature 1 discloses an image processing system according to the background art. The image processing system includes an encoder (encoding device) and a decoder (decoding device). Input images of various modalities are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image analysis task based on the input features, thereby outputting a segmentation map.

[0003] However, the background art does not consider at all the transmission of image modality information from an encoding device to a decoding device.

[0004] US Patent Application Publication No. 2024 / 0046453

[0005] The present disclosure aims to provide a decoding device, an encoding device, a decoding method, and an encoding method that can transmit image modality information from an encoding device to a decoding device, thereby improving the accuracy of task processing execution by the decoding device.

[0006] A decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit acquires an image and parameters associated with the image from a bitstream, and the parameters include modality information indicating the image type of the image.

[0007] 1 is a diagram showing, in a simplified form, the configuration of an image processing system according to an embodiment of the present disclosure; FIG. 1 is a diagram showing, in a simplified form, the configuration of a circuit included in an encoding device; FIG. 2 is a flowchart showing processing performed by a circuit included in an encoding device; FIG. 3 is a diagram showing, in a simplified form, the configuration of a bitstream; FIG. 4 is a diagram showing a first example of syntax related to setting modality information; FIG. 5 is a diagram showing a first example of syntax related to setting modality information; FIG. 6 is a diagram showing a second example of syntax related to setting modality information; FIG. 7 is a diagram showing, in a simplified form, the configuration of a circuit included in a decoding device; FIG. 8 is a flowchart showing processing performed by a circuit included in a decoding device; FIG. 9 is a diagram showing, in a simplified form, the configuration of a circuit included in an encoding device; FIG. 10 is a diagram showing, in a simplified form, a first example of the configuration of a circuit included in a decoding device; FIG. 11 is a diagram showing, in a simplified form, a second example of the configuration of a circuit included in a decoding device; 1 is a diagram showing an example of syntax related to setting modality information. FIG. 2 is a diagram showing an example of syntax related to setting modality information. FIG. 3 is a diagram showing a second example of a multi-layer configuration. FIG. 4 is a diagram showing a simplified configuration of a circuit included in a decoding device. FIG. 5 is a diagram showing a simplified configuration of a bit stream. FIG. 6 is a diagram showing an example of syntax related to setting modality information. FIG. 7 is a diagram showing an example of syntax related to setting modality information. FIG. 8 is a diagram showing an example of a configuration of an image made up of a plurality of components. FIG. 9 is a diagram showing an example of syntax related to setting modality information. FIG. 10 is a diagram showing a simplified configuration of a circuit included in a decoding device. A block diagram showing an example of the functional configuration of an encoding unit. A diagram showing an example of a hierarchical structure of data in a stream. A block diagram showing an example of the functional configuration of a decoding unit.

[0008] (Findings underlying the present disclosure) An image processing system according to the background art includes an encoding device and a decoding device. Input images of various modalities are input to the encoding device. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image analysis task based on the input features, thereby outputting a segmentation map.

[0009] The task processing performed by the decoding device includes human vision and machine tasks. Human vision is the viewing or viewing of video images by a human, such as an operator or user. Machine tasks include various types of task processing using an AI model, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0010] In the background art, modality information is not transmitted from an encoding device to a decoding device, and therefore, an optimal task process or an optimal AI model may not be selected according to the image type of the input image, resulting in a decrease in the accuracy of the task process.

[0011] In order to solve this problem, the inventor discovered that the above problem can be solved by including modality information indicating the image type of the image in a bitstream and transmitting it from the encoding device to the decoding device, and the decoding device then performing task processing based on the modality information, and thus came up with the present disclosure.

[0012] Next, each aspect of the present disclosure will be described.

[0013] A decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit acquires an image and parameters associated with the image from a bitstream, and the parameters include modality information indicating the image type of the image.

[0014] According to the first aspect, modality information indicating the image type of an image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby making it possible to improve the accuracy of task processing performed by the decoding device.

[0015] In the decoding device according to the second aspect of the present disclosure, in the first aspect, the image type may include at least one of a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

[0016] According to the second aspect, the decoding device is able to perform optimal task processing according to a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

[0017] A decoding device according to a third aspect of the present disclosure is the first or second aspect, wherein the circuit executes task processing based on the image and switches the task processing based on the modality information.

[0018] According to the third aspect, the decoding device can execute the optimal task processing according to the image type, and as a result, it is possible to improve the accuracy with which the decoding device executes the task processing.

[0019] A decoding device according to a fourth aspect of the present disclosure is the third aspect, wherein the task processing includes a machine task.

[0020] According to the fourth aspect, the decoding device can execute the optimum machine task according to the type of image.

[0021] A decoding device according to a fifth aspect of the present disclosure is the third or fourth aspect, wherein the task processing may include human vision.

[0022] According to the fifth aspect, the decoding device can perform optimal human vision according to the type of image.

[0023] In a decoding device according to a sixth aspect of the present disclosure, in any one of the first to fifth aspects, the circuit may perform task processing based on the image, and switch the AI ​​model or image processing used for the task processing based on the modality information.

[0024] According to the sixth aspect, the decoding device can perform task processing using the optimal AI model or optimal image processing according to the image type, thereby improving the accuracy with which the decoding device performs task processing.

[0025] In the decoding device according to the seventh aspect of the present disclosure, in any one of the first to sixth aspects, the parameters may further include conversion information for converting pixel values ​​of the image into sensor data values ​​or converting the image into a sensing image.

[0026] According to the seventh aspect, the decoding device can convert pixel values ​​of an image into sensor data values ​​or convert an image into a sensing image based on the conversion information.

[0027] In the decoding device according to the eighth aspect of the present disclosure, in the seventh aspect, the circuit may convert the pixel values ​​of the image to the sensor data values ​​or convert the image to the sensing image based on the conversion information, and perform task processing based on the sensor data values ​​or the sensing image.

[0028] According to the eighth aspect, the decoding device performs the conversion process before the task process, thereby enabling the task process to be executed based on the sensor data value or the sensed image.

[0029] A decoding device according to a ninth aspect of the present disclosure is the seventh aspect, wherein the circuit executes task processing based on the image and the transformation information.

[0030] According to the ninth aspect, the decoding device performs conversion processing during task processing, thereby enabling task processing to be executed based on sensor data values ​​or sensed images.

[0031] In a decoding device according to a tenth aspect of the present disclosure, in any one of the first to ninth aspects, the circuit, in acquiring the parameters, may acquire the parameters from a predetermined header area of ​​the bitstream, and the predetermined header area may include a VUI or an SEI.

[0032] According to the tenth aspect, the decoding device can easily obtain the parameters from a predetermined header area of ​​the bitstream.

[0033] In a decoding device according to an eleventh aspect of the present disclosure, in any one of the first to tenth aspects, the bitstream has a multi-layer structure including multiple image layers, and the circuit, in acquiring the images, may acquire multiple images of different image types from the multiple image layers.

[0034] According to the eleventh aspect, a plurality of images of different image types can be transmitted from the encoding device to the decoding device, and the decoding device can appropriately acquire the plurality of images from the bitstream.

[0035] In the decoding device according to the twelfth aspect of the present disclosure, in the eleventh aspect, the circuit may, in acquiring the parameters, acquire multiple parameters associated with the multiple images from a header area of ​​a specific image layer among the multiple image layers.

[0036] According to the twelfth aspect, the decoding device can collectively acquire a plurality of parameters associated with a plurality of images in a plurality of image layers from the header area of ​​a specific image layer.

[0037] In a decoding device according to a thirteenth aspect of the present disclosure, in the eleventh aspect, the circuit may, in acquiring the parameters, acquire multiple parameters associated with the multiple images from multiple header areas of the multiple image layers.

[0038] According to the thirteenth aspect, the decoding device can individually obtain each parameter associated with an image of each image layer from each header area of ​​each image layer.

[0039] In a decoding device according to a fourteenth aspect of the present disclosure, in any one of the first to tenth aspects, the image is divided into a plurality of sub-pictures, and the circuit, when acquiring the image, may acquire the plurality of sub-pictures of different image types from the bitstream.

[0040] According to the fourteenth aspect, a plurality of sub-pictures of different image types can be transmitted from the encoding device to the decoding device, and the decoding device can appropriately acquire the plurality of sub-pictures from the bitstream.

[0041] A decoding device according to a fifteenth aspect of the present disclosure is the fourteenth aspect, wherein the circuit, in acquiring the parameters, acquires a plurality of parameters associated with the plurality of sub-pictures from a header area of ​​the image.

[0042] According to the fifteenth aspect, the decoding device can collectively acquire a plurality of parameters associated with a plurality of sub-pictures from the header area of ​​an image.

[0043] In a decoding device according to a sixteenth aspect of the present disclosure, in any one of the first to tenth aspects, the image is composed of a plurality of components, and the circuit, when acquiring the image, may acquire the plurality of components of different image types from the bitstream.

[0044] According to the sixteenth aspect, a plurality of components of different image types can be transmitted from the encoding device to the decoding device, and the decoding device can appropriately acquire the plurality of components from the bitstream.

[0045] A decoding device according to a seventeenth aspect of the present disclosure is the sixteenth aspect, wherein the circuit, in acquiring the parameters, acquires a plurality of parameters associated with the plurality of components from a header area of ​​the image.

[0046] According to the seventeenth aspect, the decoding device can acquire a plurality of parameters associated with a plurality of components from the header area of ​​the image.

[0047] A decoding device according to an 18th aspect of the present disclosure is a decoding device according to the 16th aspect, in which the modality information is assigned to one of the plurality of components, and the circuit, when acquiring the image, acquires the one component from the bitstream.

[0048] According to the eighteenth aspect, in the case of an image type that requires only one component, modality information is assigned to the one component, allowing the decoding device to properly acquire the one component from the bitstream.

[0049] A decoding device according to a 19th aspect of the present disclosure is, in the 18th aspect, in which, when the modality information is not assigned to another component among the plurality of components that is different from the one component, the circuit may acquire the other component as a dummy image when acquiring the image.

[0050] According to the nineteenth aspect, the decoding device can reduce the processing load of the decoding device by acquiring other components to which modality information is not assigned as dummy images.

[0051] In a decoding device according to a twentieth aspect of the present disclosure, in any one of the first to nineteenth aspects, the circuit may acquire a plurality of images of different image types when acquiring the images, and the circuit may further execute at least one task processing based on the plurality of images.

[0052] According to the twentieth aspect, it is possible to improve the accuracy of execution of at least one task process by the decoding device based on a plurality of images of different image types.

[0053] A decoding device according to a 21st aspect of the present disclosure is the 20th aspect, wherein the plurality of images include visible light images, and the at least one task processing includes human vision.

[0054] According to the twenty-first aspect, by including a visible light image in the plurality of images, it becomes possible to appropriately implement human vision in the decoding device.

[0055] A coding device according to a twenty-second aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit encodes an image and parameters associated with the image into a bitstream, the parameters including modality information indicating the image type of the image.

[0056] According to the twenty-second aspect, the encoding device can transmit modality information indicating the image type of an image to the decoding device by including it in a bitstream, thereby improving the accuracy of task processing performed by the decoding device.

[0057] In the encoding device according to the 23rd aspect of the present disclosure, in the 22nd aspect, the image type may include at least one of a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

[0058] According to the 23rd aspect, it becomes possible for the decoding device to perform optimal task processing according to a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

[0059] In the encoding device according to a 24th aspect of the present disclosure, in the 22nd or 23rd aspect, the circuit may further generate the image based on sensor data values ​​or a sensed image.

[0060] According to the twenty-fourth aspect, the encoding device generates an image based on the sensor data values ​​or the sensed image, thereby enabling the image to be appropriately transmitted to the decoding device.

[0061] In the encoding device according to the 25th aspect of the present disclosure, in the 24th aspect, the parameters may further include conversion information for converting pixel values ​​of the image into the sensor data values ​​or converting the image into the sensing image.

[0062] According to the twenty-fifth aspect, the parameters include conversion information, so that the decoding device can convert pixel values ​​of an image into sensor data values ​​or convert an image into a sensing image.

[0063] In the encoding device according to the 26th aspect of the present disclosure, in any one of the 22nd to 25th aspects, the circuit, in encoding the parameters, may encode the parameters in a predetermined header area of ​​the bitstream, and the predetermined header area may include a VUI or an SEI.

[0064] According to the twenty-sixth aspect, the decoding device can easily obtain the parameters from a predetermined header area of ​​the bitstream.

[0065] In the encoding device according to the 27th aspect of the present disclosure, in any one of the 22nd to 26th aspects, the bitstream has a multi-layer structure including multiple image layers, and the circuit, in encoding the images, may encode multiple images of different image types into the multiple image layers.

[0066] According to the 27th aspect, the encoding device can transmit a plurality of images of different image types to the decoding device, and the decoding device can appropriately acquire the plurality of images from the bitstream.

[0067] In the encoding device according to the 28th aspect of the present disclosure, in the 27th aspect, the circuit may, in encoding the parameters, encode a plurality of parameters associated with the plurality of images into a header area of ​​a specific image layer among the plurality of image layers.

[0068] According to the twenty-eighth aspect, the encoding device can collectively encode a plurality of parameters associated with a plurality of images in a plurality of image layers in a header area of ​​a specific image layer.

[0069] In the encoding device of the 29th aspect of the present disclosure, in the 27th aspect, the circuit may, in encoding the parameters, encode multiple parameters associated with the multiple images into multiple header areas of the multiple image layers.

[0070] According to the twenty-ninth aspect, the encoding device can encode each parameter associated with an image of each image layer individually into each header area of ​​each image layer.

[0071] In the encoding device according to the 30th aspect of the present disclosure, in any one of the 22nd to 26th aspects, the image is divided into a plurality of sub-pictures, and the circuit, when encoding the image, may encode the plurality of sub-pictures of different image types into the bitstream.

[0072] According to the 30th aspect, the encoding device can transmit a plurality of sub-pictures of different image types to the decoding device, and the decoding device can appropriately obtain the plurality of sub-pictures from the bitstream.

[0073] In the encoding device according to the 31st aspect of the present disclosure, in the 30th aspect, the circuit may, in encoding the parameters, encode multiple parameters associated with the multiple subpictures into a header area of ​​the image.

[0074] According to the thirty-first aspect, the encoding device can encode a plurality of parameters associated with a plurality of sub-pictures collectively in the header area of ​​an image.

[0075] In the encoding device according to the 32nd aspect of the present disclosure, in any one of the 22nd to 26th aspects, the image is composed of multiple components, and the circuit, when encoding the image, may encode the multiple components of different image types into the bitstream.

[0076] According to the thirty-second aspect, the encoding device can transmit a plurality of components of different image types to the decoding device, and the decoding device can appropriately acquire the plurality of components from the bitstream.

[0077] In the encoding device according to the 33rd aspect of the present disclosure, in the 32nd aspect, the circuit may, in encoding the parameters, encode multiple parameters associated with the multiple components into a header area of ​​the image.

[0078] According to the thirty-third aspect, the encoding device can encode a plurality of parameters associated with a plurality of components into a header area of ​​an image.

[0079] In the encoding device according to the 34th aspect of the present disclosure, in the 32nd aspect, the modality information is assigned to one of the plurality of components, and the circuit may encode the one component into the bitstream when encoding the image.

[0080] According to the thirty-fourth aspect, in the case of an image type that requires only one component, the encoding device can appropriately encode the one component into a bitstream by assigning modality information to the one component.

[0081] In the encoding device according to the 35th aspect of the present disclosure, in the 34th aspect, if the modality information is not assigned to another component among the plurality of components that is different from the one component, the circuit may encode the other component as a dummy image when encoding the image.

[0082] According to the thirty-fifth aspect, the encoding device can reduce the processing load of the encoding device by encoding other components to which modality information is not assigned as dummy images.

[0083] A decoding method according to a thirty-sixth aspect of the present disclosure includes a decoding device that acquires an image and parameters associated with the image from a bitstream, and the parameters include modality information indicating the image type of the image.

[0084] According to the 36th aspect, modality information indicating the image type of an image can be included in a bitstream and transmitted from the encoding device to the decoding device, thereby making it possible to improve the accuracy with which task processing is executed by the decoding device.

[0085] In an encoding method according to a 37th aspect of the present disclosure, an encoding device encodes an image and parameters associated with the image into a bitstream, and the parameters include modality information indicating the image type of the image.

[0086] According to the 37th aspect, the encoding device can transmit modality information indicating the image type of an image to the decoding device by including it in a bitstream, thereby improving the accuracy of task processing performed by the decoding device.

[0087] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0088] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts are described as optional components. Furthermore, in all embodiments, the respective contents can be substituted or combined. Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or by any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0089] 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, and a transmission path NW.

[0090] Image data D1 is input from an external device to the encoding device 1. The external device may include a camera or the like that captures moving images. The external device inputs image data D1 of the captured moving images to the encoding device 1.

[0091] The encoding device 1 generates a bit stream BS based on the image data D1. Here, a bit stream refers to a data string or flow of digital data. The bit stream (or simply stream) may be a single stream, or may be composed of multiple streams divided into multiple layers. Furthermore, the bit stream may be transmitted by serial communication over a single transmission path, or by packet communication over multiple transmission paths. The encoding device 1 transmits the generated bit stream BS to the decoding device 2 via the transmission path NW. The decoding device 2 receives the bit stream BS.

[0092] The decoding device 2 decodes image data D1 from the bitstream BS and performs task processing based on the decoded image data D1. The task processing includes human vision and machine tasks. Human vision is the viewing or viewing of moving images by a human being such as an operator or a user. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using an artificial intelligence (AI) model, which is a machine-learned estimation model. The task processing unit that performs human vision includes a display device such as a liquid crystal display or an organic EL display. The task processing unit that performs machine tasks includes an inference device using AI.

[0093] The transmission path NW may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. The transmission path NW is preferably a private network or the like in which secure communication is ensured by access restrictions.

[0094] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 includes a processor such as a CPU. The memory 12 includes any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data to be processed by the circuit 11 or data in the middle of processing.

[0095] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 includes a processor such as a CPU. The memory 22 includes any recording medium such as a ROM, a RAM, a HDD, an SSD, or a semiconductor memory. The memory 22 stores data to be processed by the circuit 21 or data in the middle of processing.

[0096] 2 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 includes an acquisition unit 31, a setting unit 32, an encoding unit 33, and a transmission unit .

[0097] Next, the encoding unit 33 according to this embodiment will be described. Fig. 29 is a block diagram showing an example of the functional configuration of the encoding unit 33 according to this embodiment. The encoding unit 33 encodes an image in units of blocks.

[0098] 29 , the encoding unit 33 includes a dividing unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Note that the intra prediction unit 124 and the inter prediction unit 126 are configured as part of a prediction processing unit 125.

[0099] For example, the components of the encoding unit 33 shown in FIG. 29 are implemented by the circuit 11 and memory 12 shown in FIG.

[0100] The circuit 11 is configured to include a processor such as a CPU. The circuit 11 may be a dedicated or general-purpose electronic circuit for encoding images, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 11 may fulfill the roles of multiple components of the encoding unit 33 shown in FIG. 29 , excluding the component for storing information.

[0101] The memory 12 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 12 may be externally connected to the circuit 11 or may be built into the circuit 11. The memory 12 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 12 may be a non-volatile memory or a volatile memory.

[0102] The memory 12 may store an image to be encoded, or a stream corresponding to the encoded image, or may store a program for the processor to execute the image encoding process.

[0103] Furthermore, the memory 12 may serve as a component for storing information among the multiple components included in the encoding unit 33 shown in Fig. 29. Specifically, the memory 12 may serve as the block memory 118 and the frame memory 122 shown in Fig. 29. More specifically, the memory 12 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0104] 29 may be omitted from the encoding unit 33, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 29 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0105] FIG. 3 is a flowchart showing the process executed by the circuit 11 included in the encoding device 1.

[0106] First, in step SP11, the acquisition unit 31 acquires image data D11 representing the image Q to be processed, which is input from an external device. The image data D11 corresponds to the image data D1 shown in FIG.

[0107] Next, in step SP12, the setting unit 32 sets a parameter P in association with the image Q. The parameter P includes modality information D12 of the image Q. The modality information D12 indicates the image type of the image Q. The image type includes, for example, at least one of a visible light image, a thermal image, an infrared image, a LiDAR (light detection and ranging) image, a RADAR (radio detection and ranging) image, or an AI-generated image. The setting unit 32 may set the parameter P by image analysis based on the image data D11, or may set the parameter P based on setting information input by an operator of the encoding device 1.

[0108] Visible light images include natural images or RGB images, etc., and are used to provide detailed color information in human vision or machine tasks. Thermal images include temperature distribution images, etc., and are used to detect living organisms in the dark. Infrared images include images captured using an infrared camera, etc., and are used to capture images in the dark. LiDAR images include distance images detected by a ranging sensor using laser light, etc., and are used to provide accurate distance information. RADAR images include distance images detected by a ranging sensor using radio waves, etc., and are used to measure distances under any lighting conditions. AI-generated images include images generated by AI, or images in which explanatory text generated by AI is added to a visible light image, etc.

[0109] Next, in step SP13, the encoding unit 33 encodes the image Q represented by the image data D11 input from the acquisition unit 31 into a bit stream BS.

[0110] Next, in step SP14, the encoding unit 33 encodes the parameters P including the modality information D12 input from the setting unit 32 into a bit stream BS. Here, encoding the parameters P into the bit stream BS may be rephrased as saving the parameters P in the bit stream BS or storing the parameters P in the bit stream BS. Note that the order of execution of steps SP13 and SP14 may be reversed from the example in FIG. 3, or steps SP13 and SP14 may be executed simultaneously.

[0111] Next, in step SP15, the transmitting unit 34 transmits the bit stream BS input from the encoding unit 33 to the decoding device 2 via the transmission path NW.

[0112] 4 is a simplified diagram showing the structure of a bitstream BS. The bitstream BS has a header area 41 and a payload area 42. The encoding unit 33 stores encoded data of an image Q in the payload area 42, and stores encoded data of a parameter P associated with the image Q in the header area 41.

[0113] The encoding unit 33 may encode the encoded data of the parameter P in a predetermined area 43 in the header area 41. The predetermined area 43 may be VUI (video usability information) or SEI (supplemental enhancement information). However, the predetermined area 43 is not limited to VUI or SEI, and may also be VPS, SPS, PPS, PH, SH, APS, a tile header, a system layer header, or the like.

[0114] 30 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. As shown in (A) of FIG. 30, the video sequence includes, for example, a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), supplemental enhancement information (SEI), and multiple pictures.

[0115] In a video composed of multiple layers, the VPS includes coding parameters common to multiple layers and coding parameters related to multiple layers or individual layers included in the video.

[0116] The SPS includes parameters used for a sequence, i.e., encoding parameters that the decoding device 2 refers to in order to decode the sequence. The encoding parameters may indicate, for example, the width or height of a picture. Note that there may be multiple SPSs.

[0117] The PPS includes parameters used for a picture, i.e., encoding parameters that the decoding device 2 references to decode each picture in a sequence. The encoding parameters may include, for example, a reference value of the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Note that there may be multiple PPSs. The SPS and PPS may also be simply referred to as parameter sets.

[0118] As shown in (B) of Fig. 30, a picture includes a picture header and one or more slices. The picture header includes coding parameters that are referenced by the decoding device 2 to decode the one or more slices.

[0119] As shown in (C) of Fig. 30, a slice includes a slice header and one or more bricks. The slice header includes coding parameters that are referenced by the decoding device 2 to decode the one or more bricks.

[0120] A brick includes one or more coding tree units (CTUs), as shown in FIG.

[0121] Note that a picture may not contain slices, but may contain tile groups instead, where each tile group contains one or more tiles. Also, a brick may contain slices.

[0122] A CTU is also called a superblock or a basic division unit. As shown in (E) of Fig. 30, a CTU includes a CTU header and one or more coding units (CUs). The CTU header includes coding parameters that the decoding device 2 references to decode the one or more CUs.

[0123] A CU may be divided into multiple small CUs. Furthermore, as shown in (F) of FIG. 30 , a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting a CU. The residual coefficient information is information indicating a prediction residual. A CU is basically the same as a PU (Prediction Unit) or a TU (Transform Unit), but may include multiple TUs smaller than a CU. A CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is a fixed unit that can be processed in one stage, for example, when performing pipeline processing in hardware.

[0124] Note that a stream may not have some of the layers among the layers shown in Fig. 30. Furthermore, the order of these layers may be changed, and any layer may be replaced with another layer.

[0125] A picture currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current picture. If the processing is encoding, the current picture is synonymous with a picture to be encoded, and if the processing is decoding, the current picture is synonymous with a picture to be decoded. Furthermore, a block (CU or a block of a CU) currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current block. If the processing is encoding, the current block is synonymous with a block to be encoded, and if the processing is decoding, the current block is synonymous with a block to be decoded.

[0126] 5 and 6 are diagrams showing a first example of syntax related to setting the modality information D12. In the examples shown in Figs. 5 and 6, the modality information D12 is expressed as the value of the identifier vui_modality_type included in the VUI parameters.

[0127] As shown in Fig. 5, when the value of vui_modality_info_present_flag included in the VUI parameters is 1, it indicates that vui_modality_type is present. When the value of vui_modality_info_present_flag is 0, it indicates that vui_modality_type is not present. Note that when the value of vui_modality_info_present_flag is 0, it may also indicate that the value of vui_modality_type is 0.

[0128] As shown in FIG. 6 , when the value of vui_modality_type is 0, this indicates that the image type of image Q is a visible light image. When the value of vui_modality_type is 1, this indicates that the image type of image Q is a thermal image. When the value of vui_modality_type is 2, this indicates that the image type of image Q is an infrared image. When the value of vui_modality_type is 3, this indicates that the image type of image Q is a LiDAR image. When the value of vui_modality_type is 4, this indicates that the image type of image Q is a RADAR image. When the value of vui_modality_type is 5, this indicates that the image type of image Q is an AI-generated image. When the value of vui_modality_type is any other value, this indicates that this is a spare frame reserved for future use. Note that the number of images defined in FIG. 6 may be increased or decreased depending on the target image type.

[0129] 7 and 8 are diagrams showing a second example of syntax related to setting the modality information D12. In the example shown in Fig. 7 and 8, the modality information D12 is expressed as the value of an identifier of mvi_modality_type included in an SEI message such as machine_vision_indication.

[0130] As shown in Fig. 7, when the value of mvi_modality_info_present_flag included in the SEI message is 1, it indicates that mvi_modality_type is present. When the value of mvi_modality_info_present_flag is 0, it indicates that mvi_modality_type is not present. Note that when the value of mvi_modality_info_present_flag is 0, it may also indicate that the value of mvi_modality_type is 0.

[0131] As shown in FIG. 8, a value of 0 for mvi_modality_type indicates that the image type of image Q is a visible light image. A value of 1 for mvi_modality_type indicates that the image type of image Q is a thermal image. A value of 2 for mvi_modality_type indicates that the image type of image Q is an infrared image. A value of 3 for mvi_modality_type indicates that the image type of image Q is a LiDAR image. A value of 4 for mvi_modality_type indicates that the image type of image Q is a RADAR image. A value of 5 for mvi_modality_type indicates that the image type of image Q is an AI-generated image. Any other value for mvi_modality_type indicates that the frame is a spare frame reserved for future use. Note that the number of images defined in FIG. 8 may be increased or decreased depending on the target image type.

[0132] Generally, image data to be coded into a bitstream BS is composed of multiple components (image components). For example, a visible light image is composed of multiple color components (R, G, B), or one luminance component (Y) and multiple color difference components (Cb, Cr). Depending on the image type, there are images composed of three image components, such as color visible light images, and there are also images composed of only one image component (hereinafter referred to as "single-component images"), such as monochrome visible light images (visible light images with only a luminance component), thermal images, infrared images, LiDAR images, or RADAR images.

[0133] As a first example, when the bitstream BS is composed of three components and image Q, whose image type is a single-component image, is encoded, only the first component is encoded as a normal image into the bitstream BS, and the second and third components are encoded as dummy images into the bitstream BS. The encoding process for the dummy image may be simpler than the encoding process for a normal image. As a second example, when the bitstream BS is composed of three components and image Q, whose image type is a single-component image, is encoded, the single-component image Q may be converted into an image consisting of three image components, and the first to third components may be encoded as normal images into the bitstream BS. As a third example, when image Q, whose image type is a single-component image, may be encoded, in a format consisting of only one component, and only the first component may be encoded as a normal image into the bitstream BS.

[0134] 9 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 includes a receiving unit 51, a decoding unit 52, a switching unit 53A, and a plurality of n task processing units 54 (n is a natural number of 2 or more). １ ~54 ｎ The task processing unit 54 １ ~54 ｎ The tasks performed by include human vision and machine tasks, including various types of tasks using AI models, such as object detection, object tracking, object segmentation, action recognition, or pose estimation.

[0135] Next, the decoding unit 52 according to this embodiment will be described. Fig. 31 is a block diagram showing an example of the functional configuration of the decoding unit 52 according to this embodiment. The decoding unit 52 decodes a stream, which is an encoded image, in units of blocks.

[0136] 31 , the decoding unit 52 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a partition determination unit 224. Note that the intra prediction unit 216 and the inter prediction unit 218 are configured as part of a prediction processing unit 215.

[0137] For example, the components of the decoding unit 52 shown in FIG. 31 are implemented by the circuit 21 and memory 22 shown in FIG.

[0138] The circuit 21 is configured to include a processor such as a CPU. The circuit 21 may be a dedicated or general-purpose electronic circuit for decoding a stream, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 21 may fulfill the roles of multiple components of the decoding unit 52 shown in FIG. 31 , excluding the component for storing information.

[0139] The memory 22 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 22 may be externally connected to the circuit 21 or may be built into the circuit 21. The memory 22 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 22 may be a non-volatile memory or a volatile memory.

[0140] The memory 22 may store a stream to be decoded or a decoded image, or may store a program for the processor to execute the decoding process of the stream.

[0141] Furthermore, the memory 22 may serve as a component for storing information among the multiple components included in the decoding unit 52 shown in Fig. 31. Specifically, the memory 22 may serve as the block memory 210 and the frame memory 214 shown in Fig. 31. More specifically, the memory 22 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0142] 31 may be omitted from the decoding unit 52, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 31 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0143] The inverse quantization unit 204, inverse transform unit 206, addition unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter 212 included in the decoding unit 52 shown in Figure 31 perform the same processing as the inverse quantization unit 112, inverse transform unit 114, addition unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter 120 included in the encoding unit 33 shown in Figure 29.

[0144] FIG. 10 is a flowchart showing the process executed by the circuit 21 included in the decoding device 2.

[0145] First, in step SP21, the receiving unit 51 receives the bit stream BS transmitted by the encoding device 1 from the transmission path NW.

[0146] Next, in step SP22, the decoding unit 52 obtains image Q by decoding it from the payload area 42 of the bitstream BS input from the receiving unit 51. Note that the decoding may include extraction. The decoding unit 52 outputs image data D21 of image Q. The image data D21 corresponds to the image data D11 shown in FIG. 2.

[0147] Next, in step SP23, the decoding unit 52 obtains the parameter P by decoding it from the header area 41 (or a predetermined area 43 within the header area 41) of the bitstream BS input from the receiving unit 51. The decoding unit 52 extracts the modality information D22 included in the parameter P. The modality information D22 corresponds to the modality information D12 shown in Fig. 2. Note that the order of execution of steps SP22 and SP23 may be reversed from the example in Fig. 10, or they may be performed simultaneously.

[0148] Next, in step SP24A, the switching unit 53A switches the task processing unit 54 based on the modality information D22 input from the decoding unit 52. １ ~54 ｎ The switching unit 53A switches between a plurality of image types and a plurality of task processing units 54. １ ~54 ｎ The switching unit 53A holds table information (not shown) in which a correspondence relationship between the task and the task is preset. By referring to the table information, the switching unit 53A can select a task from among the plurality of task processing units 54. １ ~54 ｎ Among these, one task processing unit 54 corresponding to the image type indicated by the modality information D22 is selected. １ ~54 ｎ Select .

[0149] Next, in step SP25, the task processing unit 54 selected in step SP24A １ ~54 ｎ executes task processing based on image data D21 input from the decoding unit 52 via the switching unit 53A.

[0150] According to this embodiment, the encoding device 1 can transmit the modality information D12 (D22) indicating the image type of the image Q to the decoding device 2 by including it in the bitstream BS. As a result, it is possible to improve the accuracy of task processing execution by the decoding device 2.

[0151] Furthermore, according to this embodiment, the decoding device 2 can perform optimal processing (task processing, etc.) according to the image type, such as a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

[0152] Furthermore, according to this embodiment, the decoding device 2 can execute the optimal task processing according to the image type, and as a result, the accuracy with which the decoding device 2 executes the task processing can be improved.

[0153] Furthermore, according to this embodiment, the decoding device 2 can execute the optimum machine task according to the image type.

[0154] Furthermore, according to this embodiment, the decoding device 2 can perform optimal human vision according to the type of image.

[0155] Furthermore, according to this embodiment, the decoding device 2 can easily decode the parameter P from the predetermined area 43 of the bitstream BS.

[0156] Various modifications of the above embodiment will be described below. The modifications described below can be applied in any combination.

[0157] (First Modification) The decoding device 2 uses a plurality of AI models 55 for task processing. １ ~55 ｎ may be switched based on the modality information D22.

[0158] 11 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 has a receiving unit 51, a decoding unit 52, a switching unit 53B, and a task processing unit 54. The task processing executed by the task processing unit 54 includes human vision or machine tasks. The task processing unit 54 processes a plurality of n AI models 55. １ ~55 ｎ AI model 55 １ ~55 ｎ includes a neural network model that has undergone machine learning. For example, when the task processing performed by the task processing unit 54 is object detection, the AI ​​model 55 １ ~55 ｎ AI models include convolutional neural network models such as ResNet or YOLO. １ ~55 ｎEach of the setting parameters may be predefined in the decoding device 2, or may be transmitted from the encoding device 1 to the decoding device 2 by a bitstream other than the bitstream BS.

[0159] FIG. 12 is a flowchart showing the process executed by the circuit 21 included in the decoding device 2.

[0160] The processing in steps SP21 to SP23 is the same as that in FIG.

[0161] Next, in step SP24B, the switching unit 53B selects the AI ​​model 55 based on the modality information D22 input from the decoding unit 52. １ ~55 ｎ The switching unit 53B switches between a plurality of image types and a plurality of AI models 55. １ ~55 ｎ The switching unit 53B holds table information (not shown) in which a correspondence relationship between the AI ​​model 55 and the AI ​​model 55 is set in advance. １ ~55 ｎ Among these, one AI model 55 corresponding to the image type indicated by the modality information D22 is selected. １ ~55 ｎ Select .

[0162] Next, in step SP25, the task processing unit 54 executes the AI ​​model 55 selected in step SP24B. １ ~55 ｎ The task processing is performed based on the image data D21 input from the decoding unit 52 via the switching unit 53B using the above.

[0163] The switching unit 53B is configured to select a plurality of AI models 55 １ ~55 ｎ Alternatively, the decoding device 2 may switch between a plurality of image processing methods. The plurality of image processing methods may include any image processing method such as filtering, segmentation, edge detection, or color analysis. Furthermore, the decoding device 2 may select a task processing unit 54 based on the modality information D22. １ ~54 ｎ and AI model 55 １ ~55 ｎ Both of these may be switched.

[0164] According to this modification, the task processing unit 54 of the decoding device 2 generates an optimal AI model 55 according to the image type of the image Q. １ ~55 ｎ Alternatively, the task processing can be performed using an optimal image processing method, which can improve the accuracy of the task processing performed by the decoding device 2.

[0165] (Second Modification) The encoding device 1 may generate an image Q based on sensor data values ​​or a sensing image, and the parameter P may include conversion information D13 (D23) for converting pixel values ​​of image Q into the sensor data values ​​or converting image Q into the sensing image. The sensor data values ​​include the actual values ​​of data acquired from a sensor, or a data string in which these values ​​are arranged. The sensing image includes an image generated by arranging multiple sensor data values ​​in a matrix, or an image obtained by converting such an image. The sensing image may be a two-dimensional image or a three-dimensional image. Furthermore, the pixel values ​​of the sensing image may have any number of bits, such as 8 bits, 10 bits, or 16 bits.

[0166] 13 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 further includes a conversion unit 35 in addition to the configuration shown in FIG.

[0167] The acquisition unit 31 acquires sensor data D10 input from an external device. The external device may include an image sensor, a temperature sensor, a distance sensor, or the like.

[0168] The conversion unit 35 converts the sensor data values ​​of the sensor data D10 input from the acquisition unit 31 into pixel values ​​of the image Q, thereby generating image data D11 of the image Q based on the sensor data D10. Note that instead of converting the sensor data values ​​into pixel values, the conversion unit 35 may generate image data D11 of the image Q based on the image data of the sensing image by image conversion of the sensing image. Note that the image Q is the image to be coded. The image to be coded is an image obtained by converting the sensor data values ​​or the sensing image into an image that can be coded. The image to be coded is an image that can be recognized as an image by human vision. The image to be coded is a two-dimensional image. The pixel values ​​of the image to be coded are in bits conforming to a standard such as 8 bits or 10 bits.

[0169] The conversion by the conversion unit 35 may include conversion of units such as distance, temperature, or signal intensity. The conversion by the conversion unit 35 may also include dimensional conversion, such as conversion from a three-dimensional image to a two-dimensional image. The conversion by the conversion unit 35 may also include coordinate system conversion, such as conversion from a three-dimensional polar coordinate system to a two-dimensional Cartesian coordinate system, or conversion from a three-dimensional Cartesian coordinate system to a two-dimensional Cartesian coordinate system. The conversion by the conversion unit 35 may also include quantization conversion. The conversion by the conversion unit 35 may also include data conversion by normalization. The normalization may include conversion using a mathematical formula or conversion using table information. The conversion using a mathematical formula may include any method such as min-max scaling, Z-score normalization, or log transformation.

[0170] The conversion unit 35 inputs the image data D11 to the encoding unit 33. The conversion unit 35 also inputs conversion information D13 to the setting unit 32 for inversely converting the image data D11 into sensor data D10. The conversion information D13 includes image sample information. The image sample information includes an image sample interpretation, an image sample representation type, an angular resolution, or a focal length. The image sample interpretation defines the type of information used to represent an image sample in an image. The image sample can be represented as distance, temperature, radio waves, radiation, or signal strength such as LiDAR pulses. The image sample representation type defines how to convert image sample values ​​into sensor data values. The angular resolution or focal length includes conversion parameters for converting coordinate systems, such as from a two-dimensional Cartesian coordinate system to a three-dimensional polar coordinate system or from a two-dimensional Cartesian coordinate system to a three-dimensional Cartesian coordinate system.

[0171] The setting unit 32 sets parameters P in association with the image Q. The parameters P include modality information D12 of the image Q. The parameters P also include conversion information D13.

[0172] The encoding unit 33 encodes the image Q represented by the image data D11 input from the conversion unit 35 into a bit stream BS. The encoding unit 33 also encodes the parameters P including the modality information D12 and conversion information D13 input from the setting unit 32 into the bit stream BS.

[0173] The transmitting unit 34 transmits the bit stream BS input from the encoding unit 33 to the decoding device 2 via the transmission path NW.

[0174] 14 is a simplified diagram showing a first example of the configuration of the circuit 21 included in the decoding device 2. The circuit 21 further includes a conversion unit 55 in addition to the configuration shown in FIG.

[0175] The receiving unit 51 receives the bit stream BS transmitted by the encoding device 1 from the transmission path NW.

[0176] The decoding unit 52 decodes the image Q and the parameters P from the bit stream BS input from the receiving unit 51. The decoding unit 52 also extracts the modality information D22 and the conversion information D23 included in the parameters P. The conversion information D23 corresponds to the conversion information D13 shown in FIG.

[0177] The conversion unit 55 converts pixel values ​​of the image Q represented by the image data D21 input from the decoding unit 52 into sensor data values ​​based on the conversion information D23, thereby generating sensor data D20 based on the image data D21. The sensor data D20 corresponds to the sensor data D10 shown in Fig. 13. Note that the conversion unit 55 may convert the image Q into the above-mentioned sensing image by image conversion based on the conversion information D23, instead of converting pixel values ​​into sensor data values.

[0178] The conversion by the conversion unit 55 may include conversion of units such as distance, temperature, or signal intensity. The conversion by the conversion unit 55 may also include dimensional conversion, such as conversion from a two-dimensional image to a three-dimensional image. The conversion by the conversion unit 55 may also include coordinate system conversion, such as conversion from a two-dimensional Cartesian coordinate system to a three-dimensional polar coordinate system or conversion from a two-dimensional Cartesian coordinate system to a three-dimensional Cartesian coordinate system. The conversion by the conversion unit 55 may also include inverse quantization conversion. The conversion by the conversion unit 55 may also include data conversion by inverse normalization. The inverse normalization may include conversion using a mathematical formula or conversion using table information. The conversion using a mathematical formula may include any method such as min-max scaling, Z-score normalization, or log transformation.

[0179] The switching unit 53A switches the task processing unit 54 based on the modality information D22 input from the decoding unit 52. １ ~54 ｎ Switch.

[0180] The task processing unit 54 selected by the switching unit 53A １ ~54 ｎ executes task processing based on the sensor data D20 input from the conversion unit 55 via the switching unit 53A or the image data of the sensing image.

[0181] FIG. 15 is a simplified diagram showing a second example of the configuration of the circuit 21 included in the decoding device 2.

[0182] The receiving unit 51 receives the bit stream BS transmitted by the encoding device 1 from the transmission path NW.

[0183] The decoding unit 52 decodes the image Q and the parameters P from the bit stream BS input from the receiving unit 51. The decoding unit 52 also extracts the modality information D22 and the conversion information D23 included in the parameters P.

[0184] The switching unit 53A switches the task processing unit 54 based on the modality information D22 input from the decoding unit 52. １ ~54 ｎ Switch.

[0185] The decoding unit 52 selects one task processing unit 54 selected by the switching unit 53A. １ ~54 ｎ Image data D21 and conversion information D23 are input to the .

[0186] The task processing unit 54 selected by the switching unit 53A １ ~54 ｎ executes task processing based on the input image data D21 and conversion information D23. １ ~54 ｎ The task processing unit 54 may generate the sensor data D20 based on the image data D21 by converting pixel values ​​of the image Q represented by the image data D21 into sensor data values ​​based on the conversion information D23. １ ~54 ｎ Alternatively, the task processing unit 54 may execute task processing based on the generated sensor data D20. １ ~54 ｎ may convert the image Q into the sensing image by image conversion based on the conversion information D23, and execute the task processing based on the sensing image.

[0187] According to this modification, the encoding device 1 generates the image Q based on the sensor data values ​​or the sensed image, thereby making it possible to appropriately transmit the image Q to the decoding device 2 .

[0188] Furthermore, according to this modification, the parameter P includes the conversion information D13 (D23), so that the decoding device 2 can convert the pixel values ​​of the image Q into sensor data values ​​or convert the image Q into a sensing image.

[0189] Furthermore, according to the configuration example shown in FIG. 14, the decoding device 2 performs conversion processing before task processing, thereby enabling task processing to be executed based on sensor data values ​​or sensing images.

[0190] Furthermore, according to the configuration example shown in FIG. 15, the decoding device 2 performs conversion processing during task processing, thereby enabling task processing to be executed based on sensor data values ​​or sensing images.

[0191] (Third Modification) The bit stream BS is composed of m layers (m is a natural number equal to or greater than 2) of image layers L １ ~L ｍ It may have a multi-layer structure including:

[0192] Fig. 16 is a diagram showing a first example of a multi-layer structure. Fig. 16 shows only one access unit. An access unit is the smallest processing unit of a time attribute, and corresponds to, for example, one frame of a video image. A bit stream BS is configured to include multiple access units that are consecutive in time.

[0193] The first image layer L, which is the lowest layer １ The payload area 42 of the image Q Ｌ１ The coded data of the second image layer L is stored. ２ The payload area 42 of the image Q Ｌ２ Similarly, the encoded data of the m-th image layer L ｍ The payload area 42 of the image Q Ｌｍ The encoded data is stored.

[0194] Image QＬ１ ~Q Ｌｍ For example, in a three-layer structure with m=3, image Q Ｌ１ is the visible light image, and image Q Ｌ２ is the infrared image, and image Q Ｌ３ is a thermal image.

[0195] Image Q Ｌ１ ~Q Ｌｍ If there is a correlation between the image layers (e.g., infrared and thermal images), １ ~L ｍ Images may be referenced between images. Ｌ１ ~Q Ｌｍ If there is no correlation between the image layers (for example, a visible light image and a range image), １ ~L ｍ Images do not need to be referenced between them.

[0196] Image layer L １ The header area 41 of the image Q Ｌ１ The parameter P associated with Ｌ１ The encoded data of the parameter P is stored. Ｌ１ is image Q Ｌ１ The modality information D12 of the image layer L １ The header area 41 of the image Q Ｌ２ ~Q Ｌｍ The parameter P associated with Ｌ２ ~P Ｌｍ The encoded data of the parameter P Ｌ２ ~P Ｌｍ is image Q Ｌ２ ~Q Ｌｍ All image layers L １ ~L ｍ Parameter P Ｌ１ ~P Ｌｍ a specific image layer L １ A scalability dimension information (SDI)_SEI message may be used to store the information in the header area 41 of the .

[0197] 17 is a diagram showing an example of syntax related to setting the modality information D12. The modality information D12 is expressed as the value of the identifier sdi_aux_id[i] included in the SDI_SEI message. When the value of sdi_aux_id[i] is 0, the i-th image layer L ｉ Image Q is an auxiliary image. Ｌｉ If the value of sdi_aux_id[i] is 1 to 6, it indicates that the i-th image layer L ｉ , image Q of the image type shown in FIG. Ｌｉ It should be noted that if the value of sdi_aux_id[i] is between 7 and 127 or between 160 and 255, it indicates that the slot is reserved for future use, and if the value of sdi_aux_id[i] is between 128 and 159, it indicates that the slot is unspecified. It is also possible to increase or decrease the number of images defined in FIG. 17 depending on the type of image to be targeted.

[0198] 18 and 19 are diagrams showing examples of syntax related to setting the modality information D12 without using the SDI_SEI message. The modality information D12 is expressed as the value of the identifier mvi_modality_type[i] included in an SEI message such as machine_vision_indication.

[0199] As shown in Figures 18 and 19, the number of layers included in the multi-layer structure is specified by mvi_max_layers_minus1, and modality information for the specified number of layers is collectively described. ｉ The modality information is described as mvi_modality_type[i].

[0200] If the value of mvi_modality_type[i] is 0, image Q Ｌｉ If the value of mvi_modality_type[i] is 1, the image type of image Q is a visible light image. Ｌｉ If the value of mvi_modality_type[i] is 2, the image type of image Q is a thermal image. ＬｉIf the value of mvi_modality_type[i] is 3, the image type of image Q is an infrared image. Ｌｉ If the value of mvi_modality_type[i] is 4, the image type of image Q is LiDAR image. Ｌｉ If the value of mvi_modality_type[i] is 5, the image type of image Q is a RADAR image. Ｌｉ This indicates that the image type is an AI-generated image. If the value of mvi_modality_type[i] is any other value, it indicates a spare frame reserved for future use. Note that the number of images defined in Figure 19 may be increased or decreased depending on the target image type.

[0201] Fig. 20 is a diagram showing a second example of a multi-layer structure, in which only one access unit is shown.

[0202] Image layer L １ ~L ｍ The payload area 42 of the image Q Ｌ１ ~Q Ｌｍ The encoded data is stored.

[0203] Image layer L １ The header area 41 of the image Q Ｌ１ The parameter P associated with Ｌ１ The coded data of the image layer L is stored. ２ The header area 41 of the image Q Ｌ２ The parameter P associated with Ｌ２ Similarly, the coded data of the image layer L ｍ The header area 41 of the image Q Ｌｍ The parameter P associated with Ｌｍ In this way, the coded data of each image layer L １ ~L ｍ Parameter P Ｌ１ ~P Ｌｍ is the image layer L １ ~L ｍ The information is stored in the header area 41 of the file.

[0204] Each image layer L １ ~Lｍ Parameter P Ｌ１ ~P Ｌｍ is the image layer L １ ~L ｍ In the examples shown in Figs. 5 and 6, the modality information D12 is expressed as the value of an identifier of vui_modality_type included in the VUI parameters. In the examples shown in Figs. 7 and 8, the modality information D12 is expressed as the value of an identifier of mvi_modality_type included in an SEI message such as machine_vision_indication. Note that, in all image layers L １ ~L ｍ In order to store the above SEIs together in one header area, an SN (scalable nesting)_SEI message may be used.

[0205] FIG. 21 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2.

[0206] The decoder 52 extracts the image Q from the payload area 42 of the bit stream BS having a multi-layer structure input from the receiver 51. Ｌ１ ~Q Ｌｍ The decoding unit 52 decodes the image Q Ｌ１ ~Q Ｌｍ Image data D21 １ ~D21 ｍ Output.

[0207] The decoding unit 52 also extracts the parameter P from the header area 41 of the bit stream BS having a multi-layer structure input from the receiving unit 51. Ｌ１ ~P Ｌｍ The decoding unit 52 decodes the parameter P Ｌ１ ~P Ｌｍ Modality information D22 included in １ ~D22 ｍ Extract.

[0208] The switching unit 53A selects the modality information D22 input from the decoding unit 52. １ ~D22 ｍ Based on this, image Q Ｌ１ ~Q Ｌｍ Task processing unit 54 １ ~54ｎ The switching unit 53A switches between a plurality of image types and a plurality of task processing units 54. １ ~54 ｎ The switching unit 53A stores table information (not shown) in which a correspondence relationship between the image Q and the image Q is set in advance. Ｌ１ ~Q Ｌｍ For each task, multiple task processors 54 １ ~54 ｎ Among these, modality information D22 １ ~D22 ｍ The task processing unit 54 corresponds to the image type indicated by １ ~54 ｎ Select .

[0209] Task processing unit 54 １ ~54 ｎ is the image data D21 input from the decoding unit 52 via the switching unit 53A. １ ~D21 ｍ Perform task processing based on the

[0210] 21 shows a configuration in which one image data is input to one task processing unit 54, but a configuration in which multiple image data are input to one task processing unit 54 may also be used. Specifically, the switching unit 53A can select multiple image types and multiple task processing units 54. １ ~54 ｎ The task processing unit 54 may store table information in which a correspondence relationship between the images Q and the task processing unit 54 is set in advance, and in the table information, a plurality of images Q may be associated with one task processing unit 54.

[0211] According to this modification, the encoding device 1 encodes a plurality of images Q of different image types. Ｌ１ ~Q Ｌｍ can be transmitted to the decoding device 2, and the decoding device 2 can receive the plurality of images Q Ｌ１ ~Q Ｌｍ can be properly decoded from the bitstream BS.

[0212] According to the configuration example shown in FIG. 16, the encoding device 1 encodes a plurality of image layers L １ ~L ｍ Multiple images Q of Ｌ１ ~Q Ｌｍ A plurality of parameters P associated withＬ１ ~P Ｌｍ for a specific image layer L １ Furthermore, the decoding device 2 can encode multiple image layers L １ ~L ｍ Multiple images Q of Ｌ１ ~Q Ｌｍ A plurality of parameters P associated with Ｌ１ ~P Ｌｍ for a specific image layer L １ It is possible to decode the data collectively from the header area 41.

[0213] According to the configuration example shown in FIG. 20, the encoding device 1 encodes each image layer L １ ~L ｍ Image Q Ｌ１ ~Q Ｌｍ Each parameter P associated with Ｌ１ ~P Ｌｍ for each image layer L １ ~L ｍ In addition, the decoding device 2 can encode the image data into the header area 41 of each image layer L １ ~L ｍ Image Q Ｌ１ ~Q Ｌｍ Each parameter P associated with Ｌ１ ~P Ｌｍ for each image layer L １ ~L ｍ It is possible to decode the data individually from each header area 41.

[0214] (Fourth Modification) An image Q is divided into a plurality of j sub-pictures Q (j is a natural number equal to or greater than 2). Ｓ１ ~Q Ｓｊ It may be divided into

[0215] 22 is a simplified diagram showing the structure of the bitstream BS. Ｓ１ ~Q Ｓｊ The coded data of the sub-picture Q is stored in the payload area 42. Ｓ１ ~Q Ｓｊ The parameter P associated with Ｓ１ ~P Ｓｊ The coded data of sub-picture Q is stored in the header area 41.Ｓ１ ~Q Ｓｊ have different image types. Ｓ１ ~P Ｓｊ is the subpicture Q Ｓ１ ~Q Ｓｊ All subpictures Q Ｓ１ ~Q Ｓｊ Parameter P Ｓ１ ~P Ｓｊ In order to store these together in the header area 41, a scalable nesting (SN)_SEI message may be used.

[0216] 23 and 24 are diagrams showing examples of syntax related to the setting of the modality information D12. Ｓ１ ~Q Ｓｊ The modality information D12 associated with each of the modalities is represented as the value of the mvi_modality_type identifier included in the j-th machine_vision_indication in the SN_SEI message.

[0217] As shown in FIG. 24, when the value of mvi_modality_type is 0, the subpicture Q Ｓｊ If the value of mvi_modality_type is 1, the image type of the subpicture Q is a visible light image. Ｓｊ If the value of mvi_modality_type is 2, the image type of the subpicture Q is a thermal image. Ｓｊ If the value of mvi_modality_type is 3, the image type of the subpicture Q is an infrared image. Ｓｊ If the value of mvi_modality_type is 4, the image type of subpicture Q is LiDAR image. Ｓｊ If the value of mvi_modality_type is 5, the image type of subpicture Q is RADAR image. ＳｊThis indicates that the image type is an AI-generated image. If the value of mvi_modality_type is any other value, it indicates that the frame is reserved for future use. Note that the number of images defined in Figure 24 may be increased or decreased depending on the target image type.

[0218] According to this modification, the encoding device 1 encodes a plurality of sub-pictures Q of different image types. Ｓ１ ~Q Ｓｊ The decoding device 2 can transmit the sub-pictures Q Ｓ１ ~Q Ｓｊ can be properly decoded from the bitstream BS.

[0219] Furthermore, according to this modification, the encoding device 1 encodes a plurality of sub-pictures Q Ｓ１ ~Q Ｓｊ A plurality of parameters P associated with Ｓ１ ~P Ｓｊ In addition, according to this modification, the decoding device 2 can encode a plurality of sub-pictures Q Ｓ１ ~Q Ｓｊ A plurality of parameters P associated with Ｓ１ ~P Ｓｊ can be decoded together from the header area 41 of the image Q.

[0220] (Fifth Modification) An image Q is composed of a plurality of k components Q (k is a natural number of 2 or more). Ｃ１ ~Q Ｃｋ It may be configured by:

[0221] Component Q Ｃ１ ~Q Ｃｋ are multiple components that make up image Q. For example, a visible light image is made up of multiple color components (R, G, B), or is made up of one luminance component (Y) and multiple color difference components (Cb, Cr).

[0222] FIG. 25 shows a diagram of multiple components Q Ｃ１ ~Q Ｃｋ 10 is a diagram showing an example of the configuration of an image Q according to the present invention.

[0223] The payload area 42 of the bitstream BS contains the first image component C１ Component Q corresponding to Ｃ１ and the second image component C ２ Component Q corresponding to Ｃ２ Similarly, the encoded data of the k-th image component C ｋ Component Q corresponding to Ｃｋ The encoded data is stored.

[0224] Component Q Ｃ１ ~Q Ｃｋ For example, in a three-component configuration with k=3, the component Q Ｃ１ is the visible light image, and component Q Ｃ２ is the infrared image and the component Q Ｃ３ is a thermal image.

[0225] Component Q Ｃ１ ~Q Ｃｋ When there is a correlation between the image components C １ ~C ｋ Images may be referenced between components. Ｃ１ ~Q Ｃｋ When there is no correlation between the image components C １ ~C ｋ Images do not need to be referenced between them.

[0226] The header area 41 of the bitstream BS contains the component Q Ｃ１ ~Q Ｃｋ The parameter P associated with Ｃ１ ~P Ｃｋ The encoded data of the parameter P is stored. Ｃ１ ~P Ｃｋ is the component Q Ｃ１ ~Q Ｃｋ The modality information D12 includes:

[0227] 26 and 27 are diagrams showing examples of syntax related to setting the modality information D12. The modality information D12 is expressed as the value of the identifier mvi_modality_type[c] included in an SEI message such as machine_vision_indication.

[0228] As shown in FIG. 27, when the value of mvi_modality_type[c] is 0, component Q Ｃｋ If the value of mvi_modality_type[c] is 1, the image type of component Q is a visible light image. Ｃｋ If the value of mvi_modality_type[c] is 2, the image type of component Q is a thermal image. Ｃｋ If the value of mvi_modality_type[c] is 3, the image type of component Q is an infrared image. Ｃｋ If the value of mvi_modality_type[c] is 4, the component Q Ｃｋ If the value of mvi_modality_type[c] is 5, the image type of component Q is RADAR image. Ｃｋ This indicates that the image type is an AI-generated image. If the value of mvi_modality_type[c] is any other value, it indicates that the frame is reserved for future use. Note that the number of images defined in Figure 27 may be increased or decreased depending on the target image type.

[0229] FIG. 28 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2.

[0230] The decoder 52 extracts component Q from the payload area 42 of the multi-component bitstream BS input from the receiver 51. Ｃ１ ~Q Ｃｋ The decoding unit 52 decodes the component Q Ｃ１ ~Q Ｃｋ Image data D21 １ ~D21 ｋ Output.

[0231] The decoding unit 52 also extracts the parameter P from the header area 41 of the multi-component bit stream BS input from the receiving unit 51. Ｃ１ ~P Ｃｋ The decoding unit 52 decodes the parameter P Ｃ１ ~PＣｋ Modality information D22 included in １ ~D22 ｋ Extract.

[0232] The switching unit 53A selects the modality information D22 input from the decoding unit 52. １ ~D22 ｋ Based on the component Q Ｃ１ ~Q Ｃｋ Task processing unit 54 １ ~54 ｎ The switching unit 53A switches between a plurality of image types and a plurality of task processing units 54. １ ~54 ｎ The switching unit 53A holds table information (not shown) in which a correspondence relationship between the component Q and the component Q is set in advance. Ｃ１ ~Q Ｃｋ For each task, multiple task processors 54 １ ~54 ｎ Among these, modality information D22 １ ~D22 ｋ The task processing unit 54 corresponds to the image type indicated by １ ~54 ｎ Select .

[0233] Task processing unit 54 １ ~54 ｎ is the image data D21 input from the decoding unit 52 via the switching unit 53A. １ ~D21 ｋ Perform task processing based on the

[0234] 28 shows a configuration in which one image data is input to one task processing unit 54, but a configuration in which multiple image data are input to one task processing unit 54 may also be used. Specifically, the switching unit 53A can select multiple image types and multiple task processing units 54. １ ~54 ｎ The task processing unit 54 may store table information in which a correspondence relationship between the images Q and the task processing unit 54 is set in advance, and in the table information, a plurality of images Q may be associated with one task processing unit 54.

[0235] Furthermore, depending on the type of image, there are images that are composed of three image components, such as color visible light images, and there are also images that are composed of only one image component (hereinafter referred to as "single-component images"), such as monochrome visible light images (visible light images with only the luminance component), thermal images, infrared images, LiDAR images, or RADAR images.

[0236] As an example, the bitstream BS is divided into three components Q Ｃ１ ~Q Ｃ３ If a color visible light image is composed of three components Q Ｃ１ ~Q Ｃ３ All modality information D22 １ ~D22 ３ On the other hand, if the bitstream BS is composed of, for example, a monochrome visible light image, an infrared image, and a thermal image, then the component Q Ｃ１ Modality information D22 １ is the visible light image, component Q Ｃ２ Modality information D22 ２ is the infrared image, component Q Ｃ３ Modality information D22 ３ On the other hand, if the bitstream BS consists of only infrared images, then the component Q Ｃ１ Modality information D22 １ is an infrared image, but component Q Ｃ２ , Q Ｃ３ No modality information is assigned to

[0237] The encoding unit 33 encodes the modality information D22 １ One component Q is assigned Ｃ１ On the other hand, the encoding unit 33 encodes the modality information D22 ２ ~D22 ｋ Other components Q that are not assigned Ｃ２ ~Q Ｃｋ is encoded as a dummy image into the bitstream BS. The encoding process for the dummy image may be simpler than the encoding process for the normal image.

[0238] The decoding unit 52 also decodes the modality information D22 １ One component Q is assigned Ｃ１ On the other hand, the decoding unit 52 decodes the modality information D22 ２ ~D22 ｋ Other components Q that are not assigned Ｃ２ ~Q Ｃｋ is decoded as a dummy image from the bitstream BS. The decoding process for the dummy image may be simpler than the decoding process for the normal image.

[0239] According to this modification, the encoding device 1 encodes a plurality of components Q having different image types. Ｃ１ ~Q Ｃｋ can be transmitted to the decoding device 2, and the decoding device 2 can transmit the plurality of components Q Ｃ１ ~Q Ｃｋ can be properly decoded from the bitstream BS.

[0240] According to this modification, the encoding device 1 encodes a plurality of components Q Ｃ１ ~Q Ｃｋ A plurality of parameters P associated with Ｃ１ ~P Ｃｋ In addition, the decoding device 2 can encode a plurality of components Q Ｃ１ ~Q Ｃｋ A plurality of parameters P associated with Ｃ１ ~P Ｃｋ can be decoded from the header area 41 of the bitstream BS.

[0241] According to this modification, one component Q Ｃ１ In the case of an image type that requires only one component Q Ｃ１ Modality information D22 １ By assigning Ｃ１ can be properly encoded into a bitstream BS, and the decoding device 2 can decode the one component Q Ｃ１ can be properly decoded from the bitstream BS.

[0242] According to this modification, the encoding device 1 receives modality information D22 ２ ~D22 ｋ Other components Q that are not assigned Ｃ２ ~Q Ｃｋ By encoding the modality information D22 as a dummy image, the processing load of the encoding device 1 can be reduced. ２ ~D22 ｋ Other components Q that are not assigned Ｃ２ ~Q Ｃｋ By decoding as a dummy image, the processing load on the decoding device 2 can be reduced.

[0243] The present disclosure is particularly useful when applied to an image processing system that includes an encoding device that encodes an image into a bitstream and transmits the bitstream, and a decoding device that decodes an image from the received bitstream.

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit obtains an image and parameters associated with the image from a bitstream, the parameters including modality information indicating an image type of the image.

2. The decoding device of claim 1, wherein the image type includes at least one of a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

3. The decoding device according to claim 1, wherein the circuit performs task processing based on the image, and switches the task processing based on the modality information.

4. The decoding device according to claim 3, wherein the task processing includes a machine task.

5. The decoding device of claim 3, wherein the task processing includes human vision.

6. The decoding device according to claim 1, wherein the circuit performs task processing based on the image, and switches the AI ​​model or image processing used for the task processing based on the modality information.

7. The decoding device according to claim 1, wherein the parameters further include conversion information for converting pixel values ​​of the image into sensor data values ​​or converting the image into a sensing image.

8. The decoding device according to claim 7, wherein the circuit converts the pixel values ​​of the image to the sensor data values ​​or converts the image to the sensing image based on the conversion information, and performs task processing based on the sensor data values ​​or the sensing image.

9. The decoding device according to claim 7, wherein the circuitry performs a task process based on the image and the transformation information.

10. The decoding device according to claim 1, wherein, in obtaining the parameters, the circuit obtains the parameters from a predetermined header area of ​​the bitstream, and the predetermined header area includes a VUI or a SEI.

11. The decoding device according to claim 1, wherein the bit stream has a multi-layer structure including a plurality of image layers, and the circuit, in acquiring the images, acquires a plurality of images of different image types from the plurality of image layers.

12. The decoding device according to claim 11, wherein, in obtaining the parameters, the circuit obtains the parameters associated with the images from a header area of ​​a specific image layer among the image layers.

13. The decoding device according to claim 11, wherein the circuit, in obtaining the parameters, obtains a plurality of parameters associated with the plurality of images from a plurality of header areas of the plurality of image layers.

14. The decoding device according to claim 1, wherein the image is divided into a plurality of sub-pictures, and the circuit, in obtaining the image, obtains the plurality of sub-pictures having different image types from the bit stream.

15. The decoding device according to claim 14, wherein said circuit, in obtaining said parameters, obtains a plurality of parameters associated with said plurality of sub-pictures from a header area of ​​said image.

16. The decoding device according to claim 1, wherein the image is composed of a plurality of components, and the circuit, in acquiring the image, acquires the plurality of components having different image types from the bit stream.

17. The decoding device according to claim 16, wherein said circuit, in obtaining said parameters, obtains a plurality of parameters associated with said plurality of components from a header area of ​​said image.

18. The decoding device according to claim 16, wherein the modality information is assigned to one component among the plurality of components, and the circuitry obtains the one component from the bitstream during acquisition of the image.

19. The decoding device according to claim 18, wherein, when the modality information is not assigned to another component among the plurality of components that is different from the one component, the circuit acquires the other component as a dummy image when acquiring the image.

20. The decoding device according to claim 1, wherein the circuit acquires a plurality of images of different image types in acquiring the images, and the circuit further executes at least one task process based on the plurality of images.

21. The decoding device of claim 20, wherein the plurality of images include visible light images, and the at least one processing task includes human vision.

22. An encoding device comprising: a circuit; and a memory connected to the circuit, the circuit encoding an image and parameters associated with the image into a bitstream, the parameters including modality information indicating an image type of the image.

23. The encoding device of claim 22, wherein the image type includes at least one of a visible light image, a thermal image, an infrared image, a LiDAR image, a RADAR image, or an AI-generated image.

24. The encoding device of claim 22, wherein the circuitry is further configured to generate the image based on sensor data values ​​or a sensed image.

25. The encoding device according to claim 24, wherein the parameters further include conversion information for converting pixel values ​​of the image to the sensor data values ​​or converting the image to the sensing image.

26. The encoding device according to claim 22, wherein, in encoding the parameters, the circuit encodes the parameters in a predetermined header area of ​​the bitstream, the predetermined header area including a VUI or a SEI.

27. The encoding device according to claim 22, wherein the bit stream has a multi-layer structure including a plurality of image layers, and the circuit, in encoding the images, encodes a plurality of images of different image types into the plurality of image layers.

28. The encoding device according to claim 27, wherein said circuit, in encoding the parameters, encodes a plurality of parameters associated with the plurality of images into a header area of ​​a particular image layer among the plurality of image layers.

29. The encoding device of claim 27, wherein said circuitry, in encoding the parameters, encodes a plurality of parameters associated with the plurality of images into a plurality of header regions of the plurality of image layers.

30. The encoding device according to claim 22, wherein the image is divided into a plurality of sub-pictures, and the circuit, in encoding the image, encodes the plurality of sub-pictures having different image types into the bit stream.

31. The encoding device according to claim 30, wherein said circuitry, in encoding said parameters, encodes a plurality of parameters associated with said plurality of sub-pictures into a header area of ​​said image.

32. The encoding device according to claim 22, wherein the image is composed of a plurality of components, and the circuit, in encoding the image, encodes the plurality of components having different image types into the bitstream.

33. The encoding device according to claim 32, wherein said circuit, in encoding said parameters, encodes a plurality of parameters associated with said plurality of components into a header area of ​​said image.

34. The encoding device according to claim 32, wherein the modality information is assigned to one component of the plurality of components, and the circuit encodes the one component into the bitstream in encoding the image.

35. The encoding device according to claim 34, wherein, when the modality information is not assigned to another component among the plurality of components that is different from the one component, the circuit encodes the other component as a dummy image when encoding the image.

36. A decoding method, comprising: a decoding device obtaining an image and parameters associated with the image from a bitstream; the parameters including modality information indicating an image type of the image.

37. A method for encoding, comprising: an encoding device encoding an image and parameters associated with the image into a bitstream, the parameters including modality information indicating an image type of the image.

Images