Decoding apparatus, encoding apparatus, decoding method, and encoding method

Abstract

The decoding device includes circuitry and a memory connected to the circuitry. The circuitry acquires an image from a bitstream and a first parameter associated with the image. The first parameter includes modal information representing the image category, which indicates that the image category is at least one of the following: a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension.

Description

Technical Field

[0001] This disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method. Background Technology

[0002] Patent Document 1 discloses an image processing system according to the background art. This image processing system includes an encoder (encoding device) and a decoder (decoding device). In the encoding device, various modalities of input images are input. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image parsing task based on the input features, thereby outputting a segmentation map.

[0003] However, in the background art, no research has been conducted on the transmission of modal information of an image from the encoding device to the decoding device.

[0004] Prior art literature

[0005] Patent documents

[0006] Patent Document 1: U.S. Patent Application Publication No. 2024 / 0046453 Summary of the Invention

[0007] The purpose of this disclosure is to provide a decoding device, encoding device, decoding method, and encoding method that can transmit modal information of an image from an encoding device to a decoding device, thereby improving the execution accuracy of task processing performed by the decoding device.

[0008] The decoding apparatus according to one aspect of this disclosure includes circuitry and a memory connected to the circuitry. The circuitry acquires an image from a bitstream and a first parameter associated with the image. The first parameter includes modal information representing the image category, which indicates that the image category is at least one of the following: a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension. Attached Figure Description

[0009] Figure 1 This is a simplified diagram illustrating the structure of an image processing system according to an embodiment of the present disclosure.

[0010] Figure 2 This diagram is a simplified representation of the circuitry of the encoding device.

[0011] Figure 3 This is a flowchart illustrating the processing performed by the circuitry of the encoding device.

[0012] Figure 4 It is a diagram that simplifies the structure of a bitstream.

[0013] Figure 5 This is a simplified illustration of the syntax for VUI parameters.

[0014] Figure 6 This is a diagram illustrating an example of the correspondence between the value of vui_modality_type and the image category.

[0015] Figure 7 This is a simplified illustration of the syntax for VUI parameters.

[0016] Figure 8 This is a simplified illustration of the syntax for VUI parameters.

[0017] Figure 9 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0018] Figure 10 This is a diagram illustrating an example of the correspondence between index values ​​and wavelength information.

[0019] Figure 11 This is a simplified diagram illustrating an example of the syntax for VUI parameters.

[0020] Figure 12 This is a simplified illustration of the syntax for VUI parameters.

[0021] Figure 13 This is a diagram illustrating an example of the correspondence between the values ​​of the representation type information and the interpretation of each value.

[0022] Figure 14 This is a simplified illustration of the syntax for VUI parameters.

[0023] Figure 15 This is a diagram illustrating an example of the correspondence between the values ​​of the interpolation information and their interpretations.

[0024] Figure 16 This is a diagram illustrating an example of the structure of a bitstream.

[0025] Figure 17 This diagram is a simplified representation of the circuitry of a decoding device.

[0026] Figure 18 This is a flowchart illustrating the processing performed by the circuitry of the decoding device.

[0027] Figure 19 This is a simplified diagram illustrating an example of table information.

[0028] Figure 20 This is a simplified diagram illustrating an example of table information.

[0029] Figure 21 This is a block diagram illustrating an example of the functional structure of the encoding section.

[0030] Figure 22 This is a diagram illustrating an example of the hierarchical structure of data in a stream.

[0031] Figure 23 This is a block diagram illustrating an example of the functional configuration of the decoding unit. Detailed Implementation

[0032] (The understanding that forms the basis of this disclosure)

[0033] The image processing system described in the background technology includes an encoding device and a decoding device. In the encoding device, various modalities of input images are taken as input. The encoding device extracts features from the input images and inputs the extracted features to the decoding device. The decoding device performs an image parsing task based on the input features, thereby outputting a segmentation map.

[0034] The tasks performed by the decoding device include both human vision and machine tasks. Human vision involves visual recognition or audiovisual processing based on dynamic images of operators or users. Machine tasks include a wide variety of tasks such as object detection, object tracking, object segmentation, action recognition, and pose estimation using AI models.

[0035] In the prior art, modal information is not transmitted from the encoding device to the decoding device. Therefore, sometimes the optimal task processing corresponding to the image category of the input image is not selected, resulting in reduced execution accuracy of the task processing.

[0036] In order to solve this problem, the inventors came to the following understanding and thus conceived of this disclosure: by including modal information representing the image category in the bitstream and sending it from the encoding device to the decoding device, the decoding device performs task processing based on the modal information, thereby solving the above-mentioned problem.

[0037] The various methods disclosed herein will now be explained.

[0038] The decoding apparatus according to the first aspect of this disclosure includes a circuit and a memory connected to the circuit. The circuit acquires an image from a bitstream and establishes a first parameter associated with the image. The first parameter includes modal information representing the image category, which indicates that the image category is at least one of a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension.

[0039] According to the first method, modal information representing the image category can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby improving the execution accuracy of the task processing performed by the decoding device. Furthermore, since modal information representing an undefined image category or a preliminary image category for future expansion is included, it is also possible to address situations where the image characteristics are unclear in the encoding device and the image category cannot be determined, or future expansion of the image category.

[0040] In the first embodiment, the decoding apparatus according to the second aspect of this disclosure may involve the circuit obtaining a second parameter from the bitstream, the second parameter including wavelength information representing the band of light radiation of the image, wherein the wavelength information can represent bands spanning multiple image categories when the modal information indicates that the image category is not defined.

[0041] According to the second method, wavelength information representing the light radiation bands of an image can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby further improving the execution accuracy of task processing performed by the decoding device. Furthermore, by not defining image categories in the modal information, it is also possible to handle situations where the image bands span multiple image categories.

[0042] In the third aspect of this disclosure, the decoding device may, in the first or second aspect, involve the circuit obtaining a third parameter from the bitstream, the third parameter including: a flag indicating whether color information of the image is included, and the color information. If the modal information indicates that the image category is an infrared image or an ultraviolet image, the circuit performs: (1) obtaining the flag indicating that the color information is not included, or (2) ignoring the color information even if the flag indicating that the color information is included is obtained.

[0043] According to the third method, when the modal information represents an image category of infrared or ultraviolet, the circuit does not acquire color information, thereby avoiding the execution of useless processing.

[0044] In any of the first to third methods, the decoding apparatus involved in the fourth method of this disclosure may be such that the circuit obtains a fourth parameter from the bit stream, the fourth parameter including: extended bit information corresponding to the prepared image category when the modal information indicates that the image category is an extended prepared image category.

[0045] According to the fourth method, extended bit information corresponding to the prepared image category for extension can be transmitted from the encoding device to the decoding device, thereby enabling the decoding device to appropriately perform task processing corresponding to the prepared image category.

[0046] The decoding apparatus involved in the fifth aspect of this disclosure, in the fourth aspect, may include, where the extended bit information comprises: information representing the bit length of additional information related to the prepared image category; and the additional information having the bit length.

[0047] According to the fifth method, the extended bit information can be accurately transmitted from the encoding device to the decoding device.

[0048] In any of the first to fifth embodiments, the decoding apparatus according to the sixth aspect of this disclosure may, when the modal information indicates that the image category is an infrared image, have the circuit obtain performance type information from the bit stream indicating the performance type of infrared radiation intensity, wherein the performance type information indicates at least one of the following: the pixel value of the image represents the intensity value of infrared radiation intensity, and the pixel value of the image represents the temperature value converted from the intensity value of infrared radiation intensity.

[0049] According to the sixth method, the representation type information indicating the representation type of infrared radiation intensity can be transmitted from the encoding device to the decoding device, thereby enabling the decoding device to appropriately perform task processing using the infrared image.

[0050] In the sixth embodiment, the decoding apparatus involved in the seventh aspect of this disclosure may, when the representation type information indicates that the pixel value indicates the temperature value, involve the circuit obtaining temperature information from the bitstream, including the lowest and highest temperatures of the temperature value.

[0051] According to the seventh method, by using temperature information including the lowest and highest temperatures, temperature information can be accurately transmitted from the encoding device to the decoding device.

[0052] In the seventh method, the decoding apparatus involved in the eighth method of this disclosure may include, respectively, information representing positive or negative signs for the lowest temperature and the highest temperature, and information representing the mantissa and the exponent in the exponential representation using a given base.

[0053] According to method 8, the minimum and maximum temperatures can be appropriately represented in the bitstream using exponential representation.

[0054] In any of the first to fifth methods, the decoding apparatus involved in the ninth method of this disclosure may be such that the circuit obtains flag information from the bitstream, the flag information indicating whether the pixel value of the image is represented using virtual color representation, and when the flag information indicates that the pixel value is represented using virtual color representation, the circuit performs the following: obtaining generation information from the bitstream, the generation information being used to generate table information that establishes a correspondence between the pixel value and the color code of the virtual color representation, and generating the table information based on the generation information.

[0055] According to the ninth method, the decoding device can appropriately generate table information that establishes a correspondence between pixel values ​​and color codes representing virtual colors based on the generation information obtained from the bitstream.

[0056] In the 9th method, the decoding apparatus involved in the 10th method of this disclosure may include the generation information comprising: multiple groups of the pixel value and the color code, and interpolation information for interpolating the values ​​between the multiple groups.

[0057] According to the 10th method, the decoding device can generate table information appropriately and with high precision based on multiple groups of pixel values ​​and color codes, and interpolation information for interpolating values ​​between these multiple groups.

[0058] In any of the 1st to 10th embodiments of this disclosure, the decoding apparatus may, during the acquisition of the first parameter, be such that the circuit acquires the first parameter from a given header region of the bit stream, the given header region including VUI or SEI.

[0059] According to method 11, the decoding device can easily obtain the first parameter from a given header region of the bitstream.

[0060] The encoding apparatus according to the 12th aspect of this disclosure includes: a circuit; and a memory connected to the circuit, the circuit encoding an image and a first parameter associated with the image into a bit stream, the first parameter including modal information representing an image category of the image, the modal information representing that the image category is at least one of a visible light image, an infrared image, an ultraviolet image, an undefined image, and a pre-defined image category for extension.

[0061] According to the 12th method, modal information representing the image category can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby improving the execution accuracy of the task processing performed by the decoding device. Furthermore, since modal information representing an undefined image category or a preliminary image category for future expansion is included, it is also possible to address situations where the image characteristics are unclear in the encoding device and the image category cannot be determined, or future expansion of the image category.

[0062] In the 12th embodiment, the encoding apparatus involved in the 13th embodiment of this disclosure may be such that the circuit encodes a second parameter into the bitstream, the second parameter including wavelength information representing a band of light radiation of the image, wherein the wavelength information can represent a band spanning multiple image categories when the modal information indicates that the image category is not defined.

[0063] According to the 13th method, wavelength information representing the light radiation bands of an image can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby further improving the execution accuracy of task processing performed by the decoding device. Furthermore, by not defining image categories in the modal information, it is also possible to handle situations where the image bands span multiple image categories.

[0064] In the 12th or 13th embodiment, the encoding device involved in the 14th embodiment of this disclosure may be such that the circuit encodes the third parameter into the bit stream, the third parameter including: a flag indicating whether color information of the image is included, and the color information. If the image is an infrared image or an ultraviolet image, the circuit includes the flag indicating that the color information is not included in the third parameter.

[0065] According to the 14th method, when the image is an infrared image or an ultraviolet image, the marking information that does not include color information is included in the 3rd parameter, thereby avoiding the execution of useless processing by the decoding device.

[0066] In any of the 12th to 14th embodiments, the encoding apparatus involved in the 15th aspect of this disclosure may be such that the circuit encodes the fourth parameter into the bit stream, the fourth parameter including: extended bit information corresponding to the prepared image category when the modality information indicates that the image category is an extended prepared image category.

[0067] According to the 15th method, extended bit information corresponding to the prepared image category for extension can be transmitted from the encoding device to the decoding device, thereby enabling the decoding device to appropriately perform task processing corresponding to the prepared image category.

[0068] In the 15th embodiment, the encoding apparatus involved in the 16th embodiment of this disclosure may include, in which the extended bit information comprises: information representing the bit length of additional information related to the prepared image category; and the additional information having the bit length.

[0069] According to method 16, the extended bit information can be accurately transmitted from the encoding device to the decoding device.

[0070] In any of the 12th to 16th embodiments, the encoding device according to the 17th aspect of this disclosure may, when the image is an infrared image, encode representation type information representing the representation type of infrared radiation intensity into the bit stream, wherein the representation type information represents at least one of: the pixel value of the image representing the intensity value of infrared radiation intensity, and the pixel value of the image representing the temperature value converted from the intensity value of infrared radiation intensity.

[0071] According to the 17th method, the representation type information indicating the representation type of infrared radiation intensity can be transmitted from the encoding device to the decoding device, thereby enabling the decoding device to appropriately perform task processing using infrared images.

[0072] In the 17th embodiment, the encoding apparatus involved in the 18th embodiment of this disclosure may, when the representation type information indicates that the pixel value indicates that the temperature value, encode temperature information including the lowest temperature and the highest temperature of the temperature value into the bit stream.

[0073] According to method 18, the temperature information includes the lowest and highest temperatures, thereby enabling accurate transmission of temperature information from the encoding device to the decoding device.

[0074] In the 18th embodiment, the encoding device involved in the 19th embodiment of this disclosure may include, respectively, information representing a positive or negative sign for the lowest temperature and the highest temperature, and information representing a mantissa and an exponent in an exponential representation using a given base.

[0075] According to method 19, the minimum and maximum temperatures can be appropriately represented in the bitstream using exponential representation.

[0076] In any of the 12th to 16th embodiments, the encoding device involved in the 20th embodiment of this disclosure may be such that the circuit encodes flag information into the bit stream, the flag information indicating whether the pixel value of the image is represented using virtual color representation, and when the flag information indicates that the pixel value is represented using virtual color representation, the circuit encodes generation information into the bit stream, the generation information being used to generate table information that establishes a correspondence between the pixel value and the color code of the virtual color representation.

[0077] According to method 20, the encoding device encodes the generated information into a bit stream, thereby enabling the decoding device to appropriately generate table information that establishes a correspondence between pixel values ​​and color codes representing virtual colors based on the generated information obtained from the bit stream.

[0078] In the 20th embodiment, the encoding apparatus involved in the 21st embodiment of this disclosure may include, in the case of, the generation information comprising: multiple groups of the pixel value and the color code, and interpolation information for interpolating the values ​​between the multiple groups.

[0079] According to the 21st method, the generated information includes multiple groups of pixel values ​​and color codes, and interpolation information for interpolating the values ​​between the multiple groups, thereby enabling the decoding device to generate table information appropriately and with high precision based on this information obtained from the bitstream.

[0080] In any of the 12th to 19th embodiments, the encoding apparatus involved in the 22nd aspect of this disclosure may be, in the encoding of the first parameter, the circuit encodes the first parameter into a given header region of the bit stream, the given header region including VUI or SEI.

[0081] According to method 22, the decoding device can easily obtain the first parameter from a given header region of the bitstream.

[0082] In the decoding method according to the 23rd aspect of this disclosure, the decoding device acquires an image from a bitstream and a first parameter associated with the image, the first parameter including modal information representing the image category, the modal information representing that the image category is at least one of a visible light image, an infrared image, an ultraviolet image, an undefined image, and a pre-defined image category for extension.

[0083] According to the 23rd method, modal information representing the image category can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby improving the execution accuracy of the task processing performed by the decoding device. Furthermore, since modal information representing an undefined image category or a preliminary image category for future expansion is included, it is also possible to address situations where the image characteristics are unclear in the encoding device and the image category cannot be determined, or future expansion of the image category.

[0084] In the encoding method according to the 24th aspect of this disclosure, the encoding device encodes an image and a first parameter associated with the image into a bit stream, the first parameter including modal information representing the image category, the modal information representing that the image category is at least one of visible light image, infrared image, ultraviolet image, undefined, and extended preparatory image category.

[0085] According to the 24th method, modal information representing the image category can be included in the bitstream and transmitted from the encoding device to the decoding device, thereby improving the execution accuracy of the task processing performed by the decoding device. Furthermore, since modal information representing an undefined image category or a preliminary image category for future expansion is included, it is also possible to address situations where the image characteristics are unclear in the encoding device and the image category cannot be determined, or future expansion of the image category.

[0086] (Implementation of this disclosure)

[0087] Hereinafter, embodiments of the present disclosure will be described in detail using the accompanying drawings. Furthermore, elements labeled with the same symbols in different drawings represent the same or corresponding elements.

[0088] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, constituent elements in the following embodiments that are not described as constituent elements of independent technical solutions representing the highest-level concept are described as arbitrary constituent elements. Furthermore, in all embodiments, the contents can be substituted or combined. In addition, these overall or specific methods can be implemented by systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, or recording media.

[0089] Figure 1 This diagram is a simplified representation of the structure of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, and a transmission path NW.

[0090] In the encoding device 1, image data D1 is input from an external device. The external device includes a camera or similar device that captures moving images. The external device inputs the image data D1 of the captured moving images into the encoding device 1.

[0091] Encoding device 1 generates a bitstream BS based on image data D1. Here, a bitstream refers to a data string or flow of digital data. A bitstream (or simply a stream) can be a single stream or composed of multiple streams divided into multiple levels. Furthermore, a bitstream can be transmitted via serial communication on a single transmission path or via packet communication on multiple transmission paths. Encoding device 1 sends the generated bitstream BS to decoding device 2 via transmission path NW. Decoding device 2 receives the bitstream BS.

[0092] Decoding device 2 decodes image data D1 based on bitstream BS, and performs task processing based on the decoded image data D1. Task processing includes human vision and machine tasks. Human vision involves visual recognition or audiovisual processing of dynamic images from operators or users. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using machine learning-based inference models, i.e., artificial intelligence (AI) models. The task processing unit performing human vision tasks includes display devices such as liquid crystal displays or organic EL displays. The task processing unit performing machine tasks includes an inferrer using AI.

[0093] The transport path NW can be the Internet, a WAN (wide area network), a LAN (local area network), or any combination thereof. Ideally, the transport path NW should be a dedicated network that ensures secure communication through access restrictions.

[0094] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 is configured to include a processor such as a CPU. The memory 12 is configured to include any recording medium such as ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores the data processed by the circuit 11 or the data processed during the process.

[0095] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 is configured to include a processor such as a CPU. The memory 22 is configured to include any recording medium such as ROM, RAM, HDD, SSD, or semiconductor memory. The memory 22 stores the data processed by the circuit 21 or the data processed during the process.

[0096] Figure 2 This diagram shows a simplified representation of the structure of the circuit 11 included in the encoding device 1. The circuit 11 includes an acquisition unit 31, a setting unit 32, an encoding unit 33, and a transmission unit 34.

[0097] Next, the coding unit 33 involved in this embodiment will be described. Figure 21 This is a block diagram illustrating an example of the functional configuration of the encoding unit 33 according to this embodiment. The encoding unit 33 encodes the image in blocks.

[0098] like Figure 21As shown, the encoding unit 33 includes: a segmentation unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy coding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Furthermore, the intra-frame prediction unit 124 and the inter-frame prediction unit 126 are configured as part of the prediction processing unit 125.

[0099] For example, Figure 21 The multiple components of the coding unit 33 shown are obtained through Figure 1 The circuit 11 and memory 12 shown are installed.

[0100] Circuit 11 is configured to include a processor such as a CPU. Circuit 11 can be a dedicated or general-purpose electronic circuit for encoding images, or it can be a collection of multiple electronic circuits. Alternatively, for example, circuit 11 can also implement... Figure 21 The coding unit 33 shown has multiple components, including the components other than those used for storing information.

[0101] The memory 12 can be a dedicated or general-purpose electronic circuit for storing information, or it can be a collection of multiple electronic circuits. The memory 12 can be externally connected to the circuit 11, or it can be built into the circuit 11. In addition, the memory 12 can be a disk or an optical disk, or it can be a storage device or a recording medium. Furthermore, the memory 12 can be non-volatile memory or volatile memory.

[0102] The memory 12 can store encoded images or the stream corresponding to the encoded images. Additionally, the memory 12 can also store a program for the processor to perform image encoding processing.

[0103] Alternatively, memory 12 can also be implemented. Figure 21 The encoding unit 33 shown here functions as one of the multiple components used for storing information. Specifically, the memory 12 can also be implemented... Figure 21 The block memory 118 and frame memory 122 shown below illustrate their functions. More specifically, the memory 12 can also store reconstructed images (specifically, reconstructed blocks or reconstructed images, etc.).

[0104] Furthermore, in the coding section 33, Figure 21 Some of the multiple components shown may be omitted from installation, and some of the multiple processes performed by these components may also be omitted from execution. Alternatively, Figure 21A portion of the multiple components shown may also be mounted on other devices, and a portion of the multiple processes performed by the multiple components may also be performed by other devices.

[0105] Figure 3 This is a flowchart showing the process performed by the circuit 11 of the encoding device 1.

[0106] First, in step SP11, the acquisition unit 31 acquires image data D11, representing the processing object, i.e., image Q, input from an external device. Image data D11 is equivalent to... Figure 1 The image data D1 shown is shown.

[0107] Next, in step SP12, the setting unit 32 sets parameter P in association with the image Q. Parameter P includes a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 includes modal information indicating the image category of image Q. The second parameter P2 includes wavelength information indicating the band of light radiation of image Q. The third parameter P3 includes color information of image Q. The fourth parameter P4 includes modal information indicating the extended bit information corresponding to the prepared image category when the image category is an extended version. The setting unit 32 inputs data D12 including the setting information of parameter P to the encoding unit 33.

[0108] The first parameter P1 includes modal information representing the image category of image Q. The image category may include at least one of visible light images, infrared images, and ultraviolet images. Visible light images include natural images or RGB images, used for providing detailed color information in human vision or machine tasks. Infrared images include images captured using an infrared camera, used for shooting in low light. Ultraviolet images include images captured using an ultraviolet camera, used for detecting scratches on products. The modal information indicates that the image category is at least one of visible light images, infrared images, and ultraviolet images. Additionally, the modal information may indicate at least one of undefined image categories and one of the reserved image categories for expansion. In other words, the modal information indicates that the image category is at least one of visible light images, infrared images, ultraviolet images, undefined images, and one of the reserved image categories for expansion. Furthermore, the modal information may also include other image categories. The setting unit 32 can set the first parameter P1 based on image analysis of image data D11, or it can set the first parameter P1 based on setting information input by the operator of the encoding device 1.

[0109] Next, in step SP13, the encoding unit 33 encodes the image Q represented by the image data D11 input from the acquisition unit 31 into a bit stream BS.

[0110] Next, in step SP14, the encoding unit 33 encodes the parameter P represented by the data D12 input from the setting unit 32 into the bitstream BS. Here, encoding the parameter P into the bitstream BS can also be described as storing the parameter P in the bitstream BS, or simply storing the parameter P in the bitstream BS. Furthermore, the execution order of steps SP13 and SP14 can also be... Figure 3 The opposite of the example can also be executed simultaneously.

[0111] Next, in step SP15, the transmitting unit 34 transmits the bit stream BS input from the encoding unit 33 to the decoding device 2 via the transmission path NW.

[0112] Figure 4 This diagram illustrates a simplified structure of the bitstream BS. The bitstream BS has a header region 41 and a payload region 42. The encoding unit 33 stores the encoded data of the image Q in the payload region 42 and the encoded data of the parameter P associated with the image Q in the header region 41.

[0113] The encoding unit 33 can also encode the encoded data of parameter P into a given region 43 within the header region 41. The given region 43 can be VUI (video usability information) or SEI (Supplemental Enhancement Information). This specification describes an example using the VUI parameter; in the case of SEI, the VUI parameter can be referred to as the SEI parameter. However, the given region 43 is not limited to VUI or SEI; it can also be VPS, SPS, PPS, PH, SH, APS, tile header, or system layer header, etc.

[0114] Figure 22 This is a diagram illustrating an example of the hierarchical structure of data in a stream. A stream, for example, includes video sequences. Figure 22 As shown in (A), the video sequence includes, for example, VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information), and multiple pictures.

[0115] VPS includes encoding parameters common to multiple layers in a dynamic image composed of multiple layers, and encoding parameters associated with the multiple layers or individual layers included in the dynamic image.

[0116] The SPS includes parameters for the sequence, that is, encoding parameters referenced by the decoding device 2 for decoding the sequence. These encoding parameters may, for example, represent the width or height of the image. Furthermore, multiple SPSs may exist.

[0117] The PPS includes parameters for the images, that is, the encoding parameters that the decoding device 2 refers to in order to decode each image in the sequence. These encoding parameters may, for example, include a reference value for the quantization width used in image decoding and a flag indicating the application of weighted prediction. Furthermore, multiple PPSs may exist. The SPS and PPS are sometimes simply referred to as a parameter set.

[0118] like Figure 22 As shown in (B), the image includes an image header and one or more slices. The image header includes encoding parameters referenced by the decoding device 2 for decoding the one or more slices.

[0119] like Figure 22 As shown in (C), the slice includes a slice header and one or more bricks. The slice header includes encoding parameters referenced by the decoding device 2 for decoding the one or more bricks.

[0120] like Figure 22 As shown in (D), the bricks include more than one CTU (Coding Tree Unit).

[0121] Alternatively, the image may exclude slices and include a set of tiles instead. In this case, the set of tiles includes more than one tile. Additionally, slices may also be included within the brickwork.

[0122] CTU is also known as a superblock or basic partitioning unit. For example... Figure 22 As shown in (E), the CTU includes a CTU header and one or more CUs (Coding Units). The CTU header includes encoding parameters referenced by the decoding device 2 for decoding the one or more CUs.

[0123] A CU can also be divided into multiple smaller CUs. Additionally, as... Figure 22As shown in (F), the CU includes a CU header, prediction information, and residual coefficient information. Prediction information is used to predict the CU. Residual coefficient information represents the prediction residuals. Furthermore, the CU is essentially the same as a PU (Prediction Unit) or TU (Transform Unit), but it can also include multiple TUs smaller than the CU. Additionally, the CU can be processed at the level of each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is, for example, a fixed unit that can be processed in one stage during pipelined processing in hardware.

[0124] Furthermore, a flow may not have Figure 22 This is a subset of the multiple hierarchical levels shown. Furthermore, the order of these levels can be rearranged, and any level can be replaced with another level.

[0125] The image that is the object of the processing performed by the encoding device 1 or decoding device 2 at the current point in time is called the current image. If the processing is encoding, the current image is synonymous with the image to be encoded; if the processing is decoding, the current image is synonymous with the image to be decoded. Furthermore, the block (CU or a block of CU) that is the object of the processing performed by the encoding device 1 or decoding device 2 at the current point in time is called the current block. If the processing is encoding, the current block is synonymous with the block to be encoded; if the processing is decoding, the current block is synonymous with the block to be decoded.

[0126] Figure 5 This is a simplified illustration of the syntax for VUI parameters, including the first parameter P1 and the third parameter P3.

[0127] The first parameter P1 includes a flag indicating whether modal information is recorded in the bitstream BS (vui_modality_info_present_flag). A value of 1 for vui_modality_info_present_flag indicates the presence of modal information within the VUI parameter; a value of 0 indicates the absence of modal information within the VUI parameter. When vui_modality_info_present_flag indicates the presence of modal information in the bitstream BS, the modal information is represented by the value of the vui_modality_type identifier included in the VUI parameter.

[0128] Figure 6This is an example diagram showing the correspondence between the values ​​of `vui_modality_type` and image categories. For example... Figure 6 As shown, a value of 0 for `vui_modality_type` indicates that the image category of image Q is undefined (i.e., undefined). A value of 1 for `vui_modality_type` indicates that the image category of image Q is a visible light image. A value of 2 for `vui_modality_type` indicates that the image category of image Q is an infrared image. A value of 3 for `vui_modality_type` indicates that the image category of image Q is an ultraviolet image. Other values ​​for `vui_modality_type` (4-255 in this example) represent reserved areas ensured for future extended use. Furthermore, the value can be increased based on the processing object, i.e., image Q. Figure 6 The number of image categories defined in the code. Additionally, the absence of `vui_modality_type` can also mean that the value of `vui_modality_type` is 0. Furthermore, a value of 0 for `vui_modality_type` allows for the definition of wavelength information spanning a wider band, such as for hyperspectral images, through user settings.

[0129] like Figure 5As shown, the third parameter P3 includes a flag indicating whether color information is recorded in the bitstream BS (vui_colour_description_present_flag). A value of 1 for vui_colour_description_present_flag indicates the presence of color information within the VUI parameter; a value of 0 indicates the absence of color information within the VUI parameter. (Illustrations omitted, but color information includes vui_colour_primaries, vui_trasfer_characteristics, and vui_matrix_coeffs, etc.) Furthermore, in the case of infrared or ultraviolet images, the third parameter P3 may not include color information. In the encoding process, encoding device 1 sets the value of `vui_colour_description_present_flag` to 1 and records the color information in the VUI parameters when the image category represented by the modal information is a visible light image. On the other hand, when the image category represented by the modal information is an infrared image or an ultraviolet image, it sets the value of `vui_colour_description_present_flag` to 0 and does not record the color information in the VUI parameters. Furthermore, in the decoding process, decoding device 2 obtains the color information from the bitstream BS when the image category represented by the modal information is a visible light image. On the other hand, when the image category represented by the modal information is an infrared image or an ultraviolet image, it may not obtain the color information from the bitstream BS, or it may ignore the value even if it obtains the color information from the bitstream BS.

[0130] Figure 7 This is a simplified illustration of the syntax for VUI parameters, including the fourth parameter P4.

[0131] The fourth parameter, P4, includes extension bit information corresponding to the prepared image category when the modal information indicates that the image category is an extension. The extension bit information includes information indicating the bit length of the appended information needed for the prepared image category (vui_modality_type_extension_bits), and appended information with that bit length (vui_reserved_modality_type_extension). When vui_modality_type_extension_bits is 0, it indicates that vui_reserved_modality_type_extension is not recorded in the bitstream BS. Conversely, when vui_modality_type_extension_bits is greater than 0, it indicates that vui_reserved_modality_type_extension and its bit length are recorded in the bitstream BS. The maximum value of vui_modality_type_extension_bits can be arbitrarily set, for example, to 2048.

[0132] Figure 8 This is a simplified illustration of the first example of the syntax for VUI parameters, including the second parameter P2. The second parameter P2 includes wavelength information representing the band of light radiation from image Q.

[0133] exist Figure 8 In the example shown, the second parameter P2 includes a flag indicating whether wavelength information is recorded in the bitstream BS (vui_spectrum_range_present_flag). A value of 0 for vui_spectrum_range_present_flag indicates that the spectral band of the light radiation wavelength represented by the CLVS (Coded Layer Video Sequence) image does not exist within the VUI parameter; a value of 1 for vui_spectrum_range_present_flag indicates that the spectral band of the light radiation wavelength represented by the CLVS image exists within the VUI parameter.

[0134] When vui_spectrum_range_present_flag is set to 1, the wavelength information includes the minimum and maximum wavelengths (minimum wavelengths) representing the bands of image Q. Both the minimum and maximum wavelengths are in meters.

[0135] The information representing the minimum value of a band includes information about the mantissa of the minimum value in the exponential representation using a given base (e.g., 10), and information about the value obtained by adding 15 to the exponent of the minimum value (vui_min_wavelength_exponent_plus15). The minimum value of a band, Min_Wavelength, is calculated as Min_Wavelength = vui_min_wavelength_mantissa × 10. vui_min_wavelength_exponent_plus15-15 This indicates. Therefore, in Figure 8 In the example shown, it is possible to achieve 10 -15 ~10 ＋16 The range of (m) specifies the minimum value. Here, if vui_min_wavelength_mantissa is 0 or does not exist, it means the minimum value of the band is unknown or uncertain, and the minimum value is undefined, or defined by a method other than exponential representation. In this case, during decoding, decoding device 2 may not obtain vui_min_wavelength_exponent_plus15 from the bitstream BS, or it may ignore this value even if it does obtain vui_min_wavelength_exponent_plus15 from the bitstream BS.

[0136] Information indicating the maximum value of a band includes information representing the mantissa of the maximum value in the exponential representation using a given base (e.g., 10), and information representing the value obtained by adding 15 to the exponent of the maximum value (vui_max_wavelength_exponent_plus15). The maximum value of a band, Max_Wavelength, is calculated as Max_Wavelength = vui_max_wavelength_mantissa × 10. vui_max_wavelength_exponent_plus15–15 This indicates. Therefore, in Figure 8 In the example shown, it is possible to achieve 10 -15 ~10 ＋16The range of (m) specifies the maximum value. Here, if vui_max_wavelength_mantissa is 0 or does not exist, it means the maximum value of the band is unknown or uncertain, and the maximum value is undefined, or defined by a method other than exponential representation. In this case, the decoding device 2 may either not obtain vui_max_wavelength_exponent_plus15 from the bitstream BS, or it may ignore this value even if it does obtain vui_max_wavelength_exponent_plus15 from the bitstream BS.

[0137] Both `vui_min_wavelength_mantissa` and `vui_max_wavelength_mantissa` have at least three significant bits in decimal. Figure 8 In the examples shown, both `vui_min_wavelength_mantissa` and `vui_max_wavelength_mantissa` have a bit length of 10 bits in binary, but they can also have any bit length of 11 bits or more. Similarly, in... Figure 8 In the examples shown, both `vui_min_wavelength_exponent_plus15` and `vui_max_wavelength_exponent_plus15` have a bit length of 5 bits in binary, but they can also have any bit length of 6 bits or more. Furthermore, the addition of exponents is not limited to "+15"; it can also be the syntax for adding other fixed values.

[0138] Furthermore, in encoding device 1, the minimum value of the band represented by vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15 is set to be greater than or equal to the shortest wavelength corresponding to the image category represented by the modal information. Similarly, in encoding device 1, the maximum value of the band represented by vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15 is set to be less than or equal to the longest wavelength corresponding to the image category represented by the modal information.

[0139] Figure 9 This is a simplified diagram illustrating the second example of the syntax for the VUI parameters, including the second parameter P2.

[0140] exist Figure 9In the example shown, the second parameter P2 includes information about the index value representing the wavelength information (vui_spectrum_range).

[0141] Figure 10 This diagram illustrates an example of the correspondence between index values ​​and wavelength information. Minimum (minimum wavelength) and maximum (maximum wavelength) values ​​for each band are predefined and shared by encoding device 1 and decoding device 2, respectively. Figure 10 In the example shown, index values ​​2-6 correspond to the wavelength region of ultraviolet light, index value 7 corresponds to the wavelength region of visible light, and index values ​​8-10 correspond to the wavelength region of infrared light.

[0142] When the index value is, for example, 2, the minimum wavelength is defined as 1*10. -9 (m), the maximum wavelength is defined as 1*10 -7 (m), in the case of an index value of, for example, 3, the minimum wavelength is defined as 1*10. -7 (m), the maximum wavelength is defined as 19*10 -8 (m).

[0143] Additionally, when the index value is a specific value (in) Figure 10 In the example shown, a value of 0 indicates that the minimum and maximum wavelengths are not defined.

[0144] Additionally, when the index value is another specific value (in) Figure 10 In the example shown, if it is 1), it means that the desired minimum and maximum wavelengths are defined by the user settings, etc.

[0145] In addition, the index values ​​of the other values ​​(11-255 in this example) represent areas reserved for future extended use.

[0146] Furthermore, in the encoding device 1, the index value is set according to the band corresponding to the image category represented by the modal information. For example, when vui_modality_type is 1 (visible light image), vui_spectrum_range is set to any one of 0, 1, or 7. Similarly, when vui_modality_type is 2 (infrared image), vui_spectrum_range is set to any one of 0, 1, or 8 to 10. And when vui_modality_type is 3 (ultraviolet image), vui_spectrum_range is set to any one of 0, 1, or 2 to 6. In other words, by setting an appropriate index value according to the image category in the encoding device 1, the decoding device 2 can determine the appropriate band to decode the image based on the index value.

[0147] Figure 11 This is a simplified diagram illustrating the third example of the syntax for the VUI parameters, including the second parameter P2.

[0148] exist Figure 9 In the example shown, when the index value of vui_spectrum_range is set to 1 (user-defined), compared to Figure 8 Similarly, the minimum value of the band can be defined using vui_min_wavelength_mantissa and vui_min_wavelength_exponent_plus15, and the maximum value of the band can be defined using vui_max_wavelength_mantissa and vui_max_wavelength_exponent_plus15.

[0149] Figure 12 This is a simplified diagram illustrating the first variation of the VUI parameter syntax.

[0150] When the image category represented by the modal information is an infrared image, the encoding device 1 can also further encode the representation type information (vui_infrared_radiance_representation_type) representing the representation type of infrared radiation intensity into the bit stream BS.

[0151] Figure 13 This is a diagram illustrating an example of the correspondence between the values ​​of the representation type information and the interpretation of each value.

[0152] If vui_infrared_radiance_representation_type is 0 or does not exist, it means that the representation type of infrared radiation intensity is unknown or uncertain, and the representation type is not defined, or it is defined by a method other than intensity value representation or temperature value representation.

[0153] When `vui_infrared_radiance_representation_type` is set to 1, it means that the infrared radiation intensity is represented by an intensity value. In the intensity value representation, each pixel value in image Q represents the intensity value of the infrared radiation.

[0154] When the value of `vui_infrared_radiance_representation_type` is 2, it means that the infrared radiation intensity is represented as a temperature value. In the temperature value representation, each pixel value of image Q represents a temperature value converted from the intensity value of infrared radiation.

[0155] Other values ​​(3 in this case) for `vui_infrared_radiance_representation_type` indicate the prepared area to be used for future expansion.

[0156] In addition, such as Figure 12 As shown, when the value of vui_infrared_radiance_representation_type is 2 (that is, when it is represented as a temperature value), the encoding device 1 records the temperature information in the VUI parameters. The temperature information includes the lowest and highest temperatures as floating decimal values.

[0157] The information representing the minimum temperature includes: information indicating the sign of the minimum temperature in an exponential representation using a given base (e.g., 2) (vui_min_temperature_sign), information representing the exponent of the minimum temperature (vui_min_temperature_exponent), information representing the value obtained by subtracting 1 from the bit length of the minimum temperature's mantissa (vui_min_temperature_mantissa_len_minus1), and information representing the mantissa of the minimum temperature (vui_min_temperature_mantissa). Furthermore, vui_min_temperature_mantissa_len_minus1 can be omitted by setting the bit length of the minimum temperature's mantissa to a fixed value.

[0158] When the value of vui_min_temperature_sign is 0, the sign of the lowest temperature is positive; when the value of vui_min_temperature_sign is 1, the sign of the lowest temperature is negative.

[0159] The information representing the maximum temperature includes: information indicating the sign of the maximum temperature in an exponential representation using a given base (e.g., 2) (vui_max_temperature_sign), information representing the exponent of the maximum temperature (vui_max_temperature_exponent), information representing the value obtained by subtracting 1 from the bit length of the mantissa of the maximum temperature (vui_max_temperature_mantissa_len_minus1), and information representing the mantissa of the maximum temperature (vui_max_temperature_mantissa). Furthermore, vui_max_temperature_mantissa_len_minus1 can be omitted by setting the bit length of the mantissa of the maximum temperature to a fixed value.

[0160] When the value of vui_max_temperature_sign is 0, the sign of the highest temperature is positive; when the value of vui_max_temperature_sign is 1, the sign of the highest temperature is negative.

[0161] Figure 14 This is a simplified illustration of the second variation of the VUI parameter syntax.

[0162] The encoding device 1 can also further encode the flag information (vui_false_colour_representation_present_flag) indicating whether the pixel values ​​of the image Q are represented using virtual color representation into the bitstream BS.

[0163] When vui_false_colour_representation_present_flag is 1, it means that the pixel values ​​of image Q are represented using virtual color representation. When vui_false_colour_representation_present_flag is 0, it means that the pixel values ​​of image Q are not represented using virtual color representation.

[0164] When the pixel values ​​of image Q are represented using virtual color representation, the generation information used to generate a table that establishes a correspondence between each pixel value of image Q and the color code of the virtual color representation is recorded in the VUI parameters.

[0165] The generated information includes: information representing the number of color codes minus 2 from the encoded bitstream BS (vui_num_hex_colour_code_minus2), information representing the number of groups of pixel values ​​and color codes minus 2 (vui_num_value_minus2), interpolation information used to interpolate the values ​​between these groups (vui_colour_interpolation_type), and information representing the i-th hexadecimal color code (vui_hex_colour_code[i]). Each hexadecimal color code is 24 bits long, with the first 8 bits representing the shade of red, the middle 8 bits representing the shade of green, and the last 8 bits representing the shade of blue. For example, vui_hex_colour_code 000000 represents black (RGB: 0, 0, 0), and vui_hex_colour_code FFFFFF represents white (RGB: 255, 255, 255).

[0166] If the value of vui_num_hex_colour_code_minus2 is 0, it means that the number of color codes encoded into the bitstream BS is 2.

[0167] If the value of `vui_num_value_minus2` is, for example, 4, it means that there are 6 groups of pixel values ​​and color codes. The number of color codes encoded into the bitstream is 2, and the number of groups is 6. This means that based on the values ​​of the 2 color codes, the range of color components is divided into 5 equal parts, and the values ​​of the middle 4 groups of color codes are interpolated to generate a table containing a total of 6 groups of pixel values ​​and color codes.

[0168] Figure 15 This is a diagram illustrating an example of the correspondence between the values ​​of the interpolation information and their interpretations.

[0169] If vui_colour_interpolation_type is 0 or does not exist, it means that the interpolation type is unknown or uncertain, and the interpolation type is not defined, or it is defined by a method other than linear interpolation or bilinear interpolation.

[0170] When vui_colour_interpolation_type is 1, it means that the interpolation type is linear interpolation.

[0171] When vui_colour_interpolation_type is 2, it means that the interpolation type is bilinear interpolation.

[0172] Other values ​​for vui_colour_interpolation_type (3-15 in this example) indicate the prepared regions to be ensured for future extended use.

[0173] Figure 16 This is a diagram illustrating an example of the structure of a bitstream (BS).

[0174] Bitstream BS can also have multiple m-layer image layers L1 to L1 (where m is a natural number greater than 2). m A multi-layered structure. In Figure 16 Only one access unit is shown. An access unit is the smallest unit of processing a temporal attribute, such as one frame of a moving image. A bitstream (BS) is constructed by comprising multiple temporally consecutive access units.

[0175] In the payload area 42 of the first image layer L1, which is the lowest layer, the image Q, which serves as the main image, is stored. L1 The encoded data. In the payload area 42 of the second image layer L2, the image Q, serving as an auxiliary image, is stored. L2 The encoded data. Similarly, in the image layer L at the m-th layer. m The payload area 42 stores the image Q, which serves as an auxiliary image. Lm Encoded data.

[0176] Image Q L1 ~Q Lm They can also have different image categories or bands. For example, in a multi-layered structure with m=3 and 3 layers, the image Q could also be... L1 For visible light images, image Q L2 This is an infrared image, image Q L3 This is an ultraviolet image. For example, when encoding the output signal from a multispectral camera, the encoding device 1 can divide the spectral range into m regions and encode the image Q of each spectral range. L1 ~Q Lm Encode the bitstream BS as a multi-layered structure.

[0177] In image Q L1 ~Q Lm When there is a correlation between them, it is also possible to do so in image layers L1 to L2. m Interstitial reference image. On the other hand, in image Q L1 ~Q Lm If there is no correlation between them, it is also possible not to use image layers L1 to L2. m Refer to the image in between.

[0178] In the header region 41 of image layer L1, information related to image Q is stored. L1The associated parameter P was established. L1 The encoded data. Parameter P L1 Including image Q L1 Modal information and wavelength information, etc.

[0179] In the header region 41 of image layer L2, information related to image Q is stored. L2 The associated parameter P was established. L2 The encoded data. Parameter P L2 Including image Q L2 Modal information and wavelength information, etc.

[0180] In image layer L m The header area 41 stores information related to image Q. Lm The associated parameter P was established. Lm The encoded data. Parameter P Lm Including image Q Lm Modal information and wavelength information, etc.

[0181] Figure 17 This diagram shows a simplified representation of the structure of the circuit 21 included in the decoding device 2. The circuit 21 includes: a receiving unit 51, a decoding unit 52, a switching unit 53, and multiple (n being a natural number of 2 or more) task processing units 541 to 542. n Task Processing Department 541-54 n The tasks performed include both human vision and machine tasks. Machine tasks include a wide variety of tasks that utilize AI models, such as object detection, object tracking, object segmentation, action recognition, and pose estimation.

[0182] Next, the decoding unit 52 according to this embodiment will be described. Figure 23 This is a block diagram illustrating an example of the functional configuration of the decoding unit 52 according to this embodiment. The decoding unit 52 decodes the encoded image, i.e., the stream, on a block-by-block basis.

[0183] like Figure 23 As shown, the decoding unit 52 includes: an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter 212, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a segmentation determination unit 224. Furthermore, the intra-frame prediction unit 216 and the inter-frame prediction unit 218 are configured as part of the prediction processing unit 215.

[0184] For example, Figure 23 The decoding unit 52 shown has multiple components that are transmitted through Figure 1 The circuit 21 and memory 22 shown are installed.

[0185] Circuit 21 is configured to include a processor such as a CPU. Circuit 21 can be a dedicated or general-purpose electronic circuit for decoding streams, or it can be a collection of multiple electronic circuits. Alternatively, for example, circuit 21 can also implement... Figure 23 The decoding unit 52 shown has multiple components, including the components other than those used for storing information.

[0186] The memory 22 can be a dedicated or general-purpose electronic circuit for storing information, or it can be a collection of multiple electronic circuits. The memory 22 can be externally connected to the circuit 21, or it can be built into the circuit 21. Furthermore, the memory 22 can be a disk or optical disk, or it can be a storage device or recording medium. Additionally, the memory 22 can be either non-volatile or volatile.

[0187] The memory 22 can store the decoded stream or the decoded image. Additionally, the memory 22 can also store a program for the processor to perform the decoding process on the stream.

[0188] Alternatively, memory 22 can also be implemented Figure 23 The decoding unit 52 shown here has the function of storing information among its various constituent elements. Specifically, the memory 22 can also be implemented... Figure 23 The block memory 210 and frame memory 214 are shown to have specific functions. More specifically, the memory 22 can also store reconstructed images (specifically, reconstructed blocks or reconstructed images, etc.).

[0189] Furthermore, in the decoding unit 52, Figure 23 Some of the multiple components shown may be omitted from installation, and some of the multiple processes performed by these components may also be omitted from execution. Alternatively, Figure 23 A portion of the multiple components shown may also be mounted on other devices, and a portion of the multiple processes performed by the multiple components may also be performed by other devices.

[0190] Figure 23 The decoding unit 52 shown includes an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, a prediction control unit 220, and a loop filter 212, which are respectively connected to... Figure 21 The encoding unit 33 shown includes the same processing as the inverse quantization unit 112, inverse transform unit 114, adder unit 116, block memory 118, frame memory 122, intra-frame prediction unit 124, inter-frame prediction unit 126, prediction control unit 128, and loop filter 120.

[0191] Figure 18 This is a flowchart showing the process performed by the circuit 21 of the decoding device 2.

[0192] First, in step SP21, the receiving unit 51 receives the bit stream BS sent by the encoding device 1 from the transmission path NW.

[0193] Next, in step SP22, the decoding unit 52 decodes the bitstream BS input from the receiving unit 51 based on the payload region 42 to obtain the image Q. Furthermore, decoding may also include extraction. The decoding unit 52 outputs image data D21 of the image Q. Image data D21 is equivalent to... Figure 2 The image data shown is D11.

[0194] In such Figure 16 The bitstream BS shown has multiple image layers L1 to L2. m In the case of a multi-layer structure, the decoding unit 52 can also decode from multiple image layers L1 to L2. m Acquire multiple images of different image categories or bands Q L1 ~Q Lm .

[0195] Next, in step SP23, the decoding unit 52 decodes the header region 41 (or a given region 43 within the header region 41) of the bit stream BS input from the receiving unit 51 to obtain parameter P. As described above, parameter P includes a first parameter P1, a second parameter P2, a third parameter P3, and a fourth parameter P4. The first parameter P1 includes modal information indicating the image category of image Q. The second parameter P2 includes wavelength information indicating the band of light radiation of image Q. The third parameter P3 includes color information of image Q. The fourth parameter P4 includes modal information indicating the extended bit information corresponding to the prepared image category when the image category is an extended prepared image category. Furthermore, the execution order of steps SP22 and SP23 can be the same as... Figure 18 The opposite of the example can also be executed simultaneously.

[0196] exist Figure 12 When the VUI parameters shown are included in parameter P, and the modal information indicates that the image category is an infrared image (that is, when the value of vui_modality_type is 2), the decoding unit 52 further obtains the aforementioned performance type information indicating the performance type of infrared radiation intensity from the bitstream BS. Additionally, when the performance type information indicates that the pixel value represents a temperature value (that is, when it is a temperature value performance), the decoding unit 52 further obtains temperature information including the lowest and highest temperatures from the bitstream BS.

[0197] exist Figure 14When the VUI parameters shown are included in parameter P, and the flag information indicates that pixel values ​​are represented using virtual color representation (that is, when the value of vui_false_colour_representation_present_flag is 1), the decoding unit 52 further obtains from the bitstream BS the aforementioned generation information used to generate table information that establishes a correspondence between the pixel values ​​of image Q and the color codes of the virtual color representation. Furthermore, the decoding unit 52 generates the table information based on the obtained generation information.

[0198] Figure 19 , 20 This is a simplified diagram illustrating an example of table information.

[0199] exist Figure 19 , 20 In the example shown, there are 6 groups of pixel values ​​and color codes (meaning the value of vui_num_value_minus2 is 4), and the image category is infrared image. The interpolation type can be linear interpolation or bilinear interpolation.

[0200] exist Figure 19 In the example shown, the table information represents the correspondence between hexadecimal color codes (Hex color codes), RGB color codes, and shades of gray.

[0201] exist Figure 20 In the example shown, the table information illustrates the correspondence between Hex color codes, shades of gray, and temperatures, with gray shades mapped to temperatures. For example, black represents a temperature of 300K, gray represents a temperature of 330K, and white represents a temperature of 350K.

[0202] Next, in step SP24, the switching unit 53 switches the task processing units 541 to 54 based on the modal information input from the decoding unit 52. n The switching unit 53 maintains multiple image categories and multiple task processing units 541-54. n The correspondence is pre-set in a table of information (illustration omitted). Switching unit 53 refers to this table information to switch between multiple task processing units 541-542. n Among them, a task processing unit 541-54 is selected that corresponds to the image category represented by modal information. n .

[0203] Reference Figure 18 Next, in step SP25, a task processing unit 541 to 542 selected in step SP24... n Task processing is performed based on the image data D21 input from the decoding unit 52 via the switching unit 53.

[0204] According to the encoding device 1 and decoding device 2 of this embodiment, modal information representing the image category of image Q and wavelength information representing the band of light radiation of image Q can be included in the bit stream BS and transmitted from the encoding device 1 to the decoding device 2, thereby improving the execution accuracy of the task processing performed by the decoding device 2.

[0205] Furthermore, according to the encoding apparatus 1 and decoding apparatus 2 of this embodiment, the modal information indicates that the image category is at least one of the following: a visible light image, an infrared image, an ultraviolet image, an undefined image, or a planned image category for future expansion. Therefore, since the modal information includes information indicating that the image category is undefined or that the image category is a planned image category for future expansion, it is also possible to address situations where the image characteristics are unclear in the encoding apparatus 1 and the image category cannot be determined, or situations where the image category is to be expanded in the future.

[0206] Industrial availability

[0207] This disclosure is particularly useful for applications of image processing systems that include an encoding device for encoding an image into a bitstream and transmitting the bitstream, and a decoding device for decoding the image based on the received bitstream.

Claims

1. A decoding device, comprising: Circuit; and The memory is connected to the circuit. The circuit acquires the image from the bitstream and a first parameter associated with the image. The first parameter includes modal information representing the image category. The modal information indicates that the image category is at least a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension purposes.

2. The decoding device according to claim 1, wherein, The circuit obtains the second parameter from the bit stream. The second parameter includes wavelength information representing the band of light radiation from the image. When the modal information indicates that the image category is not defined, the wavelength information can represent bands spanning multiple image categories.

3. The decoding device according to claim 1, wherein, The circuit obtains the third parameter from the bit stream. The third parameter includes: a flag indicating whether the image's color information is included, and the color information itself. When the modal information indicates that the image category is an infrared image or an ultraviolet image, the circuit performs: (1) acquiring the marker information indicating that the color information is not included, or (2) ignoring the color information even if the marker information indicating that the color information is included is acquired.

4. The decoding device according to claim 1, wherein, The circuit obtains the fourth parameter from the bit stream. The fourth parameter includes: the modality information representing the extended bit information corresponding to the prepared image category when the image category is an extended prepared image category.

5. The decoding device according to claim 4, wherein, The extended bit information includes: Information representing the bit length of additional information related to the prepared image category; and The appended information having the specified bit length.

6. The decoding apparatus according to claim 1, wherein, When the modal information indicates that the image category is an infrared image, the circuit obtains representation type information representing the representation type of infrared radiation intensity from the bitstream. The representation type information indicates at least one of the following: the pixel value of the image represents the intensity value of infrared radiation, and the pixel value of the image represents the temperature value converted from the intensity value of infrared radiation.

7. The decoding apparatus according to claim 6, wherein, When the representation type information indicates that the pixel value represents the temperature value, the circuit obtains temperature information from the bitstream, including the lowest and highest temperatures of the temperature value.

8. The decoding apparatus according to claim 7, wherein, The temperature information includes, for the minimum temperature and the maximum temperature respectively: information indicating the sign of the sign, and information indicating the mantissa and the exponent in the exponent representation using a given base.

9. The decoding device according to claim 1, wherein, The circuit obtains flag information from the bitstream, the flag information indicating whether the pixel values ​​of the image are represented using virtual color representation. When the flag information indicates that the pixel value is represented using a virtual color representation, the circuit performs: Generation information is obtained from the bitstream. This generation information is used to generate a table that establishes a correspondence between the pixel values ​​and the color codes representing virtual colors. The table information is generated based on the generated information.

10. The decoding apparatus according to claim 9, wherein, The generated information includes: multiple groups of the pixel value and the color code, and interpolation information for interpolating the values ​​between the multiple groups.

11. The decoding apparatus according to claim 1, wherein, In obtaining the first parameter, the circuit obtains the first parameter from a given header region of the bit stream. The given header area includes either VUI or SEI.

12. An encoding device comprising: Circuit; and The memory is connected to the circuit. The circuit encodes the image and a first parameter associated with the image into a bit stream. The first parameter includes modal information representing the image category. The modal information indicates that the image category is at least a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension purposes.

13. The encoding device according to claim 12, wherein, The circuit encodes the second parameter into the bit stream. The second parameter includes wavelength information representing the band of light radiation from the image. When the modal information indicates that the image category is not defined, the wavelength information can represent bands spanning multiple image categories.

14. The encoding device according to claim 12, wherein, The circuit encodes the third parameter into the bit stream. The third parameter includes: a flag indicating whether the image's color information is included, and the color information itself. If the image is an infrared image or an ultraviolet image, the circuit will indicate that the marking information, which does not include the color information, is included in the third parameter.

15. The encoding device according to claim 12, wherein, The circuit encodes the fourth parameter into the bit stream. The fourth parameter includes: the modality information representing the extended bit information corresponding to the prepared image category when the image category is an extended prepared image category.

16. The encoding device according to claim 15, wherein, The extended bit information includes: Information representing the bit length of additional information related to the prepared image category; and The appended information having the specified bit length.

17. The encoding device according to claim 12, wherein, In the case that the image is an infrared image, the circuit encodes the representation type information indicating the representation type of infrared radiation intensity into the bit stream. The representation type information indicates at least one of the following: the pixel value of the image represents the intensity value of infrared radiation, and the pixel value of the image represents the temperature value converted from the intensity value of infrared radiation.

18. The encoding device according to claim 17, wherein, When the representation type information indicates that the pixel value represents the temperature value, the circuit encodes temperature information, including the lowest and highest temperatures of the temperature value, into the bitstream.

19. The encoding device according to claim 18, wherein, The temperature information includes, for the minimum temperature and the maximum temperature respectively: information indicating the sign of the sign, and information indicating the mantissa and the exponent in the exponent representation using a given base.

20. The encoding device according to claim 12, wherein, The circuit encodes flag information into the bitstream, the flag information indicating whether the pixel values ​​of the image are represented using virtual color representation. When the flag information indicates that the pixel value is represented using a virtual color representation, the circuit encodes the generated information into the bit stream. The generated information is used to generate table information that establishes a correspondence between the pixel value and the color code of the virtual color representation.

21. The encoding device according to claim 20, wherein, The generated information includes: multiple groups of the pixel value and the color code, and interpolation information for interpolating the values ​​between the multiple groups.

22. The encoding device according to claim 12, wherein, In the encoding of the first parameter, the circuit encodes the first parameter into a given header region of the bitstream. The given header area includes either VUI or SEI.

23. A decoding method, The decoding device acquires the image and a first parameter associated with the image from the bitstream. The first parameter includes modal information representing the image category. The modal information indicates that the image category is at least a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension purposes.

24. An encoding method, The encoding device encodes the image and a first parameter associated with the image into a bitstream. The first parameter includes modal information representing the image category. The modal information indicates that the image category is at least a visible light image, an infrared image, an ultraviolet image, an undefined image, or a pre-defined image category for extension purposes.

Images