Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device comprises a circuit, and a memory connected to the circuit. The circuit decodes, from a bitstream having a multi-layer configuration including at least one image layer, at least one parameter associated with the image layer. The parameter indicates whether or not an image decoded from the image layer associated with the parameter is suitable for a predetermined task process.

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method.

[0002] US Pat. No. 6,299,499 discloses a video encoding and decoding method using an adaptive combined pre-filter and post-filter.

[0003] US Patent No. 6,299,649 discloses a method for encoding image data for loading into an artificial intelligence (AI) integrated circuit.

[0004] However, in Patent Documents 1 and 2, in an image processing system that transmits a bitstream with a multi-layer structure from an encoding device to a decoding device, sufficient consideration is not given to reducing the processing load on the decoding device.

[0005] U.S. Patent No. 9,883,207 U.S. Patent No. 1,045,2955

[0006] The present disclosure aims to reduce the processing load on a decoding device in an image processing system that transmits a bitstream with a multi-layer structure from an encoding device to a decoding device.

[0007] A decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit decodes at least one parameter associated with an image layer from a multi-layered bitstream including at least one image layer, the parameter indicating whether an image decoded from the image layer associated with the parameter is suitable for a predetermined task processing.

[0008] 1 is a diagram showing, in a simplified form, the configuration of an image processing system according to an embodiment of the present disclosure. FIG. 1 is a diagram showing, in a simplified form, the configuration of a circuit included in an encoding device. FIG. 2 is a flowchart showing processing performed by a circuit included in an encoding device. FIG. 2 is a diagram showing, in a simplified form, a portion of a bitstream with a multi-layer configuration. FIG. 3 is a diagram showing an example of parameter setting by a setting unit. FIG. 4 is a diagram showing a first example of syntax related to parameter setting. FIG. 5 is a diagram showing a second example of syntax related to parameter setting. FIG. 3 is a diagram showing, in a simplified form, the configuration of a circuit included in a decoding device. FIG. 4 is a flowchart showing processing performed by a circuit included in a decoding device. FIG. 4 is a diagram showing, in a simplified form, a portion of a bitstream with a multi-layer configuration. FIG. 5 is a diagram showing an example of parameter setting by a setting unit. FIG. 6 is a diagram showing an example of syntax related to parameter setting. FIG. 6 is a diagram showing a first example of parameter setting by a setting unit. FIG. 7 is a diagram showing a second example of parameter setting by a setting unit. FIG. 7 is a diagram showing an example of processing in a decoding device. FIG. 8 is a diagram showing an example of processing in a decoding device. FIG. 8 is a diagram showing an example of processing in a decoding device. FIG. 9 is a diagram showing an example of processing in a decoding device. It is a block diagram showing an example of the functional configuration of a decoding unit.It is a diagram showing an example of the hierarchical structure of data in a stream.It is a diagram showing an example of the configuration of a bit stream.

[0009] (Findings forming the basis of the present disclosure) An image processing system according to the background art includes an encoding device and a decoding device. The encoding device encodes an image into a bitstream and transmits the bitstream containing the encoded image to a decoding device. The decoding device decodes the image from the received bitstream and executes task processing using the decoded image.

[0010] Task processing includes machine vision and human vision, where machine vision includes object detection, object tracking, object segmentation, action recognition, or pose estimation using machine-learned inference models, and human vision includes viewing or watching video images by a human, such as an operator or user.

[0011] When a bitstream has a multi-layer structure including multiple image layers, different images are stored in the multiple image layers. The preferred image layer containing the image to be used for task processing varies depending on the content of the task processing. However, in the background art, there is no information indicating the correspondence between the content of the task processing and the preferred image layer. Therefore, the decoding device decodes all image layers, including unpreferred image layers, resulting in a large processing load on the decoding device.

[0012] In order to solve this problem, the present inventor discovered that by including information indicating whether an image decoded from an image layer is suitable for task processing in a bitstream and transmitting the information from the encoding device to the decoding device, unnecessary decoding in the decoding device can be avoided, thereby solving the above problem, and came up with the present disclosure.

[0013] Next, each aspect of the present disclosure will be described.

[0014] A decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit decodes at least one parameter associated with an image layer from a multi-layered bitstream including at least one image layer, the parameter indicating whether an image decoded from the image layer associated with the parameter is suitable for a predetermined task processing.

[0015] According to the first aspect, the decoding device can avoid unnecessary decoding based on the parameters, thereby reducing the processing load on the decoding device and improving the efficiency of processing.

[0016] In the decoding device according to the second aspect of the present disclosure, in the first aspect, the circuit may further decode an image from an image layer selected from the at least one image layer based on the parameters, and perform the task processing using the image decoded from the image layer.

[0017] According to the second aspect, the decoding device can appropriately perform task processing using an image suitable for the task processing.

[0018] In the decoding device according to the third aspect of the present disclosure, in the first or second aspect, the task processing may include machine vision.

[0019] According to the third aspect, the decoding device can appropriately perform machine vision using an image suitable for machine vision.

[0020] In a decoding device according to a fourth aspect of the present disclosure, in any one of the first to third aspects, the task processing may include human vision.

[0021] According to the fourth aspect, the decoding device can appropriately perform human vision using images suitable for human vision.

[0022] A decoding device according to a fifth aspect of the present disclosure may be any one of the first to fourth aspects, wherein the task processing includes machine vision and human vision, and the at least one parameter includes a first parameter indicating whether an image decoded from the image layer is suitable for the machine vision, and a second parameter indicating whether the image decoded from the image layer is suitable for the human vision.

[0023] According to the fifth aspect, the decoding device can appropriately perform machine vision using an image suitable for machine vision, and can appropriately perform human vision using an image suitable for human vision.

[0024] A decoding device according to a sixth aspect of the present disclosure may be configured such that, in any one of the first to fifth aspects, the parameters include a first value and a second value, the first value indicating that the image decoded from the image layer is suitable for the task processing, and the second value indicating that the image decoded from the image layer is not suitable for the task processing.

[0025] According to the sixth aspect, it is possible to prevent the decoding device from decoding an image that is not suitable for task processing.

[0026] A decoding device according to a seventh aspect of the present disclosure may be configured such that, in any one of the first to fifth aspects, the parameters include a first value and a second value, the first value indicating that the image decoded from the image layer is suitable for the task processing, and the second value indicating that it is not determined whether the image decoded from the image layer is suitable for the task processing.

[0027] According to the seventh aspect, the decoding device can arbitrarily determine whether to decode an image from an image layer associated with a parameter indicating the second value, depending on the processing load status and the like.

[0028] In the decoding device according to the eighth aspect of the present disclosure, in the sixth or seventh aspect, the circuit may further decode an image from only an image layer of the at least one image layer that is associated with the parameter indicating the first value, and perform the task processing using the image decoded from that image layer.

[0029] According to the eighth aspect, the decoding device decodes the image from only the image layer associated with the parameter indicating the first value, which makes it possible to further reduce the processing load of the decoding device.

[0030] In a decoding device according to a ninth aspect of the present disclosure, in any one of the first to eighth aspects, the at least one image layer may include an image layer to which the parameter is not associated, and the fact that the parameter is not associated with the image layer may indicate that it has not been determined whether the image decoded from the image layer is suitable for the task processing.

[0031] According to the ninth aspect, the decoding device can arbitrarily determine whether to decode an image from an image layer that is not associated with a parameter, depending on the processing load status, etc.

[0032] A decoding device according to a tenth aspect of the present disclosure is, in any one of the first to ninth aspects, preferably wherein the circuit decodes the at least one parameter from a predetermined header region of the bitstream, and the predetermined header region includes an SEI.

[0033] According to the tenth aspect, the decoding device can easily decode the parameters from a predetermined header area of ​​the bitstream.

[0034] In a decoding device according to an eleventh aspect of the present disclosure, in the tenth aspect, the at least one image layer may include a base layer which is the lowest layer of the multi-layer structure, and the at least one parameter associated with the at least one image layer may be stored in the header area of ​​the base layer.

[0035] According to the eleventh aspect, the decoding device can acquire all parameters associated with all image layers collectively from the header area of ​​the base layer.

[0036] A decoding device according to a twelfth aspect of the present disclosure is, in the tenth aspect, preferably wherein the at least one parameter associated with the at least one image layer is stored in the header area of ​​each of the at least one image layer.

[0037] According to the twelfth aspect, the decoding device can individually obtain each parameter associated with each image layer from the header area of ​​each image layer.

[0038] A coding device according to a thirteenth aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit encodes at least one parameter associated with at least one image layer into a multi-layered bitstream including the image layer, the parameter indicating whether an image in the image layer associated with the parameter is suitable for processing a predetermined task.

[0039] According to the thirteenth aspect, a decoding device that receives a bitstream can avoid unnecessary decoding based on the parameters, thereby reducing the processing load on the decoding device and improving the efficiency of processing.

[0040] In a coding device according to a fourteenth aspect of the present disclosure, in the thirteenth aspect, the task processing may include machine vision.

[0041] According to the fourteenth aspect, a decoding device that receives a bitstream can appropriately perform machine vision using an image suitable for machine vision.

[0042] In the encoding device according to a fifteenth aspect of the present disclosure, in the thirteenth or fourteenth aspect, the task processing may include human vision.

[0043] According to the fifteenth aspect, a decoding device that receives a bitstream can appropriately perform human vision using images that are suitable for human vision.

[0044] In the encoding device according to a 16th aspect of the present disclosure, in any one of the 13th to 15th aspects, the task processing may include machine vision and human vision, and the at least one parameter may include a first parameter indicating whether an image in the image layer is suitable for the machine vision, and a second parameter indicating whether the image in the image layer is suitable for the human vision.

[0045] According to the sixteenth aspect, a decoding device that receives a bitstream can appropriately perform machine vision using images suitable for machine vision, and can appropriately perform human vision using images suitable for human vision.

[0046] In the encoding device according to a seventeenth aspect of the present disclosure, in any one of the thirteenth to sixteenth aspects, the parameters may include a first value and a second value, the first value indicating that an image in the image layer is suitable for the task processing, and the second value indicating that the image in the image layer is not suitable for the task processing.

[0047] According to the seventeenth aspect, it is possible to prevent a decoding device that receives a bitstream from decoding an image that is not suitable for task processing.

[0048] In an encoding device according to an 18th aspect of the present disclosure, in any one of the 13th to 16th aspects, the parameters may include a first value and a second value, the first value indicating that an image in the image layer is suitable for the task processing, and the second value indicating that the image in the image layer is not suitable for the task processing.

[0049] According to the 18th aspect, a decoding device that receives a bitstream can arbitrarily decide whether or not to decode an image from an image layer associated with a parameter indicating a second value, depending on the processing load status, etc.

[0050] In the encoding device according to the 19th aspect of the present disclosure, in any one of the 13th to 18th aspects, the at least one image layer may include an image layer to which the parameter is not associated, and not associating the parameter with the image layer may indicate that it is not determined whether the image in the image layer is suitable for the task processing.

[0051] According to the nineteenth aspect, the decoding device that has received the bitstream can arbitrarily determine whether or not to decode an image from an image layer that is not associated with a parameter, depending on the processing load status, etc.

[0052] In the encoding device according to the 20th aspect of the present disclosure, in any one of the 13th to 19th aspects, the circuit may encode the at least one parameter in a predetermined header area of ​​the bitstream, and the predetermined header area may include an SEI.

[0053] According to the twentieth aspect, a decoding device that receives a bitstream can easily decode parameters from a predetermined header area of ​​the bitstream.

[0054] In the encoding device according to the 21st aspect of the present disclosure, in the 20th aspect, the at least one image layer includes a base layer which is the lowest layer of the multi-layer structure, and the at least one parameter associated with the at least one image layer is stored in the header area of ​​the base layer.

[0055] According to the twenty-first aspect, the decoding device that receives the bitstream can collectively acquire all parameters associated with all image layers from the header area of ​​the base layer.

[0056] In the encoding device according to the 22nd aspect of the present disclosure, in the 20th aspect, the at least one parameter associated with the at least one image layer may be stored in the header area of ​​each of the at least one image layer.

[0057] According to the twenty-second aspect, the decoding device that receives the bitstream can individually obtain each parameter associated with each image layer from the header area of ​​each image layer.

[0058] A decoding method according to a 23rd aspect of the present disclosure includes a decoding device that decodes at least one parameter associated with at least one image layer from a multi-layered bitstream including the image layer, the parameter indicating whether an image decoded from the image layer associated with the parameter is suitable for processing a predetermined task.

[0059] According to the twenty-third aspect, the decoding device can avoid unnecessary decoding based on the parameters, thereby reducing the processing load on the decoding device and improving the efficiency of processing.

[0060] An encoding method according to a 24th aspect of the present disclosure includes an encoding device encoding at least one parameter associated with at least one image layer into a multi-layered bitstream including the image layer, the parameter indicating whether an image in the image layer associated with the parameter is suitable for processing a predetermined task.

[0061] According to the twenty-fourth aspect, a decoding device that receives a bitstream can avoid unnecessary decoding based on the parameters, thereby reducing the processing load on the decoding device and improving the efficiency of processing.

[0062] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0063] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts are described as optional components. Furthermore, in all embodiments, the respective contents can be substituted or combined. Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or by any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0064] 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, and a transmission path NW.

[0065] Image data D1 is input from an external device to the encoding device 1. The external device may include a camera or the like that captures moving images. The external device inputs image data D1 of the captured moving images to the encoding device 1.

[0066] The encoding device 1 generates a bitstream BS based on image data D1. Fig. 20 is a diagram showing an example of the structure of the bitstream BS. The bitstream BS has a multi-layer structure including at least one image layer L. In this example of the present embodiment, the bitstream BS has a three-layer multi-layer structure. However, this is not limiting. The three-layer multi-layer structure includes a first image layer L, which is the lowest layer, which is the base layer.１ and a second image layer L, which is an enhancement layer at a higher level. ２ and the third image layer L ３ Includes:

[0067] For example, the first image layer L １ The second image layer L includes image data of an I frame of a region of interest (ROI). The ROI corresponds to an object or the like included in the image. ２ The third image layer L includes image data of the P and B frames of the ROI region. ３ The first image layer L includes image data of the background region excluding the ROI region. １ A low-quality image of the ROI region is obtained by the first image layer L １ and the second image layer L ２ By this, a high quality image of the ROI region can be obtained. １ and the second image layer L ２ and the third image layer L ３ A complete image including the ROI region and the background region can be obtained by the above process. Note that the multi-layer configuration is not limited to the above example.

[0068] The encoding device 1 transmits the generated bit stream BS to the decoding device 2 via a transmission path NW. The decoding device 2 receives the bit stream BS.

[0069] The decoding device 2 obtains images from the bitstream BS by decoding them, and performs task processing based on the obtained images. The process of obtaining images from the bitstream may be referred to as extraction or decoding. The task processing includes machine vision and human vision. Machine vision includes object detection, object tracking, object segmentation, action recognition, pose estimation, etc., using an artificial intelligence (AI) model, which is a machine-learned estimation model. When performing machine vision, the task processing unit includes an inference device using AI. Human vision includes viewing or watching of moving images by a human, such as an operator or user. When performing human vision, the task processing unit includes a display device, such as a liquid crystal display or an organic electroluminescence (EL) display.

[0070] The transmission path NW may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. The transmission path NW is preferably a private network or the like in which secure communication is ensured by access restrictions.

[0071] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 includes a processor such as a CPU. The memory 12 includes any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data to be processed by the circuit 11 or data in the middle of processing.

[0072] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 includes a processor such as a CPU. The memory 22 includes any recording medium such as a ROM, a RAM, a HDD, an SSD, or a semiconductor memory. The memory 22 stores data to be processed by the circuit 21 or data in the middle of processing.

[0073] 2 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 includes an acquisition unit 31, a setting unit 32, an encoding unit 33, and a transmission unit .

[0074] Next, the encoding unit 33 according to this embodiment will be described. Fig. 17 is a block diagram showing an example of the functional configuration of the encoding unit 33 according to this embodiment. The encoding unit 33 encodes an image in units of blocks.

[0075] 17 , the encoding unit 33 includes a dividing unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Note that the intra prediction unit 124 and the inter prediction unit 126 are configured as part of a prediction processing unit 125.

[0076] For example, the components of the encoding unit 33 shown in FIG. 17 are implemented by the circuit 11 and memory 12 shown in FIG.

[0077] The circuit 11 is configured to include a processor such as a CPU. The circuit 11 may be a dedicated or general-purpose electronic circuit for encoding images, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 11 may fulfill the roles of multiple components of the encoding unit 33 shown in FIG. 17 , excluding the component for storing information.

[0078] The memory 12 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 12 may be externally connected to the circuit 11 or may be built into the circuit 11. The memory 12 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 12 may be a non-volatile memory or a volatile memory.

[0079] The memory 12 may store an image to be encoded, or a stream corresponding to the encoded image, or may store a program for the processor to execute the image encoding process.

[0080] Furthermore, the memory 12 may serve as a component for storing information among the multiple components included in the encoding unit 33 shown in Fig. 29. Specifically, the memory 12 may serve as the block memory 118 and the frame memory 122 shown in Fig. 17. More specifically, the memory 12 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0081] 17 may be omitted from the encoding unit 33, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 17 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0082] FIG. 3 is a flowchart showing the process executed by the circuit 11 included in the encoding device 1.

[0083] First, in step SP11, the acquisition unit 31 acquires image data D1 representing the image X to be processed, which is input from an external device.

[0084] Next, in step SP12, the encoding unit 33 encodes the image X into a bitstream BS. In this embodiment, the image X is encoded into the first image layer L. １ Image X corresponding to １ and the second image layer L ２ Image X corresponding to ２ and the third image layer L ３ Image X corresponding to ３ Includes:

[0085] Fig. 4 is a simplified diagram showing a portion of a multi-layered bitstream BS. Fig. 4 shows only one access unit. An access unit is the smallest processing unit of a time attribute, and corresponds to, for example, one frame of a video image. The bitstream BS is composed of multiple access units that are consecutive in time.

[0086] Each image layer L has a header area 41 and a payload area 42. The encoding unit 33 encodes the first image layer L １ The payload area 42 of the image X １ The encoding unit 33 stores the encoded image obtained by encoding the second image layer L ２ The payload area 42 of the image X ２ The encoding unit 33 stores the encoded image obtained by encoding the third image layer L ３ The payload area 42 of the image X ３ The encoded image is stored.

[0087] Referring to FIG. 3 , next, in step SP13, the setting unit 32 sets a parameter P in association with the image layer L. The parameter P indicates whether the image X encoded in the image layer L associated with the parameter P is suitable for a predetermined task processing. The predetermined task processing is arbitrarily set by the setting unit 32 from among machine vision and human vision. In the example of this embodiment, the predetermined task processing is object tracking, which is one type of machine vision. However, this example is not limiting. The setting information for the predetermined task processing may be shared in advance between the encoding device 1 and the decoding device 2, or may be included in a bitstream BS and transmitted from the encoding device 1 to the decoding device 2.

[0088] 5 is a diagram showing an example of setting the parameter P by the setting unit 32. In this embodiment, the parameter P is １ ~P ３ The setting unit 32 includes the first image layer L １ Parameter P １ and the second image layer L ２ Parameter P ２ and the third image layer L ３ Parameter P ３ Set.

[0089] The setting unit 32 sets each parameter P １ ~P ３is set to "1" (first value) or "0" (second value). The value "1" of the parameter P indicates that the image X encoded in the image layer L is suitable for the above-mentioned predetermined task processing (object tracking). The suitability of an image for task processing means that it has been encoded as an image suitable for task processing. The value "0" of the parameter P indicates that the image X encoded in the image layer L is not suitable for the above-mentioned predetermined task processing (object tracking). The unsuitability of an image for task processing means that it has not been encoded as an image suitable for task processing.

[0090] In the example shown in FIG. 5, the setting unit 32 sets the parameter P １ , P ２ , P ３ The values ​​of are set to "1", "0", and "0" respectively. In this setting example, image X １ is suitable for object tracking, and image X ２ , X ３ means that the parameter P is not suitable for object tracking. The setting unit 32 inputs data D2 indicating the setting value of the parameter P to the encoding unit 33.

[0091] 3, next, in step SP14, the encoding unit 33 encodes the parameters P into a bit stream BS. Here, the process of encoding the parameters into a bit stream may be rephrased as saving or storing.

[0092] Referring to FIG. 4, the encoding unit 33 encodes the first image layer L １ The header area 41 of １ The encoding unit 33 also encodes the second image layer L ２ The header area 41 of ２ The encoding unit 33 also encodes the third image layer L ３ The header area 41 of ３The parameter P is stored in a predetermined area of ​​the header area 41. The predetermined area is SEI (Supplemental Enhancement Information). However, the predetermined area may be VUI (Video Usability Information), VPS, SPS, PPS, PH, SH, APS, a tile header, a system layer header, or the like.

[0093] 19 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. As shown in (A) of FIG. 19, the video sequence includes, for example, a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), supplemental enhancement information (SEI), and multiple pictures.

[0094] In a video composed of multiple layers, the VPS includes coding parameters common to multiple layers and coding parameters related to multiple layers or individual layers included in the video.

[0095] The SPS includes parameters used for a sequence, i.e., encoding parameters that the decoding device 2 refers to in order to decode the sequence. The encoding parameters may indicate, for example, the width or height of a picture. Note that there may be multiple SPSs.

[0096] The PPS includes parameters used for a picture, i.e., encoding parameters that the decoding device 2 references to decode each picture in a sequence. The encoding parameters may include, for example, a reference value of the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Note that there may be multiple PPSs. The SPS and PPS may also be simply referred to as parameter sets.

[0097] As shown in (B) of Fig. 19, a picture includes a picture header and one or more slices. The picture header includes coding parameters that are referenced by the decoding device 2 to decode the one or more slices.

[0098] As shown in Fig. 19C, a slice includes a slice header and one or more bricks. The slice header includes coding parameters that are referenced by the decoding device 2 to decode the one or more bricks.

[0099] A brick includes one or more coding tree units (CTUs), as shown in FIG. 19(D).

[0100] Note that a picture may not contain slices, but may contain tile groups instead, where each tile group contains one or more tiles. Also, a brick may contain slices.

[0101] A CTU is also called a superblock or a basic division unit. As shown in (E) of Fig. 19, a CTU includes a CTU header and one or more coding units (CUs). The CTU header includes coding parameters that the decoding device 2 references to decode the one or more CUs.

[0102] A CU may be divided into multiple smaller CUs. Furthermore, as shown in (F) of FIG. 19 , a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting a CU. The residual coefficient information is information indicating a prediction residual. A CU is basically the same as a PU (Prediction Unit) or a TU (Transform Unit), but may include multiple TUs smaller than a CU. A CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is a fixed unit that can be processed in one stage, for example, when performing pipeline processing in hardware.

[0103] Note that a stream may not have some of the layers among the layers shown in Fig. 19. Furthermore, the order of these layers may be changed, and any layer may be replaced with another layer.

[0104] A picture currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current picture. If the processing is encoding, the current picture is synonymous with a picture to be encoded, and if the processing is decoding, the current picture is synonymous with a picture to be decoded. Furthermore, a block (CU or a block of a CU) currently being processed by a device such as the encoding device 1 or the decoding device 2 is called a current block. If the processing is encoding, the current block is synonymous with a block to be encoded, and if the processing is decoding, the current block is synonymous with a block to be decoded.

[0105] 6A is a diagram showing a first example of syntax related to setting the parameter P. In this example, the parameter P is set as the value of mvi_optimized_for_first_vision_task_flag included in an SEI message such as machine_vision_indication. If the value of the identifier of the flag is "1", this indicates that the image X encoded in the image layer L associated with the parameter P is suitable for task processing. If the value of the identifier of the flag is "0", this indicates that the image X encoded in the image layer L associated with the parameter P is not suitable for task processing.

[0106] 6B is a diagram showing a second example of syntax related to setting the parameter P. In this example, the parameter P is set as the value of mvi_not_optimized_for_first_vision_task_flag included in an SEI message such as machine_vision_indication. If the value of the identifier of the flag is "1", this indicates that the image X encoded in the image layer L associated with the parameter P is not suitable for task processing. If the value of the identifier of the flag is "0", this indicates that the image X encoded in the image layer L associated with the parameter P is suitable for task processing.

[0107] Referring to FIG. 3, next, in step SP15, the transmitter 34 transmits the bit stream BS input from the encoder 33 to the decoder 2 via the transmission path NW.

[0108] 7 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 includes a receiving unit 51, a decoding unit 52, and a task processing unit 53.

[0109] Next, the decoding unit 52 according to this embodiment will be described. Fig. 18 is a block diagram showing an example of the functional configuration of the decoding unit 52 according to this embodiment. The decoding unit 52 decodes a stream, which is an encoded image, in units of blocks.

[0110] 18 , the decoding unit 52 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a partition determination unit 224. Note that the intra prediction unit 216 and the inter prediction unit 218 are configured as part of a prediction processing unit 215.

[0111] For example, the components of the decoding unit 52 shown in FIG. 18 are implemented by the circuit 21 and memory 22 shown in FIG.

[0112] The circuit 21 is configured to include a processor such as a CPU. The circuit 21 may be a dedicated or general-purpose electronic circuit for decoding streams, or may be a collection of multiple electronic circuits. Furthermore, for example, the circuit 21 may fulfill the roles of multiple components of the decoding unit 52 shown in FIG. 18 , excluding the component for storing information.

[0113] The memory 22 may be a dedicated or general-purpose electronic circuit for storing information, or may be a collection of multiple electronic circuits. The memory 22 may be externally connected to the circuit 21 or may be built into the circuit 21. The memory 22 may be a magnetic disk, an optical disk, or the like, and may be expressed as a storage or a recording medium, or the like. The memory 22 may be a non-volatile memory or a volatile memory.

[0114] The memory 22 may store a stream to be decoded or a decoded image, or may store a program for the processor to execute the decoding process of the stream.

[0115] Furthermore, the memory 22 may serve as a component for storing information among the multiple components included in the decoding unit 52 shown in Fig. 18. Specifically, the memory 22 may serve as the block memory 210 and the frame memory 214 shown in Fig. 18. More specifically, the memory 22 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0116] 18 may be omitted from the decoding unit 52, or some of the processes executed by the components may be omitted. Alternatively, some of the components shown in FIG. 18 may be implemented in another device, or some of the processes executed by the components may be executed by another device.

[0117] The inverse quantization unit 204, inverse transform unit 206, addition unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter 212 included in the decoding unit 52 shown in Figure 18 perform processing similar to that of the inverse quantization unit 112, inverse transform unit 114, addition unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter 120 included in the encoding unit 33 shown in Figure 17.

[0118] FIG. 8 is a flowchart showing the process executed by the circuit 21 included in the decoding device 2.

[0119] First, in step SP21, the receiving unit 51 receives the bit stream BS via the transmission path NW from the encoding device 1. The receiving unit 51 inputs the received bit stream BS to the decoding unit 52.

[0120] Next, in step SP22, the decoding unit 52 decodes the parameter P from the bitstream BS. Referring to FIG. 4, the decoding unit 52 decodes the parameter P from the first image layer L １ From the header area 41 of １ The decoding unit 52 also decodes the second image layer L ２ From the header area 41 of ２ The decoding unit 52 also decodes the third image layer L ３ From the header area 41 of ３ Referring to FIG. 5, in the example of this embodiment, the parameter P １ , P ２ , P ３ The values ​​are set to "1", "0", and "0", respectively.

[0121] 8, next, in step SP23, the decoding unit 52 selects an image layer L based on the parameter P decoded in step SP22, and decodes an image X from the selected image layer L. In the example of this embodiment, the decoding unit 52 selects an image layer L based on the parameter P １ The image layer L whose value is set to "1" １ Select the parameter P ２ , P３ Image layer L whose value is set to "0" ２ , L ３ Therefore, the decoding unit 52 does not select the selected image layer L １ Image X from １ and decodes the image layer L ２ , L ３ Image X from ２ , X ３ The decoding unit 52 does not decode the decoded image X １ The image data D3 is input to the task processing unit 53.

[0122] In the decoding process of a bitstream having a multi-layer structure, when decoding an image of an upper image layer, an image of a lower image layer is referenced. ２ If you select Image X １ , X ２ Decode the image X ３ In addition, the decoding unit 52 does not decode the third image layer L ３ If you select Image X １ ~X ３ However, if the correlation between images between image layers is low, the decoding unit 52 does not need to refer to images in lower image layers when decoding images in higher image layers.

[0123] Next, in step SP24, the task processing unit 53 processes the image X indicated by the image data D3. １ In this embodiment, the task processing unit 53 executes task processing using the image X １ Object tracking is performed using

[0124] According to this embodiment, the parameter P indicates whether the image X decoded from the image layer L associated with the parameter P is suitable for a predetermined task processing. Therefore, the decoding device 2 can avoid unnecessary decoding based on the parameter P, thereby reducing the processing load of the decoding device 2.

[0125] Furthermore, the decoding device 2 decodes an image X from an image layer L selected based on the parameter P, and executes task processing using the image X decoded from the image layer L. Therefore, task processing can be appropriately executed using an image X suitable for the task processing.

[0126] Furthermore, the task processing includes machine vision, so the decoding device 2 can appropriately perform machine vision using the image X that is suitable for machine vision.

[0127] Furthermore, the task processing includes human vision, so the decoding device 2 can appropriately perform human vision using the image X that is suitable for human vision.

[0128] Furthermore, the parameter P includes a first value and a second value, where the first value indicates that the image X decoded from the image layer L is suitable for task processing, and the second value indicates that the image X decoded from the image layer L is not suitable for task processing. Therefore, it is possible to prevent the decoding device 2 from decoding the image X that is not suitable for task processing.

[0129] Furthermore, the decoding device 2 decodes the image X only from the image layer L associated with the parameter P indicating the first value, which makes it possible to further reduce the processing load on the decoding device 2.

[0130] Furthermore, the decoding device 2 decodes the parameter P from the SEI in the header area 41 of the bitstream BS. Therefore, the decoding device 2 can easily decode the parameter P.

[0131] Also, the image layer L １ ~L ３ The parameter P associated with １ ~P ３ is the image layer L １ ~L ３ The image data is stored in the header area 41 of each image layer L. １ ~L ３ Each parameter P associated with １ ~P ３ for each image layer L １ ~L ３ The information can be individually obtained from the header area 41 of the

[0132] With the above configuration, the present disclosure may improve the accuracy of machine vision models and reduce the computational load of the decoder 2. A feature of this technique is the adoption of a layered coding scheme. Layered coding is a method of encoding a video stream by dividing it into multiple layers, each containing different video data. Each layer is associated with a different vision task, and these layers function collectively to compress the image. The decoder 2 has the flexibility to use these vision task instructions and perform the vision task according to the decoded layers. Vision tasks such as object detection and object tracking rely on accurate and relevant visual information for accurate prediction. Therefore, for such vision tasks and human vision, it may be effective to use high-quality images by including an enhancement layer. On the other hand, for other vision tasks, using only the base layer may be sufficient. For example, a base layer containing only an ROI is optimized for machine vision and may not be suitable for human vision. On the other hand, an enhancement layer containing residual (non-ROI) data or temporal upsampling data is suitable for human vision. Therefore, the method of the present disclosure improves the accuracy of machine vision models and saves the computational power of the decoder 2.

[0133] Various modifications of the above embodiment will be described below. The modifications described below can be applied in any combination.

[0134] (First Modification) The predetermined task processing may include machine vision and human vision. Furthermore, the at least one parameter may include a parameter P (first parameter) indicating whether the image X decoded from the image layer L is suitable for machine vision, and a parameter Q (second parameter) indicating whether the image X decoded from the image layer L is suitable for human vision.

[0135] 9 is a simplified diagram showing a part of a bitstream BS having a multi-layer structure. Only one access unit is shown in FIG. 9. The parameter P is a parameter P１ ~P ３ and the parameter Q is the parameter Q １ ~Q ３ The setting unit 32 includes the first image layer L １ Parameter P １ , Q １ and the second image layer L ２ Parameter P ２ , Q ２ and the third image layer L ３ Parameter P ３ , Q ３ Set.

[0136] 10 is a diagram showing an example of setting the parameters P and Q by the setting unit 32. In this example, the setting unit 32 sets the parameters P １ , P ２ , P ３ The values ​​of are set to "1", "0", and "0", respectively, and the parameter Q １ , Q ２ , Q ３ The values ​​of are set to "0", "0", and "1". In this setting example, image X １ is suitable for object tracking, and image X ２ , X ３ means that it is not suitable for object tracking. ３ (and lower layer image X １ , X ２ ) means it is suitable for human vision.

[0137] 11 is a diagram showing an example of syntax for setting parameters P and Q. In this example, parameter P is set as the value of mvi_optimized_for_first_vision_task_flag. If the value of the flag identifier is "1", this indicates that image X encoded in the image layer L associated with parameter P is suitable for machine vision. If the value of the flag identifier is "0", this indicates that image X encoded in the image layer L associated with parameter P is not suitable for machine vision.

[0138] Also, in this example, the parameter Q is set as the value of mvi_optimized_for_second_vision_task_flag. If the value of the identifier of this flag is "1", it indicates that the image X encoded in the image layer L associated with the parameter Q is suitable for human vision. If the value of the identifier of this flag is "0", it indicates that the image X encoded in the image layer L associated with the parameter Q is not suitable for human vision.

[0139] If parameter P or parameter Q is a parameter indicating whether it is suitable for human vision, a value indicating suitability for human vision may be set for all image layers in the multi-layer configuration. Also, if parameter P or parameter Q is a parameter indicating whether it is suitable for machine vision, a constraint may be imposed on the set value for at least one of the multiple image layers in the multi-layer configuration, so that one parameter is set to a value indicating it is not suitable for machine vision, and the other parameter is set to a value indicating it is suitable for machine vision.

[0140] According to this modification, the at least one parameter includes a parameter P indicating whether or not the image X is suitable for machine vision, and a parameter Q indicating whether or not the image X is suitable for human vision. Therefore, based on the parameters P and Q, the decoding device 2 can appropriately perform machine vision using the image X that is suitable for machine vision, and can appropriately perform human vision using the image X that is suitable for human vision.

[0141] (Second Modification) FIG. 12 is a diagram showing a first example of setting the parameter P by the setting unit 32. The setting unit 32 sets each parameter P １ ~P ３The value of the parameter P is set to "1" (first value) or "0" (second value). The value "1" of the parameter P indicates that the image X encoded in the image layer L is suitable for task processing. The value "0" of the parameter P indicates that it is not specified whether the image X encoded in the image layer L is suitable for task processing.

[0142] According to the first setting example, the decoding device 2 that receives the bitstream BS can arbitrarily decide whether or not to decode the image X from the image layer L associated with the parameter P indicating the second value, depending on the processing load situation, etc.

[0143] 13 is a diagram showing a second example of setting the parameter P by the setting unit 32. The setting unit 32 sets the parameter P １ , P ３ The setting unit 32 sets the value of the image layer L ２ The parameter P ２ The parameter P is not associated with the image layer L. A value of "1" of the parameter P indicates that the image X encoded in the image layer L is suitable for task processing. A value of "0" of the parameter P indicates that the image X encoded in the image layer L is not suitable for task processing. Not associating the parameter P with the image layer L indicates that it is not specified whether the image X encoded in the image layer L is suitable for task processing.

[0144] According to the second setting example, the decoding device 2 that receives the bitstream BS can arbitrarily decide whether or not to decode the image X from the image layer L that is not associated with the parameter P, depending on the processing load situation, etc.

[0145] 14A to 14E are diagrams illustrating an example of a processing method for the decoder 2 that has received the bitstream BS to control the image layer to be decoded depending on the processing load status, etc. The decoder 2 controls the first image layer L, which is the lowest layer. １ The number of image layers that need to be decoded is counted in order from the first to the second, and the image layer that needs to be decoded is identified according to the count value C.

[0146] As shown in FIG. 14A, the parameter P １ , P ２ , P ３14B, when the values ​​of the parameter P １ , P ２ , P ３ If the values ​​of the parameters P are "1", "1", and "0", respectively, the count value C is "2". １ , P ２ , P ３ When the values ​​of the parameters P are "1", "0", and "0", respectively, the count value C is "1". １ , P ２ , P ３ When the values ​​of the parameters P are "0", "0", and "0", respectively, the count value C is "0". １ , P ２ , P ３ If the values ​​of are "0", "1", and "0", respectively, the count value C is "2".

[0147] By performing the above processing, the first image layer L, which is the lowest layer in the decoding device 2, is １ Since the number of image layers that need to be decoded can be easily expressed in order from the top, it is possible to simplify the control for determining the image layer to be decoded depending on the processing load status, etc.

[0148] (Third Modification) Fig. 15 is a simplified diagram showing part of a bitstream BS having a multi-layer structure, in which only one access unit is shown.

[0149] The encoding unit 33 encodes a plurality of image layers L １ ~L ３ A plurality of parameters P associated with １ ~P ３ All of these are stored in the first image layer L, which is the base layer. １ The data is stored together in the header area 41 of the file.

[0150] 16 is a diagram illustrating an example of syntax related to setting the parameter P. In this example, the parameter P is set as the value of mvi_optimized_for_first_vision_task_flag[i] in the SEI message, using a parameter i indicating the layer number of the image layer L.

[0151] According to this modification, the decoding device 2 that receives the bitstream BS decodes all the image layers L １ ~L ３ All parameters P associated with １ ~P ３ can be obtained collectively from the header area 41 of the base layer.

[0152] The encoding device 1 assigns the parameter P １ ~P ３ However, the encoding device 1 stores the parameter P only in the base layer of the first access unit constituting the bitstream BS. １ ~P ３ In this case, the parameter P １ ~P ３ The setting contents of the parameter P are inherited to the second and subsequent access units. Also, the encoding device 1 sets the parameter P only to the base layer of the intermediate access units constituting the bit stream BS. １ ~P ３ In this case, for the access units from the first access unit to the access unit immediately before the intermediate access unit, it is not determined whether the coded image is suitable for task processing, and for the access units after the intermediate access unit, the parameter P １ ~P ３ The settings will be carried over.

[0153] The encoding unit 33 also encodes a plurality of image layers L １ ~L ３ A plurality of parameters P associated with １ ~P ３ A part of the first image layer L１ The encoding unit 33 may store the parameters P １ , P ２ the first image layer L １ The parameter P ３ the third image layer L ３ The data is stored in the header area 41 of the data.

[0154] The encoding unit 33 also encodes a plurality of image layers L １ ~L ３ A plurality of parameters P associated with １ ~P ３ 6 or 11 as independent SEI messages having the syntax configurations described in FIG. 6 or 11, １ In this case, the image layers L may be stored together in the header area 41 of the image layer L. １ ~L ３ In order to store the above SEI messages together in one header area, an SN (scalable nesting)_SEI message may be used.

[0155] The present disclosure is particularly useful when applied to an image processing system that includes an encoding device that encodes an image into a bitstream and transmits the bitstream, and a decoding device that decodes an image from the received bitstream.

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, the circuit decoding at least one parameter associated with at least one image layer from a multi-layered bitstream including the image layer, the parameter indicating whether an image decoded from the image layer associated with the parameter is suitable for a predetermined task processing.

2. The decoding device according to claim 1, wherein the circuit further decodes an image from an image layer selected from the at least one image layer based on the parameters, and executes the task processing using the image decoded from the image layer.

3. The decoding device of claim 1, wherein the task processing includes machine vision.

4. The decoding device of claim 1, wherein the task processing includes human vision.

5. The decoding device of claim 1, wherein the task processing includes machine vision and human vision, and the at least one parameter includes a first parameter indicating whether an image decoded from the image layer is suitable for the machine vision, and a second parameter indicating whether an image decoded from the image layer is suitable for the human vision.

6. The decoding device of claim 1, wherein the parameters include a first value and a second value, the first value indicating that an image decoded from the image layer is suitable for the task processing, and the second value indicating that an image decoded from the image layer is not suitable for the task processing.

7. The decoding device of claim 1, wherein the parameters include a first value and a second value, the first value indicating that an image decoded from the image layer is suitable for the task processing, and the second value indicating that it is not specified whether the image decoded from the image layer is suitable for the task processing.

8. The decoding device according to claim 6 or 7, wherein the circuit further decodes an image from only an image layer among the at least one image layer that is associated with the parameter indicating the first value, and performs the task processing using the image decoded from the image layer.

9. The decoding device of claim 1, wherein the at least one image layer includes an image layer to which the parameter is not associated, and the absence of the parameter associated with the image layer indicates that it has not been determined whether an image decoded from the image layer is suitable for the task processing.

10. The decoding device of claim 1, wherein the circuit decodes the at least one parameter from a predetermined header region of the bitstream, the predetermined header region including an SEI.

11. The decoding device according to claim 10, wherein the at least one image layer includes a base layer which is the lowest layer of the multi-layer structure, and the at least one parameter associated with the at least one image layer is stored in the header area of ​​the base layer.

12. The decoding device according to claim 10, wherein the at least one parameter associated with the at least one image layer is stored in the header area of ​​each of the at least one image layer.

13. An encoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit encodes at least one parameter associated with at least one image layer into a multi-layered bitstream including the image layer, the parameter indicating whether an image in the image layer associated with the parameter is suitable for a predetermined task processing.

14. The encoding device of claim 13, wherein the processing task includes machine vision.

15. The encoding device of claim 13, wherein the task processing includes human vision.

16. The encoding device of claim 13, wherein the task processing includes machine vision and human vision, and the at least one parameter includes a first parameter indicating whether an image in the image layer is suitable for the machine vision, and a second parameter indicating whether an image in the image layer is suitable for the human vision.

17. The encoding device of claim 13, wherein the parameter includes a first value and a second value, the first value indicating that an image in the image layer is suitable for the task processing, and the second value indicating that an image in the image layer is not suitable for the task processing.

18. The encoding device of claim 13, wherein the parameter includes a first value and a second value, the first value indicating that an image in the image layer is suitable for the task processing, and the second value indicating that the image in the image layer is not specified as to whether it is suitable for the task processing.

19. The encoding device of claim 13, wherein the at least one image layer includes an image layer to which the parameter is not associated, and not associating the parameter with the image layer indicates that the image in the image layer is not specified as being suitable for the task processing.

20. The encoding device of claim 13, wherein the circuitry encodes the at least one parameter in a predetermined header region of the bitstream, the predetermined header region including an SEI.

21. The encoding device according to claim 20, wherein the at least one image layer includes a base layer which is the lowest layer in the multi-layer structure, and the at least one parameter associated with the at least one image layer is stored in the header area of ​​the base layer.

22. The encoding device according to claim 20, wherein said at least one parameter associated with said at least one image layer is stored in said header area of ​​each of said at least one image layer.

23. A decoding method, comprising: a decoding device decoding at least one parameter associated with at least one image layer from a multi-layered bitstream including the image layer, the parameter indicating whether an image decoded from the image layer associated with the parameter is suitable for a predetermined task processing.

24. An encoding method, in which an encoding device encodes at least one parameter associated with at least one image layer into a multi-layered bitstream including the image layer, the parameter indicating whether an image in the image layer associated with the parameter is suitable for a predetermined task processing.

Images