Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device is provided with: a circuit that decodes, on the basis of a bit stream, an image and an identifier associated with the image, the identifier indicating at least one of the condition that the image is suitable for task processing corresponding to the identifier and the image is not suitable for the task processing;

Description

Technical Field

[0001] This disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method. Background Technology

[0002] Patent document 1 discloses a method for determining computing power resources as described in the background art. The control node includes at least one of a terminal-side computing power control node and a network-side computing power control node. The control node receives first information representing resource information indicating the computing power available to the terminal-side computing power control node. Based on the first information, the control node determines context information of the computing power resources.

[0003] Prior art literature

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2023 / 050431 Summary of the Invention

[0006] The purpose of this disclosure is to improve the execution accuracy of task processing.

[0007] One aspect of the present disclosure relates to a decoding apparatus comprising a circuit and a memory connected to the circuit, the circuit decoding an image and an identifier associated with the image based on a bitstream, the identifier indicating at least one of the following: the image is suitable for task processing corresponding to the identifier, and the image is not suitable for task processing. Attached Figure Description

[0008] Figure 1 This is a simplified diagram illustrating the structure of an image processing system according to an embodiment of the present disclosure.

[0009] Figure 2 This diagram is a simplified representation of the circuitry of the encoding device.

[0010] Figure 3 This is a flowchart illustrating the processing performed by the circuitry of the encoding device.

[0011] Figure 4 This is a diagram showing the first example of image analysis based on the setting unit.

[0012] Figure 5 This is a diagram showing the second example of image analysis based on the setting unit.

[0013] Figure 6 This is a diagram showing the third example of image analysis based on the setting unit.

[0014] Figure 7 It is a diagram showing image data and identifier data.

[0015] Figure 8 This is a diagram illustrating the first example of syntax related to the setting of identifiers.

[0016] Figure 9 This is a diagram illustrating the first example of setting the value of an identifier in the syntax.

[0017] Figure 10 This is the second example of the syntax related to the setting of identifiers.

[0018] Figure 11 This is a diagram illustrating the second setting example of the value of an identifier in the syntax.

[0019] Figure 12 It is a simplified diagram showing the structure of the packet data containing the encoded images.

[0020] Figure 13 This diagram is a simplified representation of the circuitry of a decoding device.

[0021] Figure 14 This is a flowchart illustrating the processing performed by the circuitry of the decoding device.

[0022] Figure 15 This is a diagram illustrating the first example of image selection based on identifier data.

[0023] Figure 16 This is the second example of image selection based on identifier data.

[0024] Figure 17 It is a diagram showing image data and identifier data.

[0025] Figure 18 This is a diagram illustrating an example of syntax related to the setting of identifiers.

[0026] Figure 19 This is a diagram illustrating an example of syntax related to the setting of identifiers.

[0027] Figure 20 Is with Figure 9 The diagram correspondingly shows the first setting example of the value of the identifier in the syntax.

[0028] Figure 21 Is with Figure 10 The diagram correspondingly shows the second setting example of the value of the identifier in the syntax.

[0029] Figure 22 Therefore Figure 18 The diagram illustrates examples of syntax related to identifier settings.

[0030] Figure 23Therefore Figure 19 The diagram illustrates examples of syntax related to identifier settings.

[0031] Figure 24 This is a diagram illustrating examples of syntax related to task processing settings.

[0032] Figure 25 This diagram is a simplified representation of the circuitry of the encoding device.

[0033] Figure 26 This diagram is a simplified representation of the circuitry of the encoding device.

[0034] Figure 27 This is a simplified diagram illustrating the structure of the image processing system involved in the seventh variation.

[0035] Figure 28 This is a diagram illustrating an example of task processing settings.

[0036] Figure 29 This diagram is a simplified representation of the circuitry of the encoding device.

[0037] Figure 30 This is a flowchart illustrating the processing performed by the circuitry of the encoding device.

[0038] Figure 31 It is a diagram showing image data, first identifier data, and second identifier data.

[0039] Figure 32 This is a diagram showing the bit string containing the first identifier data and the second identifier data.

[0040] Figure 33A This is a diagram illustrating the first example of the syntax related to the setting of the first identifier and the second identifier.

[0041] Figure 33B This is a diagram illustrating the first example of the syntax related to the setting of the first identifier and the second identifier.

[0042] Figure 34A This is a diagram illustrating the second example of the syntax related to the setting of the first identifier and the second identifier.

[0043] Figure 34B This is a diagram illustrating the second example of the syntax related to the setting of the first identifier and the second identifier.

[0044] Figure 35 This diagram is a simplified representation of the circuitry of a decoding device.

[0045] Figure 36This is a flowchart illustrating the processing performed by the circuitry of the decoding device.

[0046] Figure 37 This is a diagram illustrating an example of the selection of a first image and a second image based on first identifier data and second identifier data.

[0047] Figure 38 This is a diagram illustrating an example of how the values ​​of identifiers are set in the syntax.

[0048] Figure 39 This is a diagram illustrating an example of the selection of a first image and a second image based on first identifier data and second identifier data.

[0049] Figure 40 This is a diagram illustrating examples of syntax related to task processing settings.

[0050] Figure 41 This is a simplified diagram illustrating the structure of a bitstream.

[0051] Figure 42 This is a diagram illustrating an example of the syntax related to the setting of the first identifier and the second identifier.

[0052] Figure 43 This is a diagram illustrating an example of the syntax related to the setting of the first identifier and the second identifier.

[0053] Figure 44 This is a diagram illustrating an example of the syntax related to the setting of the first identifier and the second identifier.

[0054] Figure 45 This is a diagram illustrating an example of the syntax related to the setting of the first identifier and the second identifier. Detailed Implementation

[0055] (The understanding that forms the basis of this disclosure)

[0056] The image processing system described in the background technology includes an encoding device, a decoding device, and a task processing unit. The encoding device acquires image data of a moving image captured by a camera and encodes the image data into a bitstream. The encoding device sends the bitstream to the decoding device. The decoding device receives the bitstream, decodes the image based on the received bitstream, and inputs the decoded image to the task processing unit. The task processing unit performs task processing based on the image input from the decoding device.

[0057] Task processing encompasses both human vision and machine tasks. Human vision involves visual recognition or audiovisual processing based on dynamic images of operators or users. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using inference models derived from machine learning.

[0058] The processing speed varies depending on the type of task, as the processing power of the hardware resources executing each task is determined. Therefore, the decoding device performs sparse processing on multiple images decoded from the bitstream and inputs the sparsely processed images into the task processing unit. For example, the decoding device performs sparse processing on the even-numbered frames of the decoded 60fps image, thereby inputting a 30fps image into the task processing unit.

[0059] However, when the sparse even-numbered frames contain images that are important to the task processing, the estimation accuracy of the estimation model decreases, which in turn reduces the execution accuracy of the task processing.

[0060] In order to solve this problem, the inventors came to the following understanding and thus conceived of this disclosure: information indicating whether each image is suitable for task processing is included in the bit stream and sent from the encoding device to the decoding device, and the decoding device performs sparse processing corresponding to the task processing based on the information, thereby solving the above-mentioned problem.

[0061] The various methods disclosed herein will now be explained.

[0062] The decoding apparatus according to the first aspect of this disclosure includes a circuit and a memory connected to the circuit, the circuit decoding an image and an identifier associated with the image according to a bit stream, the identifier indicating that the image is suitable for task processing corresponding to the identifier and at least one of the following: the image is not suitable for task processing.

[0063] According to the first method, the identifier indicates at least one of the following: an image is suitable for the task processing corresponding to the identifier, and an image is not suitable for the task processing. Therefore, at least one image used in the task processing can be appropriately selected from multiple images decoded from the bitstream based on the identifier, thereby improving the execution accuracy of the task processing.

[0064] In the first embodiment, the decoding apparatus of the second aspect of this disclosure may contain multiple images, and the circuit further selects at least one image for use in the task processing from among the multiple images based on the identifier.

[0065] According to the second method, images suitable for task processing can be selected based on identifiers, thereby improving the execution accuracy of task processing.

[0066] The decoding apparatus involved in the third aspect of this disclosure is in the first or second aspect, wherein the task processing may include machine tasks.

[0067] According to the third method, it is possible to select an image suitable for the machine task based on an identifier, thereby enabling the machine task to be performed appropriately.

[0068] The decoding apparatus involved in the fourth aspect of this disclosure, in any of the first to third aspects, may include human vision in the task processing.

[0069] According to the fourth method, it is possible to select images suitable for human vision based on identifiers, thereby enabling the proper execution of human vision.

[0070] In any of the first to fourth embodiments of the decoding apparatus disclosed herein, the identifier may include multiple identifiers, the task processing includes multiple task processing, and the multiple identifiers correspond to different task processing among the multiple task processing.

[0071] According to method 5, multiple identifiers can correspond to multiple task processing.

[0072] In the fifth embodiment, the decoding apparatus of the sixth aspect of this disclosure includes a plurality of identifiers, which may include a first identifier and a second identifier. The task processing corresponding to the first identifier includes a machine task, and the task processing corresponding to the second identifier includes human vision.

[0073] According to the sixth method, an image suitable for the machine task can be selected based on the first identifier, thereby enabling the appropriate execution of the machine task. Furthermore, an image suitable for human vision can be selected based on the second identifier, thereby enabling the appropriate execution of human vision. In other words, depending on the nature of the image, either the machine task or human vision can be appropriately executed based on both the first and second identifiers.

[0074] In any of the first to sixth methods of the decoding apparatus disclosed in the seventh method, the identifier may include a first value and a second value, wherein the first value indicates that the image is suitable for the task processing and the second value indicates that the image is not suitable for the task processing.

[0075] According to the seventh method, the circuit can easily determine the selected image and the unselected image based on the first value and the second value.

[0076] In any of the first to sixth methods of the decoding apparatus disclosed herein, the identifier may include a first value and a second value, wherein the first value indicates that the image is suitable for the task processing or the image is not suitable for the task processing, and the second value indicates that it is not determined whether the image is suitable for the task processing.

[0077] According to the eighth method, the circuit can easily determine whether to select or not select an image based on the first value. Furthermore, the circuit can arbitrarily decide whether to select an image associated with an identifier of the second value, depending on its processing capabilities, etc.

[0078] In any of the first to eighth embodiments of the decoding apparatus disclosed in the ninth embodiment, the image may comprise multiple images, including images not associated with the identifier, wherein the images not associated with the identifier represent images whose suitability for the task processing has not been determined.

[0079] According to method 9, it is possible to arbitrarily decide whether to select an image that is not associated with an identifier based on its own processing capabilities, etc.

[0080] In any of the 1st to 9th embodiments of the present disclosure, the identifier may include more than one identifier, the task processing includes more than one task processing, and the circuit further decodes the corresponding information representing the correspondence between the more than one identifier and the more than one task processing according to the bit stream.

[0081] According to the 10th method, more than one task processing can be arbitrarily set in the corresponding information, and for each of the more than one task processing set, the execution accuracy of the task processing can be improved.

[0082] In the 10th embodiment of the decoding apparatus according to the 11th embodiment of this disclosure, the corresponding information can be stored in the header area of ​​the bit stream.

[0083] According to method 11, the circuit can easily obtain the corresponding information from the header region of the bit stream.

[0084] In any of the 1st to 11th embodiments of the decoding apparatus disclosed herein, the identifier may be stored in the header area of ​​the image associated with the identifier.

[0085] According to method 12, the circuit can easily obtain the identifier associated with each image from the header area of ​​each image.

[0086] In any of the 1st to 11th embodiments of the decoding apparatus disclosed herein, the image may contain multiple images, the identifier may contain one or more identifiers, the one or more identifiers are stored in the header area of ​​one of the multiple images, and are associated with the one image and one or more images different from the one image.

[0087] According to method 13, the circuit can easily obtain more than one identifier associated with more than one image from the header area of ​​an image.

[0088] In any of the 11th to 13th embodiments of the decoding apparatus involved in the 14th embodiment of this disclosure, the header region can be the SEI region.

[0089] According to method 14, the circuit can easily obtain the identifier from the SEI region.

[0090] In any of the 1st to 14th embodiments of the present disclosure, the image may comprise multiple images, and the circuit further decodes, based on the bit stream, determination information for identifying the image among the multiple images that is associated with the identifier.

[0091] According to method 15, by determining the information, the image associated with the identifier is determined, thus eliminating the need for the number of identifiers to match the number of images, thereby reducing the amount of data for the identifiers. This reduces the number of symbols and improves coding efficiency.

[0092] The encoding apparatus according to the 16th aspect of this disclosure includes a circuit and a memory connected to the circuit, the circuit encoding an image and an identifier associated with the image into a bit stream, the identifier indicating that the image is suitable for task processing corresponding to the identifier and that the image is not suitable for the task processing at least one.

[0093] According to the 16th method, the identifier indicates at least one of the following: an image is suitable for task processing corresponding to the identifier, and an image is not suitable for task processing. Therefore, the decoding device receiving the bitstream can appropriately select at least one image for use in task processing from among multiple images decoded from the bitstream based on the identifier, thereby improving the execution accuracy of task processing.

[0094] In the 16th embodiment of the encoding apparatus according to the 17th embodiment of this disclosure, the circuit can set the value of the identifier based on the encoding conditions when encoding the image.

[0095] According to the 17th method, the circuit can appropriately determine whether an image is suitable for task processing based on the encoding conditions after the image is encoded.

[0096] In the 16th or 17th embodiment of the present disclosure, the circuitry of the encoding apparatus can set the value of the identifier based on the image quality of the encoded image.

[0097] According to method 18, the circuit can appropriately determine whether an image is suitable for task processing based on its image quality after the image is encoded.

[0098] In any of the 16th to 18th embodiments of the encoding apparatus according to the 19th embodiment of this disclosure, the circuit can obtain an image of the encoded object and associated information associated with the image from an external device, and set the value of the identifier based on the associated information.

[0099] According to method 19, the circuit can appropriately determine whether an image is suitable for task processing based on associated information before the image is encoded.

[0100] In any of the 16th to 19th embodiments of the encoding apparatus involved in the 20th embodiment of this disclosure, the circuit can perform image parsing on an image of the encoded object and set the value of the identifier based on the result of the image parsing.

[0101] According to method 20, the circuit can appropriately determine whether an image is suitable for task processing based on the results of image parsing before image encoding.

[0102] In any of the embodiments 16 to 20, the encoding apparatus involved in the 21st embodiment of this disclosure may include machine tasks in the task processing.

[0103] According to the 21st method, the decoding device that receives the bit stream can select an image suitable for the machine task based on the identifier, thereby enabling the appropriate execution of the machine task.

[0104] The encoding device involved in the 22nd aspect of this disclosure, in any of the 16th to 21st aspects, may include human vision in the task processing.

[0105] According to the 22nd method, the decoding device that receives the bit stream can select an image suitable for human vision based on an identifier, thereby enabling the appropriate execution of human vision.

[0106] In any of the embodiments 16 to 22, the encoding apparatus of the 23rd aspect of this disclosure may include multiple identifiers, the task processing includes multiple task processing, and the multiple identifiers correspond to different task processing among the multiple task processing.

[0107] According to method 23, multiple identifiers can correspond to multiple task processing.

[0108] In the 23rd aspect, the encoding apparatus of the 24th aspect of this disclosure includes a plurality of identifiers, which may include a first identifier and a second identifier, wherein the task processing corresponding to the first identifier includes a machine task and the task processing corresponding to the second identifier includes human vision.

[0109] According to the 24th method, the decoding device receiving the bitstream can select an image suitable for a machine task based on the first identifier, thereby enabling the appropriate execution of the machine task. Furthermore, the decoding device can select an image suitable for human vision based on the second identifier, thereby enabling the appropriate execution of human vision. In other words, depending on the nature of the image, either a machine task or human vision can be appropriately executed based on both the first and second identifiers.

[0110] In any of the 16th to 24th embodiments of the encoding apparatus involved in the 25th embodiment of this disclosure, the identifier may include a first value and a second value, wherein the first value indicates that the image is suitable for the task processing and the second value indicates that the image is not suitable for the task processing.

[0111] According to method 25, the decoding device that receives the bitstream can easily determine the selected image and the unselected image based on the first value and the second value.

[0112] In any of the embodiments 16 to 24, the identifier of the encoding apparatus involved in the 26th embodiment of this disclosure may include a first value and a second value, wherein the first value indicates that the image is suitable for the task processing or the image is not suitable for the task processing, and the second value indicates that it is not determined whether the image is suitable for the task processing.

[0113] According to the 26th method, the decoding device receiving the bitstream can easily determine whether to select an image or not based on the first value. Furthermore, the decoding device can arbitrarily decide whether to select an image associated with an identifier corresponding to the second value, depending on its own processing capabilities, etc.

[0114] In any of the 16th to 26th embodiments of the encoding apparatus according to the 27th embodiment of this disclosure, the image may comprise a plurality of images, the plurality of images including images not associated with the identifier, the images not associated with the identifier representing images whose suitability for the task processing is not determined.

[0115] According to method 27, the decoding device that receives the bitstream can arbitrarily decide whether to select an image that is not associated with the identifier, based on its own processing capabilities, etc.

[0116] In any of the 16th to 27th embodiments of the encoding apparatus involved in the 28th embodiment of this disclosure, the identifier may include more than one identifier, the task processing includes more than one task processing, and the circuit further encodes the correspondence information representing the more than one identifier and the more than one task processing into the bit stream.

[0117] According to method 28, more than one task processing can be arbitrarily set in the corresponding information, and for each of the more than one task processing set, the execution accuracy of the task processing can be improved.

[0118] In any of the embodiments 16 to 28, the corresponding information of the encoding device involved in the 29th embodiment of this disclosure can be stored in the header area of ​​the bit stream.

[0119] According to method 29, the decoding device that receives the bit stream can easily obtain the corresponding information from the header area of ​​the bit stream.

[0120] In any of the embodiments 16 to 29, the encoding device involved in the 30th embodiment of this disclosure may have the identifier stored in the header area of ​​the image associated with the identifier.

[0121] According to method 30, the decoding device that receives the bit stream can easily obtain the identifier associated with each image from the header region of each image.

[0122] In any of the embodiments 16 to 30, the encoding device according to the 31st aspect of this disclosure, the image may include multiple images, the identifier includes more than one identifier, the more than one identifier is stored in the header area of ​​one of the multiple images, and is associated with the one image and more than one image different from the one image.

[0123] According to method 31, the decoding device that receives the bit stream can easily obtain more than one identifier associated with more than one image from the header region of an image.

[0124] In any of the embodiments 29 to 31, the header region of the decoding apparatus involved in the 32nd embodiment of this disclosure may be the SEI region.

[0125] According to method 32, the circuit can easily obtain the identifier from the SEI region.

[0126] In any of the embodiments 16 to 32, the image may comprise multiple images, and the circuit further encodes determination information for identifying the image among the multiple images that is associated with the identifier into the bit stream.

[0127] According to method 33, the image associated with the identifier is determined by determining the information, thus eliminating the need for the number of identifiers to match the number of images, thereby reducing the amount of data for the identifiers. This reduces the number of symbols and improves coding efficiency.

[0128] In the decoding method according to the 34th aspect of this disclosure, the decoding device decodes an image and an identifier associated with the image based on a bitstream, the identifier indicating that the image is suitable for task processing corresponding to the identifier and that the image is not suitable for the task processing at least one.

[0129] According to method 34, the identifier indicates at least one of the following: an image is suitable for task processing corresponding to the identifier, and an image is not suitable for task processing. Therefore, at least one image used in task processing can be appropriately selected from multiple images decoded from the bitstream based on the identifier, thereby improving the execution accuracy of task processing.

[0130] In the encoding method according to the 35th aspect of this disclosure, the encoding device encodes an image and an identifier associated with the image into a bit stream, the identifier indicating that the image is suitable for task processing corresponding to the identifier and that the image is not suitable for the task processing at least one.

[0131] According to method 35, the identifier indicates at least one of the following: an image is suitable for task processing corresponding to the identifier, and an image is not suitable for task processing. Therefore, the decoding device receiving the bitstream can appropriately select at least one image for use in task processing from among multiple images decoded from the bitstream based on the identifier, thereby improving the execution accuracy of task processing.

[0132] (Implementation of this disclosure)

[0133] Hereinafter, embodiments of the present disclosure will be described in detail using the accompanying drawings. Furthermore, elements given the same reference numerals in different drawings represent the same or corresponding elements.

[0134] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, structural elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, structural elements in the following embodiments that are not described in the independent technical solutions representing the highest-level concept are described as arbitrary structural elements. Furthermore, various contents can be combined in all embodiments.

[0135] Figure 1 This diagram illustrates a simplified structure of the image processing system according to an embodiment of the present disclosure. The image processing system includes: an encoding device 1, a decoding device 2, an imaging unit 3, a task processing unit 4, and a transmission path NW.

[0136] The imaging unit 3 is an example of an external device, which includes a camera for capturing moving images. The imaging unit 3 inputs the image data D1 of the captured moving images to the encoding device 1.

[0137] Encoding device 1 sends bit stream BS to decoding device 2 via transmission path NW. Decoding device 2 receives bit stream BS.

[0138] The task processing unit 4 performs task processing based on the image data D2 input from the decoding device 2. Task processing includes both human vision and machine tasks. Human vision involves visual recognition or audiovisual processing of dynamic images of operators or users. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using inference models derived from machine learning. The task processing unit 4 performing human vision tasks includes a display device such as a liquid crystal display (LCD) or an organic EL display. The task processing unit 4 performing machine tasks includes an inferrer using AI.

[0139] The transmission path NW can be the Internet, WAN (Wide Area Network), LAN (Local Area Network), or any combination thereof. The desired transmission path NW is a private network that ensures secure communication through access restrictions, etc.

[0140] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 is configured to include a processor such as a CPU. The memory 12 is configured to include any recording medium such as ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data processed by the circuit 11 or data processed during the process.

[0141] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 is configured to include a processor such as a CPU. The memory 22 is configured to include any recording medium such as ROM, RAM, HDD, SSD, or semiconductor memory. The memory 22 stores data processed by the circuit 21 or data processed during the process.

[0142] Figure 2 This diagram shows a simplified representation of the structure of the circuit 11 included in the encoding device 1. The circuit 11 includes a setting unit 31 and an encoding unit 32.

[0143] Figure 3 This is a flowchart showing the process performed by the circuit 11 of the encoding device 1.

[0144] First, in step SP11, circuit 11 acquires multiple images contained in image data D1 input from imaging unit 3. Image data D1 is then input to setting unit 31 and encoding unit 32.

[0145] Next, in step SP12, the setting unit 31 performs image parsing on each image of the encoded object contained in the image data D1, and sets the value of the identifier based on the result of the image parsing. The setting unit 31 inputs the identifier data D11 containing the identifier with the set value to the encoding unit 32.

[0146] An identifier indicates at least one of the following: an image is suitable (i.e., appropriate) for the task processing corresponding to the identifier, or an image is unsuitable (i.e., inappropriate) for the task processing corresponding to the identifier. An image being suitable for task processing includes: recommending the use of the image when performing task processing on the decoding device 2 side, or performing image processing suitable for task processing on the image. An image being unsuitable for task processing includes: not recommending (i.e., not recommending) the use of the image when performing task processing on the decoding device 2 side, or not performing image processing suitable for task processing on the image.

[0147] Next, in step SP13, the encoding unit 32 encodes the multiple images contained in the image data D1 input from the shooting unit 3 and the multiple identifiers contained in the identifier data D11 input from the setting unit 31 into a bit stream BS.

[0148] Next, in step SP14, circuit 11 sends the bit stream BS to decoding device 2 via transmission path NW.

[0149] Figure 4 This diagram illustrates the first example of image analysis based on the setting unit 31. Figure 4The image shows three images in time sequence corresponding to times T0, T1, and T2. The image corresponding to time T0 includes trees and pedestrians. The image corresponding to time T1 includes trees and a pedestrian who has moved slightly since time T0. The image corresponding to time T2 includes trees, a pedestrian who has moved slightly since time T1, and a running person. In the case where the task processing corresponding to the identifier is human vision, the setting unit 31 sets all images to "appropriate" for smooth animation reproduction.

[0150] Figure 5 This diagram illustrates a second example of image analysis based on the setting unit 31. When the task processing corresponding to the identifier is target detection, since pedestrian detection has been completed at time T0, the setting unit 31 sets the images corresponding to times T0 and T2 as "appropriate" and the image corresponding to time T1 as "inappropriate".

[0151] Figure 6 This diagram illustrates the third example of image analysis based on the setting unit 31. In the case where the task processing corresponding to the identifier is target tracking, since movement needs to be tracked even if the amount of movement is small, the setting unit 31 sets all images to "appropriate".

[0152] Alternatively, the setting unit 31 can set the value of the identifier based on the image quality before encoding. The setting unit 31 evaluates the image quality before encoding based on factors such as sharpening value, contrast value, noise level, or PSNR (Peak Signal to Noise Ratio) value. The setting unit 31 can also use machine learning algorithms to evaluate image quality. Furthermore, the setting unit 31 can also analyze the image's histogram to evaluate image quality based on whether the image is overexposed, underexposed, or moderately exposed.

[0153] When the task is human vision, low-quality images reduce visual recognition. Furthermore, when the task is object detection, low-quality images reduce estimation accuracy. Therefore, when the task is human vision or object detection, the setting unit 31 can also set images with pre-encoding quality above a threshold as "appropriate" and images with pre-encoding quality below the threshold as "inappropriate".

[0154] When the task is target tracking, the tolerance for changes in image quality is increased, so even low-quality images can detect the trajectory of the target. Therefore, when the task is target tracking, the setting unit 31 can set all images to "appropriate".

[0155] Thus, the setting unit 31 determines whether the image is suitable or unsuitable for task processing based on the type of task processing corresponding to the identifier. In this embodiment, the task processing performed by the task processing unit 4 is set to a specific type, and the encoding device 1 and the decoding device 2 share this setting information in advance.

[0156] Figure 7 This diagram illustrates image data D1 and identifier data D11. Image data D1 contains multiple images in chronological order (in this example, five images P1 to P5). Identifier data D11 contains identifiers associated with each image P1 to P5. Each identifier is a 1-bit data element, and its value is set to either "1" or "0".

[0157] Figure 8 This is a diagram illustrating the first example of syntax related to the setting of identifiers. The syntax includes the item "frame_for_inference".

[0158] Figure 9 This is a diagram illustrating the first example of setting the value of an identifier in the syntax. For example... Figure 9 As shown in (A), setting the value of "frame_for_inference" to "1" indicates that the associated image is "appropriate". Conversely, setting the value of "frame_for_inference" to "0" indicates that the associated image is "inappropriate".

[0159] In addition, such as Figure 9 As shown in (B), when the value of “frame_for_inference” is set to “0”, it can also indicate that “appropriate” or “inappropriate” is not specified on the encoding device 1 side, that is, it is not determined on the encoding device 1 side whether the associated image is suitable for task processing.

[0160] Figure 10 This is the second example of the syntax related to the setting of identifiers. The syntax includes the item "frame_not_for_inference".

[0161] Figure 11 This is a diagram illustrating the second example of setting the value of an identifier in the syntax. For example... Figure 11 As shown in (A), setting the value of "frame_not_for_inference" to "1" indicates that the associated image is "inappropriate". Conversely, setting the value of "frame_not_for_inference" to "0" indicates that the associated image is "appropriate".

[0162] In addition, such as Figure 11 As shown in (B), when the value of “frame_not_for_inference” is set to “0”, it can also indicate that “appropriate” or “inappropriate” is not specified on the encoding device 1 side, that is, it is not determined on the encoding device 1 side whether the associated image is suitable for task processing.

[0163] Figure 12 This diagram shows a simplified representation of the structure of packet data PD containing encoded images P1 to P5. The packet data PD has a header area 41 and a payload area 42. The header area 41 has a Supplemental Enhancement Information (SEI) area 43 for storing additional information. The encoding unit 32 stores the encoded data of the images in the payload area 42 and the encoded data of the identifier associated with the images in the SEI area 43. However, it can also be stored in VPS, SPS, PPS, PH, SH, APS, or the block header instead of the SEI area 43. Furthermore, the SEI area 43 can be a prefix_SEI located before the payload area 42 or a suffix_SEI located after the payload area 42.

[0164] Figure 13 This diagram shows a simplified representation of the circuit 21 included in the decoding device 2. The circuit 21 includes a decoding unit 51 and a selection unit 52.

[0165] Figure 14 This is a flowchart showing the process performed by the circuit 21 of the decoding device 2.

[0166] First, in step SP21, circuit 21 receives bit stream BS from encoding device 1 via transmission path NW. Bit stream BS is then input to decoding unit 51.

[0167] Next, in step SP22, circuit 21 decodes multiple images and identifiers associated with each image based on bitstream BS, and outputs image data D21, which corresponds to image data D1, and identifier data D22, which corresponds to identifier data D11. Image data D21 and identifier data D22 are input to selection unit 52.

[0168] Next, in step SP23, the selection unit 52 selects at least one image for task processing from among the multiple images contained in the image data D21 based on the identifier contained in the identifier data D22.

[0169] Next, in step SP24, the selection unit 52 outputs image data D2 containing at least one selected image. Image data D2 is input to the task processing unit 4. The task processing unit 4 uses the at least one image contained in the input image data D2 to perform task processing.

[0170] Figure 15 This is the first example of image selection based on identifier data D22. Figure 15 In, with Figure 9 (A) correspondingly shows the following example: when the value of “frame_for_inference” is set to “1”, it means that the associated image is “appropriate”, and when the value of “frame_for_inference” is set to “0”, it means that the associated image is “inappropriate”.

[0171] Image data D21 contains multiple images in chronological order (in this example, five images P1 to P5). Identifier data D22 contains identifiers associated with each image P1 to P5. For images P1 to P5, the identifier values ​​are set to "1", "0", "0", "1", and "1", respectively.

[0172] The selection unit 52 selects images P1, P4, and P5 with an identifier value of "1" from the plurality of images P1 to P5 contained in the image data D21, and does not select images P2 and P3 with an identifier value of "0". As a result, the selection unit 52 outputs image data D2 containing the selected images P1, P4, and P5.

[0173] Figure 16 This is the second example of image selection based on identifier data D22. Figure 16 In, with Figure 11 (A) correspondingly shows the following example: when the value of “frame_not_for_inference” is set to “1”, it means that the image with which the association is established is “inappropriate”, and when the value of “frame_not_for_inference” is set to “0”, it means that the image with which the association is established is “appropriate”.

[0174] The selection unit 52 selects images P1 to P5, which are multiple images P1 to P5 contained in the image data D21, but does not select images P1, P4, and P5 whose identifier value is "1", and instead selects images P2 and P3 whose identifier value is "0". As a result, the selection unit 52 outputs image data D2 containing the selected images P2 and P3.

[0175] Furthermore, the designation of an image based on an identifier is merely a suggestion to determine whether it is appropriate or inappropriate; it does not force a selection or non-selection based on the selection unit 52. Therefore, depending on the processing capabilities of the hardware resources of the decoding device 2, the selection unit 52 may choose not to select an appropriate image when it lacks the necessary processing power, or it may choose to select an inappropriate image when it has the necessary processing power.

[0176] According to the encoding apparatus 1 of this embodiment, the identifier indicates at least one of the following: an image is suitable for task processing corresponding to the identifier, and an image is not suitable for task processing. Therefore, the decoding apparatus 2 that receives the bitstream BS can appropriately select at least one image used in task processing from among multiple images decoded from the bitstream BS based on the identifier, thereby improving the execution accuracy of task processing.

[0177] Furthermore, according to the encoding apparatus 1 of this embodiment, the circuit 11 can appropriately determine whether an image is suitable for task processing based on the result of image parsing before the image is encoded.

[0178] Furthermore, according to the encoding apparatus 1 of this embodiment, task processing includes machine tasks. Therefore, the decoding apparatus 2 that receives the bitstream BS can select an image suitable for the machine task based on an identifier, thereby enabling the appropriate execution of the machine task.

[0179] Furthermore, according to the encoding apparatus 1 of this embodiment, task processing includes human vision. Therefore, the decoding apparatus 2 that receives the bitstream BS can select an image suitable for human vision based on an identifier, thereby enabling the appropriate execution of human vision.

[0180] Furthermore, according to the encoding apparatus 1 of this embodiment, the identifier includes a first value and a second value, where the first value indicates that the image is suitable for task processing, and the second value indicates that the image is not suitable for task processing. Therefore, the decoding apparatus 2 that receives the bitstream BS can easily determine the selected image and the unselected image based on the first value and the second value.

[0181] Furthermore, according to the encoding apparatus 1 of this embodiment, the identifier includes a first value and a second value. The first value indicates whether the image is suitable for task processing or not, and the second value indicates whether it is undetermined whether the image is suitable for task processing. Therefore, the decoding apparatus 2 that receives the bitstream BS can easily determine whether to select or not select an image based on the first value. In addition, the decoding apparatus 2 can arbitrarily decide whether to select an image associated with the identifier of the second value based on its own processing capabilities, etc.

[0182] Furthermore, according to the encoding apparatus 1 of this embodiment, the identifier is stored in the header area 41 of the image associated with the identifier. Therefore, the decoding apparatus 2 that receives the bitstream BS can easily obtain the identifier associated with each image from the header area 41 of each image.

[0183] According to the decoding apparatus 2 of this embodiment, the identifier indicates at least one of the following: an image is suitable for task processing corresponding to the identifier, and an image is not suitable for task processing. Therefore, at least one image used in task processing can be appropriately selected from a plurality of images decoded from the bitstream BS based on the identifier, thereby improving the execution accuracy of task processing.

[0184] Furthermore, the decoding device 2 according to this embodiment can select an image suitable for task processing based on an identifier, thereby improving the execution accuracy of task processing.

[0185] Furthermore, according to the decoding apparatus 2 of this embodiment, task processing includes machine tasks. Therefore, it is possible to select an image suitable for the machine task based on an identifier, thereby enabling the machine task to be executed appropriately.

[0186] Furthermore, according to the decoding apparatus 2 of this embodiment, task processing includes human vision. Therefore, it is possible to select an image suitable for human vision based on an identifier, thereby enabling the appropriate execution of human vision.

[0187] Furthermore, according to the decoding device 2 of this embodiment, the identifier includes a first value and a second value, where the first value indicates that the image is suitable for task processing, and the second value indicates that the image is not suitable for task processing. Therefore, the circuit 21 can easily determine the selected image and the unselected image based on the first value and the second value.

[0188] Furthermore, according to the decoding apparatus 2 of this embodiment, the identifier includes a first value and a second value. The first value indicates whether the image is suitable for task processing or not, and the second value indicates that it is uncertain whether the image is suitable for task processing. Therefore, the circuit 21 can easily determine whether to select or not select an image based on the first value. In addition, the circuit 21 can arbitrarily decide whether to select an image associated with the identifier of the second value based on its own processing capabilities, etc.

[0189] Furthermore, according to the decoding apparatus 2 of this embodiment, the identifier is stored in the header area 41 of the image associated with the identifier. Therefore, the circuit 21 can easily obtain the identifier associated with each image from the header area 41 of each image.

[0190] Hereinafter, various modifications of the embodiments of this disclosure will be described. These modifications can be combined in any way.

[0191] (Example 1)

[0192] In the above embodiment, each identifier is stored in the header area 41 of each image associated with that identifier, but this is not a limitation. Circuit 11 may also define an image group containing multiple images, and store multiple identifiers associated with the multiple images contained in the image group in the header area 41 of one image (e.g., the first image) within the image group. The image group may be a GOP (Group of Pictures) containing I-images, P-images, and B-images, or it may be any collection of multiple images.

[0193] Figure 17 This diagram illustrates image data D1 and identifier data D11. Image data D1 contains multiple images in chronological order (32 images P1 to P32 in this example). Images P1 to P32 are contained within the same image group. Identifier data D11 contains 32 identifiers associated with the 32 images P1 to P32. This set of 32 identifiers is stored in the header area 41 of image P1 at the beginning of the image group.

[0194] In addition, Figure 17 In this context, one identifier can be associated with one image, but it is also possible to associate one identifier with multiple images. For example, if multiple identifiers with the same value are consecutive, the first identifier among these multiple identifiers can be associated with the multiple images corresponding to those identifiers.

[0195] Figure 18 This is a diagram illustrating an example of the syntax related to the setting of identifiers. The syntax includes the item "num_frames_minus1". The value of "num_frames_minus1" is set to the value obtained by subtracting "1" from the number of images contained in the image group. In the case where the image group contains 32 images P1 to P32, the value of "num_frames_minus1" is set to "31".

[0196] The `for` loop is used to set the value of `frame_for_inference` associated with each of the multiple images in the image group. Alternatively, the order in which the `for` loop processes the data can be used to determine which image each `frame_for_inference` is associated with.

[0197] Furthermore, multiple settings can be described as detailed entries instead of using repetitive statements such as for loops. The same applies to the subsequent variations.

[0198] According to this variant, the decoding device 2 that receives the bit stream BS can easily obtain multiple identifiers associated with multiple images contained in the image group from the header area 41 of an image.

[0199] (Second variation)

[0200] In the first variation, the same number of identifiers as the multiple images contained in the image group are stored in the header area 41 of one image within the image group, but this is not a limitation. Circuit 11 can also encode the identification information used to determine the image associated with the identifier among the multiple images contained in the image group into a bit stream BS.

[0201] Figure 19 This is a diagram illustrating an example of the syntax related to the setting of an identifier. The syntax includes an item called "image_identifier[i]" that represents identifying information. The value of "image_identifier[i]" is set using POC (Picture OrderCount), the difference of POC, the count, the ID of SPS (Sequence Parameter Set), or the ID of PPS (Picture Parameter Set), etc.

[0202] Set the value of "frame_for_inference" for the image identified by the given information among multiple images in the image group.

[0203] Furthermore, images that are not identified by the information, that is, images that are not associated with an identifier, indicate that it is uncertain whether the image is suitable for task processing.

[0204] Additionally, if the first image in an image group does not contain a set of identifiers, it indicates that it is not determined whether any of the images in that group are suitable for the task.

[0205] According to this variation, the image associated with the identifier is determined by identifying information, so it is not necessary for the number of identifiers in the identifier set to be consistent with the number of images in the image group, thereby reducing the amount of identifier data.

[0206] (3rd variation)

[0207] In the above embodiments, the identifier is associated with all images, but is not limited to this. Circuit 11 may also choose not to associate the identifier with images whose suitability for task processing is not yet determined.

[0208] Figure 20 Is with Figure 9 The diagram correspondingly shows the first example of setting the value of the identifier in the syntax. For example... Figure 20 As shown in (A), setting the value of "frame_for_inference" to "1" indicates that the associated image is "appropriate". Setting the value of "frame_for_inference" to "0" indicates that the associated image is "inappropriate". Furthermore, an image not associated with an identifier indicates that it has not been determined on the encoding device 1 side whether the image is suitable for task processing.

[0209] In addition, such as Figure 20 As shown in (B), it is also possible that the value of "frame_for_inference" is set to "0" or that the image not associated with the identifier indicates that it has not been determined on the encoding device 1 side whether the image is suitable for task processing.

[0210] Figure 21 Is with Figure 10 The diagram correspondingly shows the second example of setting the value of the identifier in the syntax. For example... Figure 21 As shown in (A), setting the value of "frame_not_for_inference" to "1" indicates that the associated image is "inappropriate". Setting the value of "frame_not_for_inference" to "0" indicates that the associated image is "appropriate". Furthermore, an image not associated with an identifier indicates that it has not been determined on the encoding device 1 side whether the image is suitable for task processing.

[0211] In addition, such as Figure 21 As shown in (B), it is also possible that the value of "frame_not_for_inference" is set to "0" or that the image not associated with the identifier indicates that it has not been determined on the encoding device 1 side whether the image is suitable for task processing.

[0212] Figure 22 Therefore Figure 18 A diagram illustrating examples of syntax related to the setting of identifiers. Figure 23 Therefore Figure 19The diagram illustrates an example of the syntax related to the setting of identifiers. The syntax includes the item "frame_for_inference_present_flag[i]". For images associated with an identifier, the value of "frame_for_inference_present_flag[i]" is set to "1". For images not associated with an identifier, the value of "frame_for_inference_present_flag[i]" is set to "0".

[0213] (4th variation)

[0214] In the above embodiment, the task processing performed by the task processing unit 4 is set to a specific type, but it is not limited to this. The circuit 11 can also encode setting information for arbitrarily setting the type of task processing into a bit stream BS. This setting information can be stored in the SEI area included in the header area of ​​the bit stream BS.

[0215] Figure 24 This diagram illustrates an example of the syntax related to task processing settings. The syntax includes a "vision_task" item. Within the "vision_task" item, a string representing the type of task processing is described. Alternatively, instead of a string, an integer value representing the type of task processing may be described. In this case, the correspondence between the integer value and the type of task processing is set, and this setting information is shared in advance by the encoding device 1 and the decoding device 2.

[0216] According to this variation, the type of task processing corresponding to the identifier can be arbitrarily set.

[0217] (5th variation)

[0218] In the above embodiment, the setting unit 31 sets the value of the identifier based on the result of image parsing of the image before encoding, but it is not limited to this. The setting unit 31 may also set the value of the identifier based on the encoded image.

[0219] Figure 25This diagram shows a simplified representation of the circuit 11 included in the encoding device 1. The circuit 11 includes a setting unit 31 and an encoding unit 32. Image data D1, containing multiple images before encoding, and image data D12, containing multiple images encoded by the encoding unit 32, are input to the setting unit 31. Based on the image data D1 and D12, the setting unit 31 derives the encoding conditions for the encoding unit 32 to encode each image. The encoding conditions include values ​​related to the image quality of the encoded image, such as the value of the quantization parameter, the PSNR value of the encoded image, and the temporal layer level. A higher quantization parameter value results in lower image quality after encoding. A lower PSNR value also results in lower image quality. A higher temporal layer level results in lower image quality after encoding.

[0220] When the task is human vision, low-quality images reduce visual recognition. Furthermore, when the task is object detection, low-quality images reduce estimation accuracy. Therefore, when the task is human vision or object detection, the setting unit 31 can also set images with encoded image quality above a threshold as "appropriate" and images with encoded image quality below the threshold as "inappropriate".

[0221] When the task is target tracking, the tolerance for changes in image quality is high, so even low-quality images can detect the target's trajectory. Therefore, when the task is target tracking, the setting unit 31 can set all images to "appropriate".

[0222] The setting unit 31 inputs identifier data D11, which contains identifiers whose values ​​are set based on the encoded image, into the encoding unit 32.

[0223] According to this variation, circuit 11 can appropriately determine whether an image is suitable for task processing based on the encoding conditions after the image is encoded.

[0224] Furthermore, according to this modified example, circuit 11 can appropriately determine whether an image is suitable for task processing based on image quality after the image is encoded.

[0225] (Sixth variation)

[0226] In the above embodiment, the setting unit 31 sets the value of the identifier based on the result of image parsing of the image before encoding, but it is not limited to this. The setting unit 31 may also set the value of the identifier based on association information associated with the image before encoding.

[0227] Figure 26This diagram shows a simplified representation of the structure of the circuit 11 included in the encoding device 1. The circuit 11 includes a setting unit 31 and an encoding unit 32. Image data D1, which includes multiple images before encoding, and correlation information D13 detected by the imaging unit 3 when it captures moving images, are input from the imaging unit 3 to the setting unit 31.

[0228] The camera in the imaging unit 3 has various built-in sensors, such as proximity sensors, gyroscope sensors, accelerometer sensors, or geomagnetic sensors. The associated information D13 contains the detection values ​​detected by these sensors.

[0229] The setting unit 31 sets the value of the identifier based on the image data D1 and the associated information D13.

[0230] The setting unit 31 can also set images captured during a period of time as "inappropriate" if a target near the camera is detected by the proximity sensor and this state continues for a fixed time or longer. The setting unit 31 can also set images captured during a period of time as "appropriate" if a target near the camera is detected by the proximity sensor and the target is moving at a high speed of a fixed speed or higher.

[0231] The setting unit 31 can also set the image captured during this period as "inappropriate" if the camera shake or tremor is detected by a gyroscope sensor, accelerometer sensor or geomagnetic sensor, and the level of the shake or tremor is above a threshold.

[0232] According to this variation, circuit 11 can appropriately determine whether an image is suitable for task processing based on the association information D13 before the image is encoded.

[0233] (Seventh variation)

[0234] In the above embodiments, task processing is a type, but it is not limited to this. Task processing may also include multiple types of task processing, and circuit 11 individually sets the image as "appropriate" or "inappropriate" for each task processing. The following example illustrates this: task processing includes a first task processing and a second task processing, where the first task processing is object detection and the second task processing is human vision. However, task processing may also include three or more types of task processing, and the types of task processing are not limited to object detection and human vision.

[0235] Figure 27This diagram illustrates a simplified structure of the image processing system according to the seventh variation. The image processing system includes a first task processing unit 4A and a second task processing unit 4B. The first task processing unit 4A performs a first task processing as object detection based on first image data D2A input from the decoding device 2. The second task processing unit 4B performs a second task processing as human vision based on second image data D2B input from the decoding device 2.

[0236] Figure 28 This is a diagram illustrating an example of the task processing configuration information 60. Configuration information 60 indicates that the first task processing is object detection, and the second task processing is human vision. Encoding device 1 and decoding device 2 share configuration information 60 in advance.

[0237] Figure 29 This diagram shows a simplified representation of the structure of the circuit 11 included in the encoding device 1. The circuit 11 includes a setting unit 31 and an encoding unit 32. The setting unit 31 inputs first identifier data D11A, which contains multiple first identifiers related to the processing of the first task, and second identifier data D11B, which contains multiple second identifiers related to the processing of the second task, to the encoding unit 32.

[0238] Figure 30 This is a flowchart showing the process performed by the circuit 11 of the encoding device 1.

[0239] First, in step SP11, circuit 11 acquires multiple images contained in image data D1 input from imaging unit 3. Image data D1 is then input to setting unit 31 and encoding unit 32.

[0240] Next, in step SP12A, the setting unit 31 performs image parsing on each image of the encoded object contained in the image data D1, and sets the value of the first identifier and the value of the second identifier based on the result of the image parsing. The setting unit 31 inputs the first identifier data D11A containing the set value of the first identifier and the second identifier data D11B containing the set value of the second identifier to the encoding unit 32.

[0241] Next, in step SP13A, the encoding unit 32 encodes the multiple images contained in the image data D1 input from the shooting unit 3, the multiple first identifiers contained in the first identifier data D11A input from the setting unit 31, and the multiple second identifiers contained in the second identifier data D11B input from the setting unit 31 into a bit stream BS.

[0242] Next, in step SP14, circuit 11 sends the bit stream BS to decoding device 2 via transmission path NW.

[0243] Figure 31This diagram illustrates image data D1, first identifier data D11A, and second identifier data D11B. Image data D1 contains multiple images in chronological order (in this example, five images P1 to P5). First identifier data D11A and second identifier data D11B contain a first identifier and a second identifier associated with each image P1 to P5. Each first identifier and second identifier is a 1-bit data value, set to either "1" or "0".

[0244] Figure 32 This diagram illustrates a bit string containing first identifier data D11A and second identifier data D11B. The first two bits of the bit string represent the values ​​of the first and second identifiers associated with image P1. The next two bits of the bit string represent the values ​​of the first and second identifiers associated with image P2. Similarly, thereafter, consecutive two bits of the bit string represent the values ​​of the first and second identifiers associated with images P3 through P5. Furthermore, when there are N types of task processing, consecutive N bits of the bit string represent the values ​​of N identifiers associated with the same image.

[0245] Figure 33A This is a diagram illustrating the first example of the syntax related to the setting of the first and second identifiers. The syntax includes the item "num_vision_task_minus1". The value of "num_vision_task_minus1" is set to the value obtained by subtracting "1" from the number of task processing types. The value of "frame_for_inference[i]" when i=1 is equivalent to the value of the first identifier, and the value of "frame_for_inference[i]" when i=2 is equivalent to the value of the second identifier. Setting the value of "frame_for_inference[i]" to "1" indicates that the associated image is "appropriate". On the other hand, setting the value of "frame_for_inference[i]" to "0" indicates that the associated image is "inappropriate".

[0246] In addition, it can also replace the use of repetitive statements such as for statements, and such as Figure 33B The following describes the various settings for "frame_for_inference[i]" according to the item entries.

[0247] Figure 34AThis is the second example of the syntax related to the setting of the first and second identifiers. When i = 1, the value of "frame_not_for_inference[i]" is equivalent to the value of the first identifier, and when i = 2, the value of "frame_not_for_inference[i]" is equivalent to the value of the second identifier. Setting the value of "frame_not_for_inference[i]" to "1" indicates that the associated image is "inappropriate." Conversely, setting the value of "frame_not_for_inference[i]" to "0" indicates that the associated image is "appropriate."

[0248] In addition, it can also replace the use of repetitive statements such as for statements, and such as Figure 34B The following is a detailed description of the various settings for "frame_not_for_inference[i]".

[0249] Figure 35 This diagram shows a simplified representation of the circuit 21 included in the decoding device 2. The circuit 21 includes a decoding unit 51 and a selection unit 52. The decoding unit 51 inputs image data D21, first identifier data D22A containing multiple first identifiers related to the first task processing, and second identifier data D22B containing multiple second identifiers related to the second task processing to the selection unit 52.

[0250] Figure 36 This is a flowchart showing the process performed by the circuit 21 of the decoding device 2.

[0251] First, in step SP21, circuit 21 receives bit stream BS from encoding device 1 via transmission path NW. Bit stream BS is then input to decoding unit 51.

[0252] Next, in step SP22A, circuit 21 decodes multiple images and the first identifier and second identifier associated with each image according to bitstream BS, thereby outputting image data D21 corresponding to image data D1, first identifier data D22A corresponding to first identifier data D11A, and second identifier data D22B corresponding to second identifier data D11B. Image data D21, first identifier data D22A, and second identifier data D22B are input to selection unit 52.

[0253] Next, in step SP23A, the selection unit 52 selects at least one first image used in the first task processing from among the plurality of images contained in the image data D21 based on the first identifier contained in the first identifier data D22A. Additionally, the selection unit 52 selects at least one second image used in the second task processing from among the plurality of images contained in the image data D21 based on the second identifier contained in the second identifier data D22B.

[0254] Next, in step SP24A, the selection unit 52 outputs first image data D2A containing at least one selected first image and second image data D2B containing at least one selected second image. The first image data D2A is input to the first task processing unit 4A, and the second image data D2B is input to the second task processing unit 4B. The first task processing unit 4A performs first task processing using at least one first image contained in the input first image data D2A. The second task processing unit 4B performs second task processing using at least one second image contained in the input second image data D2B.

[0255] Figure 37 This is a diagram illustrating an example of the selection of a first image and a second image based on the first identifier data D22A and the second identifier data D22B. Figure 37 The following example is shown: setting the value of "frame_for_inference[i]" to "1" indicates that the associated image is "appropriate", and setting the value of "frame_for_inference[i]" to "0" indicates that the associated image is "inappropriate".

[0256] Image data D21 contains multiple images in chronological order (five images P1 to P5 in this example). First identifier data D22A contains first identifiers associated with each image P1 to P5. For images P1 to P5, the values ​​of the first identifiers are set to "1", "0", "0", "1", and "1", respectively. Second identifier data D22B contains second identifiers associated with each image P1 to P5. For all images P1 to P5, the value of the second identifiers is set to "1".

[0257] The selection unit 52 selects images P1, P4, and P5 from the plurality of images P1 to P5 contained in the image data D21, where the value of the first identifier is "1", and does not select images P2 and P3 where the value of the first identifier is "0". As a result, the selection unit 52 outputs the first image data D2A containing the selected images P1, P4, and P5.

[0258] Furthermore, the selection unit 52 selects images P1 to P5 from the plurality of images P1 to P5 contained in the image data D21 whose second identifier value is "1". As a result, the selection unit 52 outputs second image data D2B containing all images P1 to P5.

[0259] According to this variation, a first image suitable for the machine task can be selected based on the first identifier, thereby enabling the machine task to be performed appropriately. Furthermore, a second image suitable for human vision can be selected based on the second identifier, thereby enabling the performance of human vision tasks appropriately.

[0260] (8th variation)

[0261] In the above embodiments, each identifier is a 1-bit data, but it is not limited to this. Each identifier can also be a multi-bit data.

[0262] Figure 38 This is a diagram illustrating an example of how the values ​​of identifiers are set in the syntax. In this example, each identifier is 2 bits of data.

[0263] Setting "frame_for_inference" to "0" indicates that neither "appropriate" nor "inappropriate" is specified on the encoding device 1 side; that is, the encoding device 1 side has not determined whether the associated image is suitable for task processing. Setting "frame_for_inference" to "1" indicates that the associated image is "inappropriate." Setting "frame_for_inference" to "2" indicates that the associated image is "appropriate." Furthermore, the value of "frame_for_inference" at "3" is allocated for future expansion and is currently undefined.

[0264] According to this variation, since the three states of "appropriate", "inappropriate" and "not specified" can be specified with each 2-bit identifier, it is not necessary to mix images associated with identifiers and images not associated with identifiers.

[0265] (9th variation)

[0266] In the 9th variation, a combination of the 3rd variation and the 7th variation described above will be explained.

[0267] Figure 39 This is a diagram illustrating an example of the selection of a first image and a second image based on the first identifier data D22A and the second identifier data D22B. Figure 39The following example is shown: setting the value of "frame_for_inference[i]" to "1" indicates that the associated image is "appropriate", setting the value of "frame_for_inference[i]" to "0" indicates that the associated image is "inappropriate", and an image that is not associated with the first identifier and the second identifier indicates that it has not been determined on the encoding device 1 side whether the image is suitable for task processing.

[0268] The selection unit 52 selects from the plurality of images P1 to P32 contained in the image data D21 images P2 and P32 whose first identifier value is "1", and image P1 which is not associated with the first identifier and the second identifier. The selection unit 52 does not select image P4 whose first identifier value is "0", and image P3 which is not associated with the first identifier and the second identifier. As a result, the selection unit 52 outputs first image data D2A containing the selected images P1, P2, and P32.

[0269] Selection unit 52 selects from the plurality of images P1 to P32 contained in image data D21 images P4 and P32 whose second identifier value is "1", and image P1 which is not associated with the first identifier and the second identifier. Selection unit 52 does not select image P2 whose second identifier value is "0", and image P3 which is not associated with the first identifier and the second identifier. As a result, selection unit 52 outputs second image data D2B containing the selected images P1, P4, and P32.

[0270] (Example 10)

[0271] In the seventh variation described above, the first task processing performed by the first task processing unit 4A and the second task processing performed by the second task processing unit 4B are preset by the setting information 60, but the circuit 11 may also encode the correspondence information representing the first identifier and the second identifier with the first task processing and the second task processing into a bit stream BS.

[0272] Figure 40This diagram illustrates an example of the syntax related to task processing settings. The syntax includes the item "num_identifier_minus1". The value of "num_identifier_minus1" is set to the value obtained by subtracting "1" from the number of identifiers associated with the image. Additionally, the syntax includes the item "vision_task[i]". In the "vision_task[i]" item, a string representing the type of task processing is described. Alternatively, an integer value representing the type of task processing may be used instead of a string. In this case, a correspondence between the integer value and the type of task processing is established, and this setting information is shared in advance by the encoding device 1 and the decoding device 2.

[0273] Figure 41 This diagram illustrates a simplified structure of a bitstream BS. The bitstream BS has a header area 71 and a payload area 72. The header area 71 has an SEI area 73 for storing additional information. The encoding unit 32 stores encoded data representing the correspondence between multiple identifiers and multiple task processing in the SEI area 73. However, the SEI area 73 can be replaced by the VPS, SPS, PPS, PH, SH, APS, or block header, etc. Furthermore, the SEI area 73 can be a prefix_SEI located before the payload area 72, or a suffix_SEI located after the payload area 72.

[0274] According to this variation, multiple task processing can be arbitrarily set in the corresponding information, and the execution accuracy of each of the multiple task processing can be improved.

[0275] (Example 11)

[0276] In the 11th variation, a combination of the above-mentioned 1st variation and the above-mentioned 7th variation will be described.

[0277] Figure 42 This is a diagram illustrating an example of the syntax related to the settings of the first and second identifiers. Setting the value of "frame_for_inference[i][j]" to "1" indicates that for the j-th task processing, the i-th image in the image group is "appropriate". Conversely, setting the value of "frame_for_inference[i][j]" to "0" indicates that for the j-th task processing, the i-th image in the image group is "inappropriate".

[0278] (12th variation)

[0279] In the 12th variation, a combination of the above-mentioned 2nd variation and the above-mentioned 7th variation will be described.

[0280] Figure 43 This is a diagram illustrating an example of the syntax related to the setting of the first and second identifiers. Setting the value of "frame_for_inference[i][j]" to "1" indicates that for the j-th task processing, the i-th image determined by the determination information within the image group is "appropriate". Conversely, setting the value of "frame_for_inference[i][j]" to "0" indicates that for the j-th task processing, the i-th image determined by the determination information within the image group is "inappropriate".

[0281] (Example 13)

[0282] In the 13th variation, a combination of the above-mentioned 3rd variation and the above-mentioned 7th variation will be described.

[0283] Figure 44 This is a diagram illustrating an example of the syntax related to the setting of the first and second identifiers. For images associated with identifiers, the value of "frame_for_inference_present_flag[i]" is set to "1". Conversely, for images not associated with identifiers, the value of "frame_for_inference_present_flag[i]" is set to "0".

[0284] For images associated with identifiers, setting "frame_for_inference[i][j]" to "1" indicates that for the j-th task, the i-th image in the image group is "appropriate". Setting "frame_for_inference[i][j]" to "0" indicates that for the j-th task, the i-th image in the image group is "inappropriate".

[0285] (Example 14)

[0286] In the 14th variation, a combination of the above-mentioned 2nd variation, 3rd variation and 7th variation will be described.

[0287] Figure 45This is a diagram illustrating an example of the syntax related to the setting of the first and second identifiers. For an image associated with an identifier, setting "frame_for_inference[i][j]" to "1" indicates that for the j-th task processing, the i-th image determined by the determination information within the image group is "appropriate". Setting "frame_for_inference[i][j]" to "0" indicates that for the j-th task processing, the i-th image determined by the determination information within the image group is "inappropriate".

[0288] Industrial availability

[0289] This disclosure is particularly useful for applications of image processing systems that include an encoding device for encoding an image into a bitstream and transmitting the bitstream, and a decoding device for decoding the image based on the received bitstream.

Claims

1. A decoding apparatus comprising: a circuit; and a memory connected to the circuit, the circuit decoding, from a bitstream, an image and an identifier associated with the image, the identifier indicating at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

2. The decoding apparatus according to claim 1, wherein the image includes a plurality of images, and the circuit selects at least one image used in the task processing from among the plurality of images based on the identifier.

3. The decoding apparatus according to claim 1, wherein the task processing includes a machine task.

4. The decoding apparatus according to claim 1, wherein the task processing includes human vision.

5. The decoding apparatus according to claim 1, wherein the identifier includes a plurality of identifiers, the task processing includes a plurality of task processings, and the plurality of identifiers correspond to different task processings among the plurality of task processings.

6. The decoding apparatus according to claim 5, wherein the plurality of identifiers include a first identifier and a second identifier, the task processing corresponding to the first identifier includes a machine task, and the task processing corresponding to the second identifier includes human vision.

7. The decoding apparatus according to claim 1, wherein the identifier includes a first value and a second value, the first value indicates that the image is suitable for the task processing, and the second value indicates that the image is not suitable for the task processing.

8. The decoding apparatus according to claim 1, wherein the identifier includes a first value and a second value, the first value indicates that the image is suitable for the task processing or that the image is not suitable for the task processing, and the second value indicates that it is not determined whether the image is suitable for the task processing.

9. The decoding apparatus according to claim 1, wherein the image includes a plurality of images, the plurality of images include an image not associated with the identifier, and the image not associated with the identifier indicates an image for which it is not determined whether the image is suitable for the task processing.

10. The decoding apparatus according to claim 1, wherein the identifier includes one or more identifiers, the task processing includes one or more task processings, and the circuit further decodes, from the bitstream, correspondence information indicating correspondence between the one or more identifiers and the one or more task processings.

11. The decoding apparatus according to claim 10, wherein the correspondence information is stored in a header area of the bitstream.

12. The decoding apparatus according to claim 1, wherein the identifier is stored in a header area of the image associated with the identifier.

13. The decoding apparatus according to claim 1, wherein the image includes a plurality of images, and the identifier includes one or more identifiers. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The one or more identifiers are stored in a header area of one of the plurality of images, and the one image and one or more images different from the one image are associated with the one or more identifiers.

14. The decoding apparatus according to any one of claims 11 to 13, wherein The header area is an SEI area.

15. The decoding apparatus according to claim 1, wherein The image includes a plurality of images, The circuit further decodes, from the bitstream, determination information for determining the image of the plurality of images that is associated with the identifier.

16. An encoding apparatus comprising: a circuit; and a memory connected to the circuit, The circuit encodes an image and an identifier associated with the image into a bitstream, The identifier indicates at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

17. The encoding apparatus according to claim 16, wherein The circuit sets a value of the identifier based on an encoding condition at the time of encoding the image.

18. The encoding apparatus according to claim 16, wherein The circuit sets a value of the identifier based on a quality of the encoded image.

19. The encoding apparatus according to claim 16, wherein The circuit performs a process of: acquiring, from an external apparatus, an image to be encoded and association information associated with the image, setting a value of the identifier based on the association information.

20. The encoding apparatus according to claim 16, wherein The circuit performs a process of: performing image analysis on an image to be encoded, setting a value of the identifier based on a result of the image analysis.

21. The encoding apparatus according to claim 16, wherein The task processing includes a machine task.

22. The encoding apparatus according to claim 16, wherein The task processing includes human vision.

23. The encoding apparatus according to claim 16, wherein The identifier includes a plurality of identifiers, The task processing includes a plurality of task processing, The plurality of identifiers correspond to different task processing of the plurality of task processing.

24. The encoding apparatus according to claim 23, wherein The plurality of identifiers include a first identifier and a second identifier, The task processing corresponding to the first identifier includes a machine task, The task processing corresponding to the second identifier includes human vision.

25. The encoding apparatus according to claim 16, wherein The identifier includes a first value and a second value, The first value indicates that the image is suitable for the task processing, The second value indicates that the image is not suitable for the task processing.

26. The encoding apparatus according to claim 16, wherein The identifier includes a first value and a second value, The first value indicates that the image is suitable for the task processing or that the image is not suitable for the task processing, The second value indicates that it is not determined whether the image is suitable for the task processing.

27. The encoding apparatus according to claim 16, wherein The image includes a plurality of images, The plurality of images includes an image not associated with the identifier, The image not associated with the identifier indicates an image for which it is not determined whether it is suitable for the task processing.

28. The encoding apparatus according to claim 16, wherein The identifier includes a plurality of identifiers, The task processing includes a plurality of task processings, The circuitry further encodes, into the bitstream, correspondence information indicating a correspondence between the plurality of identifiers and the plurality of task processings.

29. The encoding apparatus according to claim 28, wherein The correspondence information is stored in a header area of the bitstream.

30. The encoding apparatus according to claim 16, wherein The identifier is stored in a header area of the image with which the identifier is associated.

31. The encoding apparatus according to claim 16, wherein The image includes a plurality of images, The identifier includes a plurality of identifiers, The plurality of identifiers is stored in a header area of an image included in the plurality of images, the image being associated with the image and with a plurality of images other than the image.

32. The encoding apparatus according to any one of claims 29 to 31, wherein The header area is an SEI area.

33. The encoding apparatus according to claim 16, wherein The image includes a plurality of images, The circuitry further encodes, into the bitstream, determination information for determining the image associated with the identifier among the plurality of images.

34. A decoding method, A decoding apparatus decodes, from a bitstream, an image and an identifier associated with the image, The identifier indicates at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

35. An encoding method, An encoding apparatus encodes, into a bitstream, an image and an identifier associated with the image, The identifier indicates at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

Images