Decoder, encoder, decoding method, and encoding method

MYPI2025006243A0Undetermined Publication Date: 2024-10-12PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
MYPI2025006243
Authority / Receiving Office
MY · MY
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-12
Filing Date
2024-04-02
Publication Date
2024-10-12

AI Technical Summary

Technical Problem

Existing image processing systems face decreased execution accuracy in task processing due to the inclusion of unsuitable images in thinned-out frames, which affects human vision and machine tasks, particularly in object detection and tracking.

Method used

Incorporating identifiers in the bitstream to indicate whether images are suitable for specific task processing, allowing the decoding device to selectively choose images based on these identifiers, thereby improving task execution accuracy.

Benefits of technology

This approach enhances the accuracy of task processing by ensuring that only suitable images are used for human vision and machine tasks, such as object detection and tracking, by appropriately selecting images based on their suitability indicators.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PI2025006243_DRAWING_001
    Figure PI2025006243_DRAWING_001
  • Figure PI2025006243_DRAWING_002
    Figure PI2025006243_DRAWING_002
  • Figure PI2025006243_DRAWING_003
    Figure PI2025006243_DRAWING_003
Patent Text Reader

Abstract

A decoder includes: circuitry; and a memory connected to the circuitry, in which the circuitry, in operation, decodes, from a bitstream, an image and an identifier associated with the image, the identifier indicating at least one of (i) the image being suitable for task processing corresponding to the identifier and (ii) the image being not suitable for the task processing. Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method.

[0002] Patent Literature 1 discloses a method for determining computing power resources according to the background art. A control node includes at least one of a terminal-side computing power control node and a network-side computing power control node. The control node receives first information indicating resource information of computing power available to the terminal-side computing power control node. The control node determines context information of the computing power resources based on the first information.

[0003] International Publication No. 2023 / 050431

[0004] The present disclosure aims to improve the execution accuracy of task processing.

[0005] A decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit decodes an image and an identifier associated with the image from a bitstream, and the identifier indicates at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

[0006] 9 is a diagram showing, in a simplified form, the configuration of an image processing system according to an embodiment of the present disclosure. FIG. 10 is a diagram showing, in a simplified form, the configuration of a circuit included in an encoding device. FIG. 11 is a flowchart showing processing performed by a circuit included in an encoding device. FIG. 12 is a diagram showing a first example of image analysis by a setting unit. FIG. 13 is a diagram showing a second example of image analysis by the setting unit. FIG. 14 is a diagram showing a third example of image analysis by the setting unit. FIG. 15 is a diagram showing image data and identifier data. FIG. 16 is a diagram showing a first example of syntax related to identifier setting. FIG. 17 is a diagram showing a first setting example of identifier values ​​in the syntax. FIG. 18 is a diagram showing a second example of syntax related to identifier setting. FIG. 19 is a diagram showing a second setting example of identifier values ​​in the syntax. FIG. 19 is a diagram showing, in a simplified form, the configuration of packet data in which each encoded image is stored. FIG. 10 is a diagram showing, in a simplified form, the configuration of a circuit included in a decoding device. FIG. 11 is a flowchart showing processing performed by a circuit included in a decoding device. FIG. 12 is a diagram showing a first example of image selection based on identifier data. FIG. 13 is a diagram showing a second example of image selection based on identifier data. FIG. 14 is a diagram showing image data and identifier data. FIG. 15 is a diagram showing an example of syntax related to identifier setting. FIG. 16 is a diagram showing an example of syntax related to identifier setting. FIG. 17 is a diagram showing a first setting example of identifier values ​​in the syntax, corresponding to FIG. 9. 19 is a diagram showing an example of syntax for setting an identifier, based on FIG. 18 . FIG. 20 is a diagram showing an example of syntax for setting an identifier, based on FIG. 19 . FIG. 21 is a diagram showing an example of syntax for setting a task processing. FIG. 22 is a diagram showing a simplified configuration of a circuit provided in an encoding device. FIG. 23 is a diagram showing a simplified configuration of a circuit provided in an encoding device. FIG. 24 is a diagram showing a simplified configuration of an image processing system according to a seventh modified example. FIG. 25 is a diagram showing an example of setting information for task processing. FIG. 26 is a diagram showing a simplified configuration of a circuit provided in an encoding device. FIG. 27 is a flowchart showing processing executed by a circuit provided in an encoding device. FIG. 28 is a diagram showing image data, first identifier data, and second identifier data. FIG. 29 is a diagram showing a bit string including the first identifier data and the second identifier data. FIG. 30 is a diagram showing a first example of syntax for setting the first identifier and the second identifier.1 is a diagram showing a first example of syntax related to setting a first identifier and a second identifier. FIG. 2 is a diagram showing a second example of syntax related to setting a first identifier and a second identifier. FIG. 3 is a diagram showing a simplified configuration of a circuit included in a decoding device. FIG. 4 is a flowchart showing processing executed by a circuit included in a decoding device. FIG. 5 is a diagram showing an example of selection of a first image and a second image based on first identifier data and second identifier data. FIG. 6 is a diagram showing an example of setting identifier values ​​in syntax. FIG. 7 is a diagram showing an example of selection of a first image and a second image based on first identifier data and second identifier data. FIG. 8 is a diagram showing an example of syntax related to setting task processing. FIG. 9 is a diagram showing a simplified configuration of a bitstream. FIG. 10 is a diagram showing an example of syntax related to setting a first identifier and a second identifier. FIG. 11 is a diagram showing an example of syntax related to setting a first identifier and a second identifier. FIG. 12 is a diagram showing an example of syntax related to setting a first identifier and a second identifier.

[0007] (Knowledge forming the basis of the present disclosure) An image processing system according to the background art includes an encoding device, a decoding device, and a task processing unit. The encoding device acquires image data of a moving image captured by a camera, and encodes images included in the acquired image data into a bitstream. The encoding device transmits the bitstream to the decoding device. The decoding device receives the bitstream, decodes images from the received bitstream, and inputs the decoded images to the task processing unit. The task processing unit executes task processing based on the images input from the decoding device.

[0008] Task processing includes human vision and machine tasks. Human vision is the viewing or viewing of video images by a human, such as an operator or user. Machine tasks include various types of task processing, such as object detection, object tracking, object segmentation, action recognition, or pose estimation, using machine-learned estimation models.

[0009] The processing speed varies depending on the type of task processing, depending on factors such as the processing capacity of the hardware resources that execute each task processing. Therefore, the decoding device performs thinning processing on multiple images decoded from a bitstream and inputs the thinned images to the task processing unit. For example, the decoding device thins out even-numbered frames of a decoded 60 fps image to input a 30 fps image to the task processing unit.

[0010] However, if the images of the thinned out even frames include images that are important for task processing, the estimation accuracy of the estimation model will decrease, and as a result, the execution accuracy of the task processing will decrease.

[0011] In order to solve this problem, the inventor discovered that the above problem can be solved by including information indicating whether each image is suitable for task processing in a bitstream and transmitting it from the encoding device to the decoding device, and the decoding device then performing thinning processing according to the task processing based on that information, and thus came up with the present disclosure.

[0012] Next, each aspect of the present disclosure will be described.

[0013] A decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit decodes an image and an identifier associated with the image from a bitstream, and the identifier indicates at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

[0014] According to the first aspect, the identifier indicates at least one of whether the image is suitable for the task processing corresponding to the identifier and whether the image is unsuitable for the task processing. Therefore, at least one image to be used for the task processing can be appropriately selected from multiple images decoded from the bitstream based on the identifier, thereby improving the accuracy of the execution of the task processing.

[0015] In the decoding device according to the second aspect of the present disclosure, in the first aspect, the image may include a plurality of images, and the circuit may further select at least one image from the plurality of images to be used for the task processing based on the identifier.

[0016] According to the second aspect, an image suitable for task processing can be selected based on the identifier, thereby improving the accuracy of execution of task processing.

[0017] In the decoding device according to a third aspect of the present disclosure, in the first or second aspect, the task processing may include a machine task.

[0018] According to the third aspect, an image suitable for a machine task can be selected based on the identifier, thereby enabling the machine task to be executed appropriately.

[0019] In a decoding device according to a fourth aspect of the present disclosure, in any one of the first to third aspects, the task processing may include human vision.

[0020] According to the fourth aspect, an image suitable for human vision can be selected based on the identifier, thereby enabling human vision to be performed appropriately.

[0021] A decoding device according to a fifth aspect of the present disclosure is, in any one of the first to fourth aspects, preferably such that the identifier includes multiple identifiers, the task processing includes multiple task processing, and the multiple identifiers correspond to different task processing among the multiple task processing.

[0022] According to the fifth aspect, a plurality of task processes can be handled using a plurality of identifiers.

[0023] A decoding device according to a sixth aspect of the present disclosure may be configured such that, in the fifth aspect, the plurality of identifiers include a first identifier and a second identifier, the task processing corresponding to the first identifier includes a machine task, and the task processing corresponding to the second identifier includes human vision.

[0024] According to the sixth aspect, an image suitable for a machine task can be selected based on the first identifier, thereby enabling the machine task to be performed appropriately. Also, an image suitable for human vision can be selected based on the second identifier, thereby enabling the human vision to be performed appropriately. In other words, it is possible to perform the machine task or human vision appropriately based on the first identifier and the second identifier depending on the properties of the image.

[0025] A decoding device according to a seventh aspect of the present disclosure may be such that, in any one of the first to sixth aspects, the identifier includes a first value and a second value, the first value indicating that the image is suitable for the task processing, and the second value indicating that the image is not suitable for the task processing.

[0026] According to the seventh aspect, the circuit can easily determine which images to select and which images not to select based on the first value and the second value.

[0027] In a decoding device according to an eighth aspect of the present disclosure, in any one of the first to sixth aspects, the identifier may include a first value and a second value, the first value indicating that the image is suitable for the task processing or that the image is not suitable for the task processing, and the second value indicating that it does not specify whether the image is suitable for the task processing.

[0028] According to the eighth aspect, the circuit can easily determine which images to select or not to select based on the first value, and the circuit can arbitrarily determine whether to select an image associated with the identifier of the second value depending on its own processing power, etc.

[0029] In a decoding device according to a ninth aspect of the present disclosure, in any one of the first to eighth aspects, the image may include a plurality of images, the plurality of images may include an image that is not associated with the identifier, and the image that is not associated with the identifier may indicate an image that does not specify whether it is suitable for the task processing.

[0030] According to the ninth aspect, whether or not to select an image that is not associated with an identifier can be arbitrarily determined according to one's own processing capabilities and the like.

[0031] A decoding device according to a tenth aspect of the present disclosure is, in any one of the first to ninth aspects, such that the identifier includes one or more identifiers, the task processing includes one or more task processes, and the circuit further decodes correspondence information indicating the correspondence between the one or more identifiers and the one or more task processes from the bitstream.

[0032] According to the tenth aspect, one or more task processes can be arbitrarily set in the correspondence information, and the execution accuracy of each of the one or more set task processes can be improved.

[0033] A decoding device according to an eleventh aspect of the present disclosure is the tenth aspect, wherein the correspondence information is preferably stored in a header area of ​​the bitstream.

[0034] According to the eleventh aspect, the circuit can easily obtain the correspondence information from the header area of ​​the bitstream.

[0035] In the decoding device according to a twelfth aspect of the present disclosure, in any one of the first to eleventh aspects, the identifier may be stored in a header area of ​​the image associated with the identifier.

[0036] According to the twelfth aspect, the circuitry can easily obtain an identifier associated with each image from a header area of ​​each image.

[0037] A decoding device according to a thirteenth aspect of the present disclosure is, in any one of the first to eleventh aspects, such that the image includes a plurality of images, the identifier includes one or more identifiers, the one or more identifiers are stored in a header area of ​​one of the images included in the plurality of images, and are associated with the one image and one or more images different from the one image.

[0038] According to the thirteenth aspect, the circuitry can facilitate obtaining one or more identifiers associated with one or more images from a header region of an image.

[0039] A decoding device according to a fourteenth aspect of the present disclosure is any one of the eleventh to thirteenth aspects, wherein the header region is an SEI region.

[0040] According to the fourteenth aspect, the circuit can easily obtain the identifier from the SEI region.

[0041] In a decoding device according to a fifteenth aspect of the present disclosure, in any one of the first to fourteenth aspects, the image includes a plurality of images, and the circuit further decodes, from the bitstream, specific information for identifying the image among the plurality of images that is associated with the identifier.

[0042] According to the fifteenth aspect, by specifying the images associated with the identifiers by the specific information, it is not necessary to match the number of identifiers with the number of images, so the data amount of the identifiers can be reduced, thereby enabling a reduction in the amount of code and an improvement in coding efficiency.

[0043] A coding device according to a sixteenth aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit encodes an image and an identifier associated with the image into a bitstream, the identifier indicating at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

[0044] According to a sixteenth aspect, the identifier indicates at least one of whether the image is suitable for the task processing corresponding to the identifier and whether the image is unsuitable for the task processing. Thus, a decoding device that receives a bitstream can appropriately select, based on the identifier, at least one image to be used for the task processing from among a plurality of images decoded from the bitstream, thereby improving the accuracy of execution of the task processing.

[0045] A coding device according to a seventeenth aspect of the present disclosure is the sixteenth aspect, wherein the circuit sets the value of the identifier based on coding conditions when the image is coded.

[0046] According to the seventeenth aspect, after encoding an image, the circuit can appropriately determine whether or not the image is suitable for task processing based on the encoding conditions.

[0047] In the encoding device according to an eighteenth aspect of the present disclosure, in the sixteenth or seventeenth aspect, the circuit may set the value of the identifier based on the image quality of the image after encoding.

[0048] According to the eighteenth aspect, the circuit can appropriately determine whether or not an image is suitable for task processing based on image quality after encoding the image.

[0049] In the encoding device according to the 19th aspect of the present disclosure, in any one of the 16th to 18th aspects, the circuit may obtain the image to be encoded and related information relating to the image from an external device, and set the value of the identifier based on the related information.

[0050] According to the nineteenth aspect, the circuit can appropriately determine whether or not an image is suitable for task processing based on related information before encoding the image.

[0051] In the encoding device according to the 20th aspect of the present disclosure, in any one of the 16th to 19th aspects, the circuit may perform image analysis on the image to be encoded and set the value of the identifier based on the results of the image analysis.

[0052] According to the twentieth aspect, the circuit can appropriately determine whether or not an image is suitable for task processing based on the result of image analysis before encoding the image.

[0053] In the encoding device according to a 21st aspect of the present disclosure, in any one of the 16th to 20th aspects, the task processing may include a machine task.

[0054] According to the twenty-first aspect, a decoding device that receives a bitstream can select an image suitable for a machine task based on the identifier, thereby enabling the machine task to be executed appropriately.

[0055] In the encoding device according to a 22nd aspect of the present disclosure, in any one of the 16th to 21st aspects, the task processing may include human vision.

[0056] According to the twenty-second aspect, a decoding device that receives a bitstream can select an image suitable for human vision based on the identifier, thereby enabling appropriate execution of human vision.

[0057] In the encoding device according to the 23rd aspect of the present disclosure, in any one of the 16th to 22nd aspects, the identifier may include multiple identifiers, the task processing may include multiple task processing, and the multiple identifiers may correspond to different task processing among the multiple task processing.

[0058] According to the twenty-third aspect, a plurality of task processes can be handled using a plurality of identifiers.

[0059] In the encoding device of the 24th aspect of the present disclosure, in the 23rd aspect, the plurality of identifiers may include a first identifier and a second identifier, the task processing corresponding to the first identifier may include a machine task, and the task processing corresponding to the second identifier may include human vision.

[0060] According to the twenty-fourth aspect, a decoding device that receives a bitstream can select an image suitable for a machine task based on the first identifier, thereby enabling the machine task to be performed appropriately. The decoding device can also select an image suitable for human vision based on the second identifier, thereby enabling the human vision to be performed appropriately. In other words, it is possible to perform the machine task or human vision appropriately based on the first identifier and the second identifier depending on the properties of the image.

[0061] In the encoding device of the 25th aspect of the present disclosure, in any one of the 16th to 24th aspects, the identifier may include a first value and a second value, the first value indicating that the image is suitable for the task processing, and the second value indicating that the image is not suitable for the task processing.

[0062] According to the twenty-fifth aspect, the decoding device that receives the bitstream can easily determine which images to select and which images not to select, based on the first value and the second value.

[0063] In the encoding device of the 26th aspect of the present disclosure, in any one of the 16th to 24th aspects, the identifier may include a first value and a second value, the first value indicating that the image is suitable for the task processing or that the image is not suitable for the task processing, and the second value indicating that it does not specify whether the image is suitable for the task processing.

[0064] According to the 26th aspect, a decoding device that receives a bitstream can easily determine which images to select or not to select based on the first value. Furthermore, the decoding device can arbitrarily determine whether to select an image associated with an identifier of the second value in accordance with its own processing power, etc.

[0065] In the encoding device according to the 27th aspect of the present disclosure, in any one of the 16th to 26th aspects, the image may include a plurality of images, the plurality of images may include an image that is not associated with the identifier, and the image that is not associated with the identifier may indicate an image that does not specify whether it is suitable for the task processing.

[0066] According to the twenty-seventh aspect, the decoding device that has received the bitstream can arbitrarily determine whether or not to select an image that is not associated with an identifier, depending on its own processing capabilities and the like.

[0067] In the encoding device according to the 28th aspect of the present disclosure, in any one of the 16th to 27th aspects, the identifier includes one or more identifiers, the task processing includes one or more task processing, and the circuit further encodes correspondence information indicating the correspondence between the one or more identifiers and the one or more task processing into the bitstream.

[0068] According to the twenty-eighth aspect, one or more task processes can be arbitrarily set in the correspondence information, and the execution accuracy of each of the one or more task processes that have been set can be improved.

[0069] In the encoding device according to a 29th aspect of the present disclosure, in any one of the 16th to 28th aspects, the correspondence information may be stored in a header area of ​​the bitstream.

[0070] According to the twenty-ninth aspect, the decoding device that receives the bitstream can easily obtain the correspondence information from the header area of ​​the bitstream.

[0071] In the encoding device according to a 30th aspect of the present disclosure, in any one of the 16th to 29th aspects, the identifier may be stored in a header area of ​​the image associated with the identifier.

[0072] According to the thirtieth aspect, the decoding device that receives the bitstream can easily obtain the identifier associated with each image from the header area of ​​each image.

[0073] In the encoding device of the 31st aspect of the present disclosure, in any one of the 16th to 30th aspects, the image includes a plurality of images, the identifier includes one or more identifiers, and the one or more identifiers are stored in a header area of ​​one of the images included in the plurality of images and are associated with the one image and one or more images different from the one image.

[0074] According to the thirty-first aspect, a decoding device that receives a bitstream can easily obtain one or more identifiers associated with one or more images from the header area of ​​one image.

[0075] A decoding device according to a 32nd aspect of the present disclosure is any one of the 29th to 31st aspects, wherein the header region is an SEI region.

[0076] According to the thirty-second aspect, the circuit can easily obtain the identifier from the SEI region.

[0077] In the encoding device according to the 33rd aspect of the present disclosure, in any one of the 16th to 32nd aspects, the image includes a plurality of images, and the circuit may further encode, into the bitstream, specific information for identifying the image among the plurality of images that is associated with the identifier.

[0078] According to the thirty-third aspect, by specifying the images associated with the identifiers by the specific information, it is not necessary to match the number of identifiers with the number of images, and therefore the data amount of the identifiers can be reduced, thereby enabling a reduction in the amount of code and an improvement in coding efficiency.

[0079] A decoding method according to a thirty-fourth aspect of the present disclosure includes a decoding device that decodes an image and an identifier associated with the image from a bitstream, the identifier indicating at least one of whether the image is suitable for a task processing corresponding to the identifier and whether the image is not suitable for the task processing.

[0080] According to the thirty-fourth aspect, the identifier indicates at least one of whether the image is suitable for the task processing corresponding to the identifier and whether the image is unsuitable for the task processing. Therefore, at least one image to be used for the task processing can be appropriately selected from a plurality of images decoded from the bitstream based on the identifier, thereby improving the accuracy of the execution of the task processing.

[0081] An encoding method according to a 35th aspect of the present disclosure includes an encoding device encoding an image and an identifier associated with the image into a bitstream, the identifier indicating at least one of whether the image is suitable for a task processing corresponding to the identifier and whether the image is not suitable for the task processing.

[0082] According to the thirty-fifth aspect, the identifier indicates at least one of whether the image is suitable for the task processing corresponding to the identifier and whether the image is unsuitable for the task processing. Thus, a decoding device that receives a bitstream can appropriately select, based on the identifier, at least one image to be used for the task processing from among multiple images decoded from the bitstream, thereby improving the accuracy of execution of the task processing.

[0083] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0084] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0085] 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1, a decoding device 2, an image capturing unit 3, a task processing unit 4, and a transmission path NW.

[0086] The image capturing unit 3 is an example of an external device and includes a camera that captures moving images. The image capturing unit 3 inputs image data D1 of the captured moving images to the encoding device 1.

[0087] The encoding device 1 transmits a bit stream BS to the decoding device 2 via a transmission path NW. The decoding device 2 receives the bit stream BS.

[0088] The task processing unit 4 executes task processing based on the image data D2 input from the decoding device 2. The task processing includes human vision and machine tasks. Human vision is the viewing or listening of moving images by a human being such as an operator or a user. Machine tasks include various types of task processing such as object detection, object tracking, object segmentation, action recognition, or pose estimation using a machine-learned estimation model. The task processing unit 4 that executes human vision includes a display device such as a liquid crystal display or an organic EL display. The task processing unit 4 that executes machine tasks includes an inference unit that uses AI.

[0089] The transmission path NW is the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. The transmission path NW is preferably a private network or the like in which secure communication is ensured by access restrictions.

[0090] The encoding device 1 includes a circuit 11 and a memory 12 connected to the circuit 11. The circuit 11 includes a processor such as a CPU. The memory 12 includes any recording medium such as a ROM, RAM, HDD, SSD, or semiconductor memory. The memory 12 stores data to be processed by the circuit 11 or data in the middle of processing.

[0091] The decoding device 2 includes a circuit 21 and a memory 22 connected to the circuit 21. The circuit 21 includes a processor such as a CPU. The memory 22 includes any recording medium such as a ROM, a RAM, a HDD, an SSD, or a semiconductor memory. The memory 22 stores data to be processed by the circuit 21 or data in the middle of processing.

[0092] 2 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 includes a setting unit 31 and an encoding unit 32.

[0093] FIG. 3 is a flowchart showing the process executed by the circuit 11 included in the encoding device 1.

[0094] First, in step SP11, the circuit 11 acquires a plurality of images contained in the image data D1 input from the photographing unit 3. The image data D1 is input to the setting unit 31 and the encoding unit 32.

[0095] Next, in step SP12, the setting unit 31 performs image analysis on each image to be coded that is included in the image data D1, and sets an identifier value based on the results of the image analysis. The setting unit 31 inputs identifier data D11, including the identifier whose value has been set, to the coding unit 32.

[0096] The identifier indicates at least one of whether the image is suitable (i.e., appropriate) for the task processing corresponding to the identifier and whether the image is not suitable (i.e., inappropriate) for the task processing corresponding to the identifier. An image being suitable for the task processing includes recommending the use of the image when executing the task processing on the decoding device 2 side, or the image has undergone image processing suitable for the task processing. An image being unsuitable for the task processing includes not recommending the use of the image when executing the task processing on the decoding device 2 side (i.e., not recommended), or the image has not undergone image processing suitable for the task processing.

[0097] Next, in step SP13, the encoding unit 32 encodes the multiple images contained in the image data D1 input from the shooting unit 3 and the multiple identifiers contained in the identifier data D11 input from the setting unit 31 into a bit stream BS.

[0098] Next, in step SP14, the circuit 11 transmits the bit stream BS to the decoding device 2 via the transmission path NW.

[0099] 4 is a diagram showing a first example of image analysis by the setting unit 31. In FIG. 0 , T 1 , T 2 Three images are shown in chronological order corresponding to time T 0 The image corresponding to time T contains trees and pedestrians. 1 The image corresponding to the tree and the time T 0 This includes pedestrians who have moved slightly from time T 2 The image corresponding to the tree and the time T 1 The images include a pedestrian who has moved slightly from the center and a runner. When the task processing corresponding to the identifier is human vision, the setting unit 31 sets all images as "appropriate" in order to perform smooth video playback.

[0100] 5 is a diagram illustrating a second example of image analysis by the setting unit 31. When the task processing corresponding to the identifier is object detection, the time T 0Since a pedestrian has already been detected at time T 0 , T 2 The image corresponding to time T 1 Images corresponding to this are deemed "inappropriate."

[0101] 6 is a diagram showing a third example of image analysis by the setting unit 31. When the task processing corresponding to the identifier is object tracking, the setting unit 31 sets all images as "appropriate" because it is necessary to track movement even if the amount of movement is small.

[0102] The setting unit 31 may also set the value of the identifier based on the image quality of the image before encoding. The setting unit 31 evaluates the image quality of the image before encoding using a sharpness value, a contrast value, a noise level, a PSNR (Peak Signal to Noise Ratio) value, or the like. The setting unit 31 may also evaluate the image quality using a machine learning algorithm. The setting unit 31 may also evaluate the image quality based on whether the image is overexposed, underexposed, or properly exposed by analyzing the histogram of the image.

[0103] When the task processing is human vision, low-quality images reduce visibility. When the task processing is object detection, low-quality images reduce estimation accuracy. Therefore, when the task processing is human vision or object detection, the setting unit 31 may determine that an image whose image quality before encoding is equal to or greater than a threshold is "appropriate," and that an image whose image quality before encoding is less than the threshold is "inappropriate."

[0104] When the task processing is object tracking, tolerance for changes in image quality is high, so the trajectory of an object can be detected even in images with low image quality. Therefore, when the task processing is object tracking, the setting unit 31 may set all images as "appropriate."

[0105] In this way, the setting unit 31 determines whether an image is suitable (appropriate) or unsuitable (inappropriate) for the task processing, depending on the type of task processing corresponding to the identifier. In this embodiment, the task processing to be executed by the task processing unit 4 is set to one specific type, and the encoding device 1 and the decoding device 2 share this setting information in advance.

[0106] 7 is a diagram showing image data D1 and identifier data D11. The image data D1 includes multiple images (five images P1 to P5 in this example) in chronological order. The identifier data D11 includes identifiers associated with each of the images P1 to P5. The identifier is 1-bit data, and the value of the identifier is set to "1" or "0."

[0107] 8 is a diagram showing a first example of syntax for setting an identifier. The syntax includes an item "frame_for_inference."

[0108] 9A and 9B are diagrams illustrating a first example of setting identifier values ​​in syntax. As shown in (A), when the value of "frame_for_inference" is set to "1," it indicates that the associated image is "appropriate." When the value of "frame_for_inference" is set to "0," it indicates that the associated image is "inappropriate."

[0109] As shown in (B), if the value of "frame_for_inference" is set to "0", it may indicate that the encoding device 1 does not specify "appropriate" or "inappropriate", that is, it may indicate that the encoding device 1 does not determine whether the associated image is suitable for task processing.

[0110] 10 is a diagram showing a second example of syntax for setting an identifier. The syntax includes an item "frame_not_for_inference."

[0111] 11 is a diagram showing a second example of setting identifier values ​​in syntax. As shown in (A), when the value of "frame_not_for_inference" is set to "1," it indicates that the associated image is "inappropriate." Also, when the value of "frame_not_for_inference" is set to "0," it indicates that the associated image is "appropriate."

[0112] As shown in (B), if the value of "frame_not_for_inference" is set to "0", it may indicate that the encoding device 1 does not specify "appropriate" or "inappropriate", that is, it may indicate that the encoding device 1 does not determine whether the associated image is suitable for task processing.

[0113] 12 is a simplified diagram showing the configuration of packet data PD in which encoded images P1 to P5 are stored. The packet data PD has a header area 41 and a payload area 42. The header area 41 has an SEI (Supplemental Enhancement Information) area 43 for storing additional information. The encoding unit 32 stores encoded data of an image in the payload area 42 and encoded data of an identifier associated with the image in the SEI area 43. However, instead of the SEI area 43, data may be stored in a VPS, SPS, PPS, PH, SH, APS, tile header, or the like. Furthermore, the SEI area 43 may be a prefix_SEI located before the payload area 42 or a suffix_SEI located after the payload area 42.

[0114] 13 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 includes a decoding unit 51 and a selection unit 52.

[0115] FIG. 14 is a flowchart showing the process executed by the circuit 21 included in the decoding device 2.

[0116] First, in step SP21, the circuit 21 receives the bit stream BS via the transmission path NW from the encoding device 1. The bit stream BS is input to the decoding unit 51.

[0117] Next, in step SP22, the circuit 21 decodes the multiple images and the identifiers associated with each image from the bit stream BS, thereby outputting image data D21 corresponding to the image data D1 and identifier data D22 corresponding to the identifier data D11. The image data D21 and identifier data D22 are input to the selection unit 52.

[0118] Next, in step SP23, the selection unit 52 selects at least one image to be used for task processing from among the multiple images included in the image data D21, based on the identifier included in the identifier data D22.

[0119] Next, in step SP24, the selection unit 52 outputs image data D2 including at least one selected image. The image data D2 is input to the task processing unit 4. The task processing unit 4 executes task processing using at least one image included in the input image data D2.

[0120] Fig. 15 is a diagram showing a first example of image selection based on identifier data D22. Fig. 15 shows an example in which, corresponding to Fig. 9A, setting the value of "frame_for_inference" to "1" indicates that the associated image is "appropriate," and setting the value of "frame_for_inference" to "0" indicates that the associated image is "inappropriate."

[0121] The image data D21 includes multiple images (five images P1 to P5 in this example) in chronological order. The identifier data D22 includes identifiers associated with each of the images P1 to P5. The identifier values ​​for the images P1 to P5 are set to "1," "0," "0," "1," and "1," respectively.

[0122] Of the multiple images P1 to P5 included in the image data D21, the selection unit 52 selects images P1, P4, and P5 whose identifier value is "1," and does not select images P2 and P3 whose identifier value is "0." As a result, the selection unit 52 outputs image data D2 that includes the selected images P1, P4, and P5.

[0123] Fig. 16 is a diagram showing a second example of image selection based on identifier data D22. Fig. 16 shows an example in which, corresponding to Fig. 11A, setting the value of "frame_not_for_inference" to "1" indicates that the associated image is "inappropriate," and setting the value of "frame_not_for_inference" to "0" indicates that the associated image is "appropriate."

[0124] Of the multiple images P1 to P5 included in the image data D21, the selection unit 52 does not select images P1, P4, and P5 with an identifier value of "1," but selects images P2 and P3 with an identifier value of "0." As a result, the selection unit 52 outputs image data D2 including the selected images P2 and P3.

[0125] Note that the designation of an image by an identifier merely suggests whether the selection is appropriate or inappropriate, and does not force selection or non-selection by the selection unit 52. Therefore, depending on the processing capacity and the like of the hardware resources of the decoding device 2, the selection unit 52 may not select an image that is deemed appropriate if there is no spare capacity, or may select an image that is deemed inappropriate if there is spare capacity.

[0126] According to the encoding device 1 of this embodiment, the identifier indicates at least one of whether the image is suitable for the task processing corresponding to the identifier and whether the image is not suitable for the task processing. Therefore, the decoding device 2 that receives the bitstream BS can appropriately select at least one image to be used for the task processing from among the multiple images decoded from the bitstream BS based on the identifier, thereby improving the accuracy of the execution of the task processing.

[0127] Furthermore, according to the encoding device 1 of this embodiment, the circuit 11 can appropriately determine whether or not an image is suitable for task processing based on the result of image analysis before encoding the image.

[0128] Furthermore, according to the encoding device 1 of this embodiment, the task processing includes a machine task, and therefore the decoding device 2 that receives the bitstream BS can select an image suitable for the machine task based on the identifier, thereby enabling the machine task to be executed appropriately.

[0129] Furthermore, according to the encoding device 1 of this embodiment, the task processing includes human vision, so that the decoding device 2 that receives the bitstream BS can select images suitable for human vision based on the identifiers, thereby enabling the human vision to be executed appropriately.

[0130] Furthermore, in the encoding device 1 according to this embodiment, the identifier includes a first value and a second value, the first value indicating that the image is suitable for task processing, and the second value indicating that the image is not suitable for task processing. Therefore, the decoding device 2 that receives the bitstream BS can easily determine which images to select and which images not to select based on the first value and the second value.

[0131] Furthermore, according to the encoding device 1 of this embodiment, the identifier includes a first value and a second value, where the first value indicates that the image is suitable for task processing or that the image is not suitable for task processing, and the second value indicates that the suitability of the image for task processing is not specified. Therefore, the decoding device 2 that receives the bitstream BS can easily determine which images to select or not to select based on the first value. Furthermore, the decoding device 2 can arbitrarily determine whether to select an image associated with an identifier of the second value depending on its own processing capabilities, etc.

[0132] Furthermore, according to the encoding device 1 of this embodiment, the identifier is stored in the header area 41 of the image associated with the identifier. Therefore, the decoding device 2 that receives the bitstream BS can easily obtain the identifier associated with each image from the header area 41 of each image.

[0133] According to the decoding device 2 of this embodiment, the identifier indicates at least one of whether the image is suitable for the task processing corresponding to the identifier and whether the image is not suitable for the task processing. Therefore, at least one image to be used for the task processing can be appropriately selected from multiple images decoded from the bitstream BS based on the identifier, thereby improving the accuracy of the execution of the task processing.

[0134] Furthermore, the decoding device 2 according to this embodiment can select an image suitable for task processing based on the identifier, thereby improving the accuracy of task processing execution.

[0135] Furthermore, according to the decoding device 2 of this embodiment, task processing includes a machine task, so that an image suitable for the machine task can be selected based on the identifier, thereby enabling the machine task to be executed appropriately.

[0136] Furthermore, according to the decoding device 2 of this embodiment, the task processing includes human vision, so that an image suitable for human vision can be selected based on the identifier, thereby enabling human vision to be executed appropriately.

[0137] Furthermore, in the decoding device 2 according to this embodiment, the identifier includes a first value and a second value, the first value indicating that the image is suitable for task processing, and the second value indicating that the image is not suitable for task processing. Therefore, the circuit 21 can easily determine which images to select and which images not to select based on the first value and the second value.

[0138] Furthermore, according to the decoding device 2 of this embodiment, the identifier includes a first value and a second value, where the first value indicates that the image is suitable for task processing or that the image is not suitable for task processing, and the second value indicates that the suitability of the image for task processing is not specified. Therefore, the circuit 21 can easily determine which image to select or not to select based on the first value. Furthermore, the circuit 21 can arbitrarily determine whether to select an image associated with an identifier of the second value depending on its own processing power, etc.

[0139] Furthermore, according to the decoding device 2 of this embodiment, the identifier is stored in the header area 41 of the image associated with the identifier, so that the circuit 21 can easily obtain the identifier associated with each image from the header area 41 of each image.

[0140] Various modifications of the embodiments of the present disclosure will be described below, and the modifications can be applied in any combination.

[0141] (First Modification) In the above embodiment, each identifier is stored in the header area 41 of each image associated with that identifier, but this is not limited to this. The circuit 11 may define an image group including multiple images, and store multiple identifiers associated with multiple images included in the image group together in the header area 41 of one image (for example, the first image) in the image group. The image group may be a GOP (Group Of Pictures) including I pictures, P pictures, and B pictures, or may be an arbitrary collection of multiple images.

[0142] 17 is a diagram showing image data D1 and identifier data D11. The image data D1 includes multiple images (32 images P1 to P32 in this example) in chronological order. The images P1 to P32 are included in the same image group. The identifier data D11 includes 32 identifiers associated with the 32 images P1 to P32. The identifier set including the 32 identifiers is stored in the header area 41 of the first image P1 of the image group.

[0143] 17, one identifier is associated with one image, but one identifier may be associated with multiple images. For example, when multiple identifiers with the same value are consecutive, the first identifier among the multiple identifiers may be associated with multiple images corresponding to the multiple identifiers.

[0144] 18 is a diagram showing an example of syntax related to setting an identifier. The syntax includes an item "num_frames_minus1." The value of "num_frames_minus1" is set to the number of images included in the image group minus "1." If the image group includes 32 images P1 to P32, the value of "num_frames_minus1" is set to "31."

[0145] The loop processing indicated by the for statement sets the value of "frame_for_inference" for each of the multiple images included in the image group. At this time, which image each of the multiple "frame_for_inference" is associated with may be determined according to the order of the loop processing indicated by the for statement.

[0146] It should be noted that instead of using a repetitive statement such as a for statement, multiple setting values ​​may be described in itemized form. This also applies to the following modifications.

[0147] According to this modification, the decoding device 2 that receives the bitstream BS can easily obtain, from the header area 41 of one image, multiple identifiers associated with multiple images included in the image group.

[0148] (Second Modification) In the first modification, the same number of identifiers as the number of images included in the image group are stored together in the header area 41 of one image in the image group, but this is not limited to this. The circuit 11 may further encode, into the bitstream BS, identification information for identifying an image associated with an identifier from among the multiple images included in the image group.

[0149] 19 is a diagram showing an example of syntax for setting an identifier. The syntax includes an "image_identifier[i]" item indicating specific information. The value of "image_identifier[i]" is set using a POC (Picture Order Count), a POC difference value, a count number, an SPS (Sequence Parameter Set) ID, a PPS (Picture Parameter Set) ID, or the like.

[0150] The value of "frame_for_inference" is set for the image identified by the identification information among the multiple images included in the image group.

[0151] Note that an image that is not specified by the specification information, that is, an image that is not associated with an identifier, indicates that it is not specified whether the image is suitable for task processing.

[0152] Furthermore, if the first image in an image group does not include an identifier set, it indicates that the suitability of the image for task processing will not be determined for all images included in the group.

[0153] According to this variant, by identifying images associated with identifiers using specific information, the number of identifiers included in the identifier set does not need to match the number of images included in the image group, thereby reducing the amount of data for the identifiers.

[0154] (Third Modification) In the above embodiment, an identifier is associated with every image, but this is not limited to this. The circuit 11 may not associate an identifier with an image that is not specified as being suitable for task processing.

[0155] 20 is a diagram showing a first example of setting identifier values ​​in syntax, corresponding to FIG. 9. As shown in (A), when the value of "frame_for_inference" is set to "1", it indicates that the associated image is "appropriate". On the other hand, when the value of "frame_for_inference" is set to "0", it indicates that the associated image is "inappropriate". Furthermore, an image that is not associated with an identifier indicates that the encoding device 1 does not determine whether the image is suitable for task processing.

[0156] As shown in (B), if the value of "frame_for_inference" is set to "0", or if an image is not associated with an identifier, it may indicate that the encoding device 1 does not determine whether the image is suitable for task processing.

[0157] 21 is a diagram showing a second example of setting identifier values ​​in syntax, corresponding to FIG. 10. As shown in (A), when the value of "frame_not_for_inference" is set to "1," it indicates that the associated image is "inappropriate." When the value of "frame_not_for_inference" is set to "0," it indicates that the associated image is "appropriate." Furthermore, an image that is not associated with an identifier indicates that the encoding device 1 does not determine whether the image is suitable for task processing.

[0158] As shown in (B), if the value of "frame_not_for_inference" is set to "0", or if an image is not associated with an identifier, it may indicate that the encoding device 1 does not determine whether the image is suitable for task processing.

[0159] Fig. 22 is a diagram showing an example of syntax related to identifier setting based on Fig. 18. Fig. 23 is a diagram showing an example of syntax related to identifier setting based on Fig. 19. The syntax includes an item of "frame_for_inference_present_flag[i]". For an image associated with an identifier, the value of "frame_for_inference_present_flag[i]" is set to "1". For an image not associated with an identifier, the value of "frame_for_inference_present_flag[i]" is set to "0".

[0160] (Fourth Modification) In the above embodiment, the task processing performed by the task processing unit 4 is set to one specific type, but this is not limited to this. The circuit 11 may further encode setting information for arbitrarily setting the type of task processing into the bitstream BS. The setting information may be stored in an SEI area included in the header area of ​​the bitstream BS.

[0161] FIG. 24 is a diagram showing an example of syntax related to task processing settings. The syntax includes a "vision_task" field. A character string indicating the type of task processing is written in the "vision_task" field. Alternatively, an integer value indicating the type of task processing may be written instead of a character string. In this case, a correspondence between the integer value and the type of task processing is set, and the encoding device 1 and the decoding device 2 share this setting information in advance.

[0162] According to this modification, the type of task processing corresponding to the identifier can be set arbitrarily.

[0163] (Fifth Modification) In the above embodiment, the setting unit 31 sets the value of the identifier based on the result of image analysis of the image before encoding, but this is not limited thereto. The setting unit 31 may set the value of the identifier based on the image after encoding.

[0164] FIG. 25 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 includes a setting unit 31 and an encoding unit 32. The setting unit 31 receives image data D1 including multiple images before encoding and image data D12 including multiple images encoded by the encoding unit 32. The setting unit 31 derives encoding conditions for encoding each image by the encoding unit 32 based on the image data D1 and D12. The encoding conditions include conditions related to the image quality of the encoded image, such as the value of a quantization parameter, the PSNR value of the encoded image, or the hierarchical level of the temporal layer. The larger the value of the quantization parameter, the lower the image quality of the encoded image. The smaller the PSNR value, the lower the image quality of the encoded image. The higher the hierarchical level of the temporal layer, the lower the image quality of the encoded image.

[0165] When the task processing is human vision, low-quality images reduce visibility. When the task processing is object detection, low-quality images reduce estimation accuracy. Therefore, when the task processing is human vision or object detection, the setting unit 31 may determine that an image whose image quality after encoding is equal to or greater than a threshold is "appropriate," and that an image whose image quality after encoding is less than the threshold is "inappropriate."

[0166] When the task processing is object tracking, tolerance for changes in image quality is high, so the trajectory of an object can be detected even in images with low image quality. Therefore, when the task processing is object tracking, the setting unit 31 may set all images as "appropriate."

[0167] The setting unit 31 inputs, to the encoding unit 32, identifier data D11 including an identifier whose value is set based on the encoded image.

[0168] According to this modification, after encoding an image, the circuit 11 can appropriately determine whether the image is suitable for task processing based on the encoding conditions.

[0169] Furthermore, according to this modification, after encoding an image, the circuit 11 can appropriately determine whether the image is suitable for task processing based on the image quality.

[0170] (Sixth Modification) In the above embodiment, the setting unit 31 sets the value of the identifier based on the result of image analysis of the image before encoding, but this is not limiting. The setting unit 31 may set the value of the identifier based on related information related to the image before encoding.

[0171] 26 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 has a setting unit 31 and an encoding unit 32. Image data D1 including a plurality of images before encoding and related information D13 detected by the imaging unit 3 when the imaging unit 3 captures a moving image are input to the setting unit 31 from the imaging unit 3.

[0172] The camera of the photographing unit 3 has various built-in sensors such as a proximity sensor, a gyro sensor, an acceleration sensor, a geomagnetic sensor, etc. The related information D13 includes detection values ​​detected by these sensors.

[0173] The setting unit 31 sets the value of the identifier based on the image data D1 and related information D13.

[0174] When an object located immediately in front of the camera is detected by the proximity sensor and this state continues for a certain period of time or more, the setting unit 31 may set the image captured during this period as "inappropriate." When an object located immediately in front of the camera is detected by the proximity sensor and the object is moving at a high speed equal to or greater than a certain speed, the setting unit 31 may set the image captured during this period as "appropriate."

[0175] If camera shake or shaking, such as camera shake, is detected by a gyro sensor, acceleration sensor, or geomagnetic sensor, and the level of the shake or shaking is above a threshold, the setting unit 31 may classify images taken during that time as "inappropriate."

[0176] According to this modification, the circuit 11 can appropriately determine whether or not an image is suitable for task processing based on the related information D13 before encoding the image.

[0177] (Seventh Modification) In the above embodiment, there is only one type of task processing, but this is not limited to this. The task processing may include multiple types of task processing, and the circuit 11 may individually set the image as "appropriate" or "inappropriate" for each task processing. Below, an example will be described in which the task processing includes a first task processing and a second task processing, the first task processing is object detection, and the second task processing is human vision. However, the task processing may include three or more types of task processing, and the types of task processing are not limited to object detection and human vision.

[0178] 27 is a simplified diagram showing the configuration of an image processing system according to a seventh modified example. The image processing system includes a first task processing unit 4A and a second task processing unit 4B. The first task processing unit 4A executes a first task processing, which is object detection, based on first image data D2A input from the decoding device 2. The second task processing unit 4B executes a second task processing, which is human vision, based on second image data D2B input from the decoding device 2.

[0179] 28 is a diagram showing an example of task processing setting information 60. The setting information 60 indicates that the first task processing is object detection and the second task processing is human vision. The encoding device 1 and the decoding device 2 share the setting information 60 in advance.

[0180] 29 is a simplified diagram showing the configuration of the circuit 11 included in the encoding device 1. The circuit 11 has a setting unit 31 and an encoding unit 32. The setting unit 31 inputs, to the encoding unit 32, first identifier data D11A including a plurality of first identifiers related to a first task process and second identifier data D11B including a plurality of second identifiers related to a second task process.

[0181] FIG. 30 is a flowchart showing the processing executed by the circuit 11 included in the encoding device 1.

[0182] First, in step SP11, the circuit 11 acquires a plurality of images contained in the image data D1 input from the photographing unit 3. The image data D1 is input to the setting unit 31 and the encoding unit 32.

[0183] Next, in step SP12A, the setting unit 31 performs image analysis on each image to be encoded that is included in the image data D1, and sets values ​​for the first identifier and the second identifier based on the results of the image analysis. The setting unit 31 inputs, to the encoding unit 32, first identifier data D11A including the first identifier whose value has been set, and second identifier data D11B including the second identifier whose value has been set.

[0184] Next, in step SP13A, the encoding unit 32 encodes the multiple images contained in the image data D1 input from the photographing unit 3, the multiple first identifiers contained in the first identifier data D11A input from the setting unit 31, and the multiple second identifiers contained in the second identifier data D11B input from the setting unit 31 into a bit stream BS.

[0185] Next, in step SP14, the circuit 11 transmits the bit stream BS to the decoding device 2 via the transmission path NW.

[0186] 31 is a diagram showing image data D1, first identifier data D11A, and second identifier data D11B. The image data D1 includes multiple images (five images P1 to P5 in this example) arranged in chronological order. The first identifier data D11A and second identifier data D11B include first identifiers and second identifiers associated with each of the images P1 to P5. The first identifier and second identifier are each 1-bit data, and the values ​​of the first identifier and second identifier are set to "1" or "0."

[0187] 32 is a diagram showing a bit string including first identifier data D11A and second identifier data D11B. The first two bits of the bit string indicate the values ​​of the first identifier and second identifier associated with image P1. The next two bits of the bit string indicate the values ​​of the first identifier and second identifier associated with image P2. Similarly, each successive two bits in the bit string indicates the values ​​of the first identifier and second identifier associated with images P3 to P5. Note that if there are N types of task processing, each successive N bits in the bit string indicates the values ​​of N identifiers associated with the same image.

[0188] FIG. 33A is a diagram showing a first example of syntax related to setting a first identifier and a second identifier. The syntax includes a "num_vision_task_minus1" item. The value of "num_vision_task_minus1" is set to a value obtained by subtracting "1" from the number of task processing types. The value of "frame_for_inference[i]" when i = 1 corresponds to the value of the first identifier, and the value of "frame_for_inference[i]" when i = 2 corresponds to the value of the second identifier. Setting the value of "frame_for_inference[i]" to "1" indicates that the associated image is "appropriate." Furthermore, setting the value of "frame_for_inference[i]" to "0" indicates that the associated image is "inappropriate."

[0189] Note that instead of a description using a repetitive statement such as a for statement, multiple setting values ​​of "frame_for_inference[i]" may be described in itemized form as shown in FIG. 33B.

[0190] 34A is a diagram showing a second example of syntax related to setting the first identifier and the second identifier. The value of "frame_not_for_inference[i]" when i = 1 corresponds to the value of the first identifier, and the value of "frame_not_for_inference[i]" when i = 2 corresponds to the value of the second identifier. Setting the value of "frame_not_for_inference[i]" to "1" indicates that the associated image is "inappropriate." Also, setting the value of "frame_not_for_inference[i]" to "0" indicates that the associated image is "appropriate."

[0191] Note that instead of a description using a repetitive statement such as a for statement, multiple setting values ​​of "frame_not_for_inference[i]" may be described in itemized form as shown in FIG. 34B.

[0192] 35 is a simplified diagram showing the configuration of the circuit 21 included in the decoding device 2. The circuit 21 has a decoding unit 51 and a selection unit 52. The decoding unit 51 inputs image data D21, first identifier data D22A including a plurality of first identifiers related to a first task process, and second identifier data D22B including a plurality of second identifiers related to a second task process to the selection unit 52.

[0193] FIG. 36 is a flowchart showing the processing executed by the circuit 21 included in the decoding device 2.

[0194] First, in step SP21, the circuit 21 receives the bit stream BS via the transmission path NW from the encoding device 1. The bit stream BS is input to the decoding unit 51.

[0195] Next, in step SP22A, the circuit 21 decodes the multiple images and the first and second identifiers associated with each image from the bit stream BS, and outputs image data D21 corresponding to the image data D1, first identifier data D22A corresponding to the first identifier data D11A, and second identifier data D22B corresponding to the second identifier data D11B. The image data D21, first identifier data D22A, and second identifier data D22B are input to the selection unit 52.

[0196] Next, in step SP23A, the selection unit 52 selects at least one first image to be used in the first task processing from among the multiple images included in the image data D21, based on the first identifier included in the first identifier data D22A. The selection unit 52 also selects at least one second image to be used in the second task processing from among the multiple images included in the image data D21, based on the second identifier included in the second identifier data D22B.

[0197] Next, in step SP24A, the selection unit 52 outputs first image data D2A including at least one selected first image and second image data D2B including at least one selected second image. The first image data D2A is input to the first task processing unit 4A, and the second image data D2B is input to the second task processing unit 4B. The first task processing unit 4A executes a first task process using at least one first image included in the input first image data D2A. The second task processing unit 4B executes a second task process using at least one second image included in the input second image data D2B.

[0198] Fig. 37 is a diagram showing an example of selection of a first image and a second image based on the first identifier data D22A and the second identifier data D22B. Fig. 37 shows an example in which setting the value of "frame_for_inference[i]" to "1" indicates that the associated image is "appropriate," and setting the value of "frame_for_inference[i]" to "0" indicates that the associated image is "inappropriate."

[0199] The image data D21 includes a plurality of images (five images P1 to P5 in this example) in chronological order. The first identifier data D22A includes a first identifier associated with each of the images P1 to P5. The values ​​of the first identifiers are set to "1," "0," "0," "1," and "1" for the images P1 to P5, respectively. The second identifier data D22B includes a second identifier associated with each of the images P1 to P5. The value of the second identifier is set to "1" for all of the images P1 to P5.

[0200] Of the multiple images P1 to P5 included in the image data D21, the selection unit 52 selects images P1, P4, and P5 whose first identifier has a value of "1," and does not select images P2 and P3 whose first identifier has a value of "0." As a result, the selection unit 52 outputs first image data D2A that includes the selected images P1, P4, and P5.

[0201] Furthermore, the selection unit 52 selects, from among the multiple images P1 to P5 included in the image data D21, images P1 to P5 whose second identifier has a value of 1. As a result, the selection unit 52 outputs second image data D2B that includes all of the images P1 to P5.

[0202] According to this modification, a first image suitable for a machine task can be selected based on the first identifier, thereby enabling the machine task to be performed appropriately, and a second image suitable for human vision can be selected based on the second identifier, thereby enabling the human vision to be performed appropriately.

[0203] (Eighth Modification) In the above embodiment, each identifier is one-bit data, but this is not limiting, and each identifier may be multi-bit data.

[0204] 38 is a diagram showing an example of setting identifier values ​​in syntax. In this example, each identifier is 2-bit data.

[0205] Setting the value of "frame_for_inference" to "0" indicates that the encoding device 1 does not specify whether the associated image is "appropriate" or "inappropriate," i.e., the encoding device 1 does not specify whether the associated image is suitable for task processing. Setting the value of "frame_for_inference" to "1" indicates that the associated image is "inappropriate." Setting the value of "frame_for_inference" to "2" indicates that the associated image is "appropriate." The value of "3" for "frame_for_inference" is allocated for future expansion and is currently undefined.

[0206] According to this modified example, each 2-bit identifier can specify three states: "appropriate," "inappropriate," and "unspecified," so there is no need to mix images that are associated with identifiers with images that are not associated with identifiers.

[0207] (Ninth Modification) In the ninth modification, a combination of the third modification and the seventh modification will be described.

[0208] 39 is a diagram showing an example of selection of a first image and a second image based on the first identifier data D22A and the second identifier data D22B. In FIG. 39, setting the value of "frame_for_inference[i]" to "1" indicates that the associated image is "appropriate," and setting the value of "frame_for_inference[i]" to "0" indicates that the associated image is "inappropriate," and an example is shown in which an image that is not associated with the first identifier or the second identifier does not indicate that the encoding device 1 determines whether the image is suitable for task processing.

[0209] Of the multiple images P1 to P32 included in the image data D21, the selection unit 52 selects images P2 and P32 whose first identifier has a value of "1" and image P1 whose first identifier and second identifier are not associated with each other. The selection unit 52 does not select image P4 whose first identifier has a value of "0" and image P3 whose first identifier and second identifier are not associated with each other. As a result, the selection unit 52 outputs first image data D2A that includes the selected images P1, P2, and P32.

[0210] Of the multiple images P1 to P32 included in the image data D21, the selection unit 52 selects images P4 and P32 whose second identifier has a value of "1" and image P1 whose first identifier and second identifier are not associated with each other. The selection unit 52 does not select image P2 whose second identifier has a value of "0" and image P3 whose first identifier and second identifier are not associated with each other. As a result, the selection unit 52 outputs second image data D2B that includes the selected images P1, P4, and P32.

[0211] (Tenth Modification) In the seventh modification, the first task processing executed by the first task processing unit 4A and the second task processing executed by the second task processing unit 4B are set in advance by the setting information 60, but this is not limiting. The circuit 11 may further encode, into the bitstream BS, correspondence information indicating the correspondence between the first identifier and the second identifier and the first task processing and the second task processing.

[0212] FIG. 40 is a diagram showing an example of syntax related to task processing settings. The syntax includes a "num_identifier_minus1" field. The value of "num_identifier_minus1" is set to the number of identifiers associated with an image minus "1." The syntax also includes a "vision_task[i]" field. A character string indicating the type of task processing is written in the "vision_task[i]" field. Alternatively, an integer value indicating the type of task processing may be written instead of a character string. In this case, a correspondence between the integer value and the type of task processing is set, and the encoding device 1 and the decoding device 2 share this setting information in advance.

[0213] FIG. 41 is a diagram showing a simplified configuration of a bitstream BS. The bitstream BS has a header area 71 and a payload area 72. The header area 71 has an SEI area 73 for storing additional information. The encoding unit 32 stores encoded data of the correspondence information indicating the correspondence between multiple identifiers and multiple task processes in the SEI area 73. However, instead of the SEI area 73, the information may be stored in a VPS, SPS, PPS, PH, SH, APS, tile header, or the like. Furthermore, the SEI area 73 may be a prefix_SEI located before the payload area 72, or a suffix_SEI located after the payload area 72.

[0214] According to this modification, a plurality of task processes can be arbitrarily set in the correspondence information, and the execution accuracy of each of the plurality of task processes that have been set can be improved.

[0215] (Eleventh Modification) In the eleventh modification, a combination of the first modification and the seventh modification will be described.

[0216] 42 is a diagram showing an example of syntax for setting the first identifier and the second identifier. When the value of "frame_for_inference[i][j]" is set to "1," it indicates that the i-th image in the image group is "appropriate" for the j-th task processing. When the value of "frame_for_inference[i][j]" is set to "0," it indicates that the i-th image in the image group is "inappropriate" for the j-th task processing.

[0217] (Twelfth Modification) In the twelfth modification, a combination of the second modification and the seventh modification will be described.

[0218] 43 is a diagram showing an example of syntax for setting the first identifier and the second identifier. When the value of "frame_for_inference[i][j]" is set to "1," it indicates that the i-th image identified by the specific information in the image group is "appropriate" for the j-th task processing. When the value of "frame_for_inference[i][j]" is set to "0," it indicates that the i-th image identified by the specific information in the image group is "inappropriate" for the j-th task processing.

[0219] (Thirteenth Modification) In the thirteenth modification, a combination of the third modification and the seventh modification will be described.

[0220] 44 is a diagram showing an example of syntax for setting the first identifier and the second identifier. For an image associated with an identifier, the value of "frame_for_inference_present_flag[i]" is set to "1." For an image not associated with an identifier, the value of "frame_for_inference_present_flag[i]" is set to "0."

[0221] For an image associated with an identifier, setting the value of "frame_for_inference[i][j]" to "1" indicates that the ith image in the image group is "suitable" for the jth task. Setting the value of "frame_for_inference[i][j]" to "0" indicates that the ith image in the image group is "unsuitable" for the jth task.

[0222] (Fourteenth Modification) In the fourteenth modification, a combination of the second modification, the third modification, and the seventh modification will be described.

[0223] 45 is a diagram showing an example of syntax for setting a first identifier and a second identifier. For an image associated with an identifier, setting the value of "frame_for_inference[i][j]" to "1" indicates that the i-th image identified by specific information in the image group is "appropriate" for the j-th task processing. Setting the value of "frame_for_inference[i][j]" to "0" indicates that the i-th image identified by specific information in the image group is "inappropriate" for the j-th task processing.

[0224] The present disclosure is particularly useful when applied to an image processing system that includes an encoding device that encodes an image into a bitstream and transmits the bitstream, and a decoding device that decodes an image from the received bitstream.

Claims

1. A decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit decodes from a bitstream an image and an identifier associated with the image, the identifier indicating at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

2. The decoding device according to claim 1, wherein the image includes a plurality of images, and the circuitry further selects at least one image to be used for the task processing from among the plurality of images based on the identifier.

3. The decoding device according to claim 1, wherein the task processing includes a machine task.

4. The decoding device of claim 1, wherein the task processing includes human vision.

5. The decoding device according to claim 1, wherein the identifier includes a plurality of identifiers, the task process includes a plurality of task processes, and the plurality of identifiers correspond to different task processes among the plurality of task processes.

6. The decoding device of claim 5, wherein the plurality of identifiers include a first identifier and a second identifier, the task process corresponding to the first identifier includes a machine task, and the task process corresponding to the second identifier includes human vision.

7. The decoding device of claim 1, wherein the identifier includes a first value and a second value, the first value indicating that the image is suitable for the task processing, and the second value indicating that the image is not suitable for the task processing.

8. The decoding device of claim 1, wherein the identifier includes a first value and a second value, the first value indicating that the image is suitable for the task processing or that the image is not suitable for the task processing, and the second value indicating that it does not specify whether the image is suitable for the task processing.

9. The decoding device of claim 1, wherein the image includes a plurality of images, the plurality of images including an image not associated with the identifier, the image not associated with the identifier indicating an image that does not specify whether it is suitable for the task processing.

10. The decoding device of claim 1, wherein the identifier includes one or more identifiers, the task process includes one or more task processes, and the circuitry further decodes correspondence information from the bitstream indicating correspondence between the one or more identifiers and the one or more task processes.

11. The decoding device according to claim 10, wherein the correspondence information is stored in a header area of ​​the bitstream.

12. The decoding device according to claim 1, wherein the identifier is stored in a header area of ​​the image associated with the identifier.

13. The decoding device of claim 1, wherein the image includes a plurality of images, the identifier includes one or more identifiers, and the one or more identifiers are stored in a header area of ​​one of the images included in the plurality of images and are associated with the one image and one or more images different from the one image.

14. The decoding device according to any one of claims 11 to 13, wherein the header area is an SEI area.

15. The decoding device of claim 1, wherein the image includes a plurality of images, and the circuitry further decodes, from the bitstream, identification information for identifying the image of the plurality of images associated with the identifier.

16. An encoding device comprising: a circuit; and a memory connected to the circuit, the circuit encoding an image and an identifier associated with the image into a bitstream, the identifier indicating at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

17. The encoding device according to claim 16, wherein said circuit sets the value of said identifier based on encoding conditions when said image was encoded.

18. The encoding device of claim 16, wherein the circuit sets the value of the identifier based on the image quality of the image after encoding.

19. The encoding device according to claim 16, wherein the circuit acquires an image to be encoded and related information relating to the image from an external device, and sets the value of the identifier based on the related information.

20. The encoding device according to claim 16, wherein the circuit performs image analysis on the image to be encoded, and sets the value of the identifier based on the result of the image analysis.

21. The encoding device of claim 16, wherein the task processing includes a machine task.

22. The encoding device of claim 16, wherein the task processing includes human vision.

23. The encoding device according to claim 16, wherein the identifier includes a plurality of identifiers; the task process includes a plurality of task processes; and the plurality of identifiers correspond to different task processes among the plurality of task processes.

24. The encoding device of claim 23, wherein the plurality of identifiers include a first identifier and a second identifier, the task process corresponding to the first identifier includes a machine task, and the task process corresponding to the second identifier includes human vision.

25. The encoding device of claim 16, wherein the identifier includes a first value and a second value, the first value indicating that the image is suitable for processing the task, and the second value indicating that the image is not suitable for processing the task.

26. The encoding device of claim 16, wherein the identifier includes a first value and a second value, the first value indicating that the image is suitable for the task processing or that the image is not suitable for the task processing, and the second value indicating that the image is not suitable for the task processing.

27. The encoding device of claim 16, wherein the image includes a plurality of images, the plurality of images including an image not associated with the identifier, the image not associated with the identifier indicating an image that does not specify whether it is suitable for the task processing.

28. The encoding device of claim 16, wherein the identifier includes one or more identifiers, and the task process includes one or more task processes, and the circuitry further encodes, into the bitstream, correspondence information indicating correspondence between the one or more identifiers and the one or more task processes.

29. The encoding device according to claim 28, wherein the correspondence information is stored in a header area of ​​the bitstream.

30. The encoding device according to claim 16, wherein the identifier is stored in a header area of ​​the image associated with the identifier.

31. The encoding device of claim 16, wherein the image includes a plurality of images, the identifier includes one or more identifiers, and the one or more identifiers are stored in a header area of ​​one of the images included in the plurality of images and are associated with the one image and one or more images different from the one image.

32. The encoding device according to any one of claims 29 to 31, wherein the header region is an SEI region.

33. The encoding device of claim 16, wherein the image includes a plurality of images, and the circuitry further encodes, into the bitstream, identification information for identifying the image of the plurality of images associated with the identifier.

34. A decoding method, comprising: a decoding device decoding an image and an identifier associated with the image from a bitstream, the identifier indicating at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.

35. An encoding method, comprising: an encoding device encoding, into a bitstream, an image and an identifier associated with the image, the identifier indicating at least one of that the image is suitable for a task processing corresponding to the identifier and that the image is not suitable for the task processing.