Image decoding method, image coding method, image decoding device, and image coding device

Abstract

To provide an image decoding device capable of properly constructing a feature map.SOLUTION: The image decoding device which decodes a plurality of feature images, aligns the feature images in a raster scanning order, and thereby constructs a feature map.SELECTED DRAWING: Figure 21

Description

[Technical field]

[0001] The present disclosure relates to an image decoding method, an image encoding method, an image decoding device, and an image encoding device. [Background technology]

[0002] A neural network is a set of algorithms that attempt to recognize underlying relationships in a data set through a process that mimics the way the human brain processes data. In this sense, a neural network refers to a system of neurons that can be organic or artificial in nature. Different types of neural networks in deep learning, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and artificial neural networks (ANNs), are changing the way we interact with the world. These different types of neural networks are at the heart of the deep learning revolution, powering applications such as unmanned aerial vehicles, autonomous vehicles, and speech recognition. CNNs, which consist of multiple stacked layers, are the class of deep neural networks most commonly applied to the analysis of visual images.

[0003] A feature image is a unique representation of the characteristics of an image or the objects contained in it. For example, in a convolutional layer of a neural network, a feature image is obtained as the output of applying a desired filter to the entire image. Multiple feature images can be obtained by applying multiple filters in multiple convolutional layers, and a feature map can be created by arranging the multiple feature images.

[0004] The feature maps are typically associated with a task processor that performs task processing, such as a neural network task. This setup typically enables the best inference results for a particular machine analysis task.

[0005] When the feature map created by the encoder is used by the decoder, the encoder encodes the created feature map and transmits a bit stream including the encoded data of the feature map to the decoder. The decoder decodes the feature map based on the received bit stream. The decoder inputs the decoded feature map to a task processing device that executes a predetermined task such as a neural network task.

[0006] In the background art, when multiple task processing devices on the decoder side execute multiple neural network tasks using multiple feature maps, it is necessary to implement multiple sets of encoders and decoders corresponding to each of the multiple task processing devices, which complicates the system configuration.

[0007] Incidentally, the image coding system architecture according to the background art is disclosed in, for example, Patent Documents 1 and 2. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] US Patent No. 2010 / 0046635 [Patent Document 2] US Patent No. 2021 / 0027470 Summary of the Invention [Problem to be solved by the invention]

[0009] The present disclosure aims to appropriately construct feature maps. [Means for solving the problem]

[0010] An image decoding device according to one aspect of the present disclosure constructs a feature map by decoding a plurality of feature images and arranging the plurality of feature images in a raster scan order. Effect of the Invention

[0011] According to the present disclosure, a feature map can be appropriately constructed. [Brief description of the drawings]

[0012] [Figure 1] 5 is a flowchart showing a processing procedure of an image decoding method according to the first embodiment of the present disclosure. [Diagram 2] 4 is a flowchart showing a processing procedure of an image encoding method according to the first embodiment of the present disclosure. [Diagram 3] FIG. 1 is a diagram illustrating an example of the configuration of an image processing system according to the background art. [Figure 4] 1 is a diagram illustrating an example of the configuration of an image processing system according to a first embodiment of the present disclosure. [Diagram 5] FIG. 1 is a diagram illustrating a first exemplary configuration of an encoding device and a decoding device. [Figure 6] FIG. 2 is a diagram illustrating a second exemplary configuration of an encoding device and a decoding device. [Figure 7] 2 is a block diagram showing a configuration of a video decoder according to the first embodiment of the present disclosure. FIG. [Figure 8] 1 is a block diagram showing a configuration of a video encoder according to a first embodiment of the present disclosure. [Figure 9] FIG. 11 is a diagram illustrating a first example of a feature map creation process. [Figure 10] FIG. 11 is a diagram illustrating a first example of a feature map creation process. [Figure 11] FIG. 11 is a diagram illustrating a first example of the operation of a selection unit. [Figure 12] FIG. 11 is a diagram illustrating a second example of the operation of the selection unit. [Figure 13] FIG. 11 is a diagram illustrating a second example of the process of creating a feature map. [Figure 14] FIG. 11 is a diagram illustrating a second example of the process of creating a feature map. [Figure 15] FIG. 1 illustrates an example of a neural network task. [Figure 16] FIG. 1 illustrates an example of a neural network task. [Figure 17]FIG. 1 is a diagram showing an example of using both inter-prediction and intra-prediction. [Figure 18] 10 is a flowchart showing a processing procedure of an image decoding method according to a second embodiment of the present disclosure. [Figure 19] 10 is a flowchart showing a processing procedure of an image encoding method according to a second embodiment of the present disclosure. [Figure 20] FIG. 11 is a diagram illustrating an example of the configuration of an image processing system according to a second embodiment of the present disclosure. [Figure 21] FIG. 11 is a block diagram showing a configuration of a decoding device according to a second embodiment of the present disclosure. [Figure 22] FIG. 11 is a block diagram showing a configuration of an encoding device according to a second embodiment of the present disclosure. [Diagram 23] FIG. 13 is a diagram showing another example of the feature map. [Figure 24] FIG. 13 is a diagram showing the relationship between the size of a feature image and the size of an encoding block. [Diagram 25] FIG. 13 is a diagram showing the relationship between the size of a feature image and the size of an encoding block. [Figure 26] FIG. 13 is a diagram showing a first example of a scanning order. [Figure 27] FIG. 13 is a diagram illustrating a second example of a scanning order. [Figure 28] FIG. 13 is a diagram illustrating an example of division into a plurality of segments. [Figure 29] FIG. 13 is a diagram illustrating an example of division into a plurality of segments. [Diagram 30] FIG. 13 is a diagram illustrating an example of division into a plurality of segments. [Diagram 31] FIG. 13 is a diagram showing a scanning order when one feature image is divided into a plurality of coding blocks and coded. [Diagram 32] FIG. 13 is a diagram showing a scanning order when one feature image is divided into a plurality of coding blocks and coded. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (Findings on which this disclosure is based) 3 is a diagram showing an example of the configuration of an image processing system 1100 according to the background art. The image processing system 1100 includes a plurality of task processing units 1103A-1103N that execute predetermined task processing such as a neural network task on the decoder side. For example, the task processing unit 1103A executes face landmark detection processing, and the task processing unit 1103B executes face direction detection processing. The image processing system 1100 also includes sets of encoding devices 1101A-1101N and decoding devices 1102A-1102N corresponding to each of the plurality of task processing units 1103A-1103N.

[0014] For example, the encoding device 1101A creates a feature map A based on an input image or feature, and encodes the created feature map A to transmit a bit stream including encoded data of the feature map A to the decoding device 1102A. The decoding device 1102A decodes the feature map A based on the received bit stream, and inputs the decoded feature map A to the task processing unit 1103A. The task processing unit 1103A executes a predetermined task processing using the input feature map A, and outputs an estimation result.

[0015] The problem with the background art shown in FIG. 3 is that it is necessary to implement multiple sets of encoding devices 1101A-1101N and decoding devices 1102A-1102N corresponding to the multiple task processing devices 1103A-1103N, respectively, making the system configuration complicated.

[0016] To solve this problem, the present inventor has introduced a new method in which an image encoding device transmits multiple feature maps included in the same bitstream to an image decoding device, and the image decoding device selects a desired feature map from the multiple decoded feature maps and inputs it to each of multiple task processing devices. This makes it unnecessary to implement multiple sets of image encoding devices and image decoding devices corresponding to each of multiple task processing devices, and only one set of image encoding devices and image decoding devices is required, thereby simplifying the system configuration.

[0017] Next, each aspect of the present disclosure will be described.

[0018] In an image decoding method according to one embodiment of the present disclosure, an image decoding device receives a bit stream including encoded data of a plurality of feature maps relating to an image from an image encoding device, decodes the plurality of feature maps based on the received bit stream, selects a first feature map from the decoded plurality of feature maps, outputs the first feature map to a first task processing device that performs a first task processing based on the first feature map, and selects a second feature map from the decoded plurality of feature maps and outputs the second feature map to a second task processing device that performs a second task processing based on the second feature map.

[0019] According to this aspect, the image decoding device selects a first feature map from the multiple decoded feature maps and outputs it to the first task processing device, and selects a second feature map from the multiple decoded feature maps and outputs it to the second task processing device. This makes it possible to simplify the system configuration since it is not necessary to implement multiple sets of image encoding devices and image decoding devices corresponding to each of the multiple task processing devices.

[0020] In the above aspect, the image decoding device selects the first feature map and the second feature map based on index information of each of the plurality of feature maps.

[0021] According to this aspect, by using the index information, it is possible to appropriately select a feature map.

[0022] In the above aspect, the image decoding device selects the first feature map and the second feature map based on size information of each of the plurality of feature maps.

[0023] According to this aspect, by using size information, it is possible to easily select a feature map.

[0024] In the above aspect, the image decoding device decodes the second feature map by inter prediction using the first feature map.

[0025] According to this aspect, the amount of coding can be reduced by using inter prediction for decoding the feature map.

[0026] In the above aspect, the image decoding device decodes the first feature map and the second feature map by intra prediction.

[0027] According to this aspect, by using intra prediction for decoding the feature maps, it is possible to decode a plurality of feature maps independently.

[0028] In the above aspect, each of the plurality of feature maps includes a plurality of feature images relating to the image.

[0029] According to this aspect, the task processing device can execute task processing using a plurality of feature images included in each feature map, thereby improving the accuracy of task processing.

[0030] In the above aspect, the image decoding device constructs each of the plurality of feature maps by decoding the plurality of feature images and arranging the decoded plurality of feature images in a predetermined scanning order.

[0031] According to this aspect, by arranging a plurality of feature images in a predetermined scanning order, it is possible to appropriately construct a feature map.

[0032] In the above aspect, each of the multiple feature maps includes multiple segments, each of the multiple segments includes the multiple feature images, and the image decoding device constructs each of the multiple segments by arranging the decoded multiple feature images in a predetermined scanning order, and constructs each of the multiple feature maps by arranging the multiple segments in a predetermined order.

[0033] According to this aspect, it is possible to control the process of dividing a stream in units of segments or the decoding process in units of segments, thereby making it possible to realize a flexible system configuration.

[0034] In the above aspect, the image decoding device switches between ascending order and descending order of the predetermined scanning order based on a size of each of the plurality of decoded feature images.

[0035] According to this aspect, by switching the scanning order between ascending and descending order based on the size of each feature image, it is possible to appropriately construct a feature map.

[0036] In the above aspect, the bitstream includes order information that sets ascending or descending order of the predetermined scanning order, and the image decoding device switches between ascending and descending order of the predetermined scanning order based on the order information.

[0037] According to this aspect, by switching the scanning order between ascending order and descending order based on the order information, it is possible to appropriately construct a feature map.

[0038] In the above aspect, the multiple feature images include multiple types of feature images with different sizes, and the image decoding device decodes the multiple feature images with a constant decoding block size corresponding to the smallest size among the multiple sizes of the multiple types of feature images.

[0039] According to this aspect, by decoding a plurality of feature images with a constant decoding block size, it is possible to simplify the device configuration of the image decoding device.

[0040] In the above aspect, the plurality of feature images include a plurality of types of feature images having different sizes, and the image decoding device decodes the plurality of feature images using a plurality of decoding block sizes corresponding to the plurality of sizes of the plurality of types of feature images.

[0041] According to this aspect, by decoding each feature image using a decoding block size corresponding to the size of each feature image, the header required for each decoding block can be reduced, and encoding over a large area becomes possible, thereby improving compression efficiency.

[0042] In the above aspect, the predetermined scan order is a raster scan order.

[0043] According to this aspect, by using the raster scan order, high-speed processing by a GPU or the like is possible.

[0044] In the above aspect, the predetermined scanning order is a Z scanning order.

[0045] According to this embodiment, by using the Z scan order, it becomes possible to support common video codecs.

[0046] In the above aspect, the bit stream includes encoded data of the image, and the image decoding device decodes the image based on the received bit stream, and performs decoding of the multiple feature maps and decoding of the image using a common decoding processing unit.

[0047] According to this aspect, the feature map decoding and the image decoding are performed using a common decoding processing unit, thereby simplifying the device configuration of the image decoding device.

[0048] In the above aspect, the first task process and the second task process include at least one of object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, and hybrid vision.

[0049] According to this aspect, it is possible to improve the accuracy of each of these processes.

[0050] In an image encoding method according to one embodiment of the present disclosure, an image encoding device encodes a first feature map related to an image, encodes a second feature map related to the image, generates a bitstream including encoded data of the first feature map and the second feature map, and transmits the generated bitstream to an image decoding device.

[0051] According to this aspect, the image encoding device transmits a bit stream including encoded data of the first feature map and the second feature map to the image decoding device. This eliminates the need to implement multiple sets of image encoding devices and image decoding devices corresponding to multiple task processing devices implemented on the image decoding device side, thereby simplifying the system configuration.

[0052] An image decoding device according to one embodiment of the present disclosure receives a bit stream including encoded data of a plurality of feature maps related to an image from an image encoding device, decodes the plurality of feature maps based on the received bit stream, selects a first feature map from the decoded plurality of feature maps, outputs the first feature map to a first task processing device that performs a first task processing based on the first feature map, selects a second feature map from the plurality of decoded feature maps, and outputs the second feature map to a second task processing device that performs a second task processing based on the second feature map.

[0053] According to this aspect, the image decoding device selects a first feature map from the multiple decoded feature maps and outputs it to the first task processing device, and selects a second feature map from the multiple decoded feature maps and outputs it to the second task processing device. This makes it possible to simplify the system configuration since it is not necessary to implement multiple sets of image encoding devices and image decoding devices corresponding to each of the multiple task processing devices.

[0054] An image encoding device according to one embodiment of the present disclosure encodes a first feature map related to an image, encodes a second feature map related to the image, generates a bitstream including encoded data of the first feature map and the second feature map, and transmits the generated bitstream to an image decoding device.

[0055] According to this aspect, the image encoding device transmits a bit stream including encoded data of the first feature map and the second feature map to the image decoding device. This eliminates the need to implement multiple sets of image encoding devices and image decoding devices corresponding to multiple task processing devices implemented on the image decoding device side, thereby simplifying the system configuration.

[0056] An image decoding device according to one aspect of the present disclosure constructs a feature map by decoding a plurality of feature images and arranging the plurality of feature images in a raster scan order.

[0057] In the above aspect, the feature map includes a plurality of segments, each of the plurality of segments including the plurality of feature images, each of the plurality of segments is constructed by arranging the plurality of feature images in a raster scan order, and the feature map is constructed by arranging the plurality of segments in raster scan order.

[0058] In the above aspect, it is preferable to further decode setting information indicating a scanning order, and arrange the plurality of feature images in the raster scanning order based on the setting information.

[0059] In the above aspect, the feature map may include a plurality of segments, and the plurality of feature images may be arranged for each of the segments.

[0060] In the above aspect, the plurality of characteristic images may include a plurality of types of characteristic images having different sizes, and the plurality of characteristic images may be decoded using a decoding block size corresponding to the smallest size among the plurality of sizes of the plurality of types of characteristic images.

[0061] In the above aspect, the plurality of characteristic images may include a plurality of types of characteristic images having different sizes, and the plurality of characteristic images may be decoded using a plurality of decoding block sizes corresponding to the plurality of sizes of the plurality of types of characteristic images.

[0062] In the above aspect, it is preferable to execute a neural network task involving estimation based on the feature map, and output a result of the estimation.

[0063] In the above aspect, the neural network task may include at least one of object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, and hybrid vision.

[0064] An image encoding device according to one aspect of the present disclosure generates a feature map based on an input image, sets a scanning order for dividing the feature map into a plurality of feature images, and encodes the feature map and setting information indicating the scanning order, where the setting information indicates a raster scan order as the scanning order.

[0065] In the above aspect, the feature map may include a plurality of segments, each of the plurality of segments including the plurality of feature images, and the plurality of feature images belonging to the same segment may be encoded consecutively.

[0066] In the above aspect, a bitstream may be generated by encoding and arithmetically encoding each of the plurality of feature images with a coding block size.

[0067] In the above aspect, the arithmetic coding may be context-adaptive binary arithmetic coding (CABAC).

[0068] In the above aspect, it is preferable to reconstruct the feature map, execute a neural network task involving estimation based on the feature map, and output a result of the estimation.

[0069] In the above aspect, the neural network task may include at least one of object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, and hybrid vision.

[0070] In an image decoding method according to one aspect of the present disclosure, an image decoding device constructs a feature map by decoding a plurality of feature images and arranging the plurality of feature images in a raster scan order.

[0071] In an image coding method according to one aspect of the present disclosure, an image coding device generates a feature map based on an input image, sets a scanning order for dividing the feature map into a plurality of feature images, and encodes the feature map and setting information indicating the scanning order, wherein the setting information indicates a raster scan order as the scanning order.

[0072] (Embodiments of the present disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In addition, elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0073] Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim showing a top concept are described as optional components. Furthermore, in all embodiments, the contents of each can be combined.

[0074] (First embodiment) 4 is a diagram showing an example of the configuration of an image processing system 1200 according to the first embodiment of the present disclosure. The image processing system 1200 includes an encoding device 1201 as an image encoding device, a decoding device 1202 as an image decoding device, and a plurality of task processing units 1203A to 1203N as task processing devices.

[0075] The encoding device 1201 creates a plurality of feature maps A to N based on an input image or feature. The encoding device 1201 generates a bit stream including encoded data of the feature maps A to N by encoding the created feature maps A to N. The encoding device 1201 transmits the generated bit stream to the decoding device 1202. The decoding device 1202 decodes the feature maps A to N based on the received bit stream. The decoding device 1202 selects a feature map A as a first feature map from the decoded feature maps A to N, and inputs the selected feature map A to the task processing unit 1203A as a first task processing device. The decoding device 1202 also selects a feature map B as a second feature map from the decoded feature maps A to N, and inputs the selected feature map B to the task processing unit 1203B as a second task processing device. The task processing unit 1203A executes a first task processing such as a neural network task based on the input feature map A, and outputs an estimation result. The task processing unit 1203B executes a second task process such as a neural network task based on the input feature map B, and outputs the estimation result.

[0076] 5 is a diagram showing a first example of the configuration of the encoding device 1201 and the decoding device 1202. The encoding device 1201 includes an image encoding unit 1305, a feature extraction unit 1302, a feature conversion unit 1303, a feature encoding unit 1304, and a transmission unit 1306. The decoding device 1202 includes a reception unit 1309, an image decoding unit 1308, and a feature decoding unit 1307.

[0077] Image data is input to the image encoding unit 1305 and the feature extraction unit 1302 from the camera 1301. The image encoding unit 1305 encodes the input image and inputs the encoded data to the transmission unit 1306. The image encoding unit 1305 may directly use a general video codec or a still image codec. The feature extraction unit 1302 extracts a plurality of feature images indicating the features of the input image from the input image and inputs the extracted plurality of feature images to the feature conversion unit 1303. The feature conversion unit 1303 generates a feature map by arranging the plurality of feature images. The feature conversion unit 1303 generates a plurality of feature maps for one input image and inputs the generated plurality of feature maps to the feature encoding unit 1304. The feature encoding unit 1304 encodes the input plurality of feature maps and inputs the encoded data to the transmission unit 1306. The transmitting unit 1306 generates a bit stream including the coded data of the input image and the coded data of the multiple feature maps, and transmits the generated bit stream to the decoding device 1202.

[0078] The receiving unit 1309 receives the bitstream transmitted from the encoding device 1201, and inputs the received bitstream to the image decoding unit 1308 and the feature decoding unit 1307. The image decoding unit 1308 decodes an image based on the input bitstream. The feature decoding unit 1307 decodes a plurality of feature maps based on the input bitstream.

[0079] 5, both the image and the feature map are encoded and decoded, but if image display for human vision is not required, only the feature map may be encoded and decoded. In that case, the image encoding unit 1305 and the image decoding unit 1308 may be omitted.

[0080] Fig. 6 is a diagram showing a second example configuration of the encoding device 1201 and the decoding device 1202. With regard to the encoding device 1201, the feature encoding unit 1304 is omitted from the configuration shown in Fig. 5. With regard to the decoding device 1202, the feature decoding unit 1307 is omitted from the configuration shown in Fig. 5.

[0081] The feature conversion unit 1303 generates a plurality of feature maps for one input image, and inputs the generated plurality of feature maps to the image encoding unit 1305. The image encoding unit 1305 encodes the input image and the plurality of feature maps, and inputs the encoded data of the input image and the plurality of feature maps to the transmission unit 1306. The transmission unit 1306 generates a bit stream including the encoded data of the input image and the plurality of feature maps, and transmits the generated bit stream to the decoding device 1202.

[0082] The receiving unit 1309 receives the bit stream transmitted from the encoding device 1201, and inputs the received bit stream to the image decoding unit 1308. The image decoding unit 1308 decodes an image and multiple feature maps based on the input bit stream. That is, in the configuration shown in Fig. 6, the decoding device 1202 performs image decoding and multiple feature map decoding using the image decoding unit 1308 as a common decoding processing unit.

[0083] Fig. 8 is a block diagram showing a configuration of a video encoder according to the first embodiment of the present disclosure, and Fig. 2 is a flowchart showing a processing procedure 2000 of an image encoding method according to the first embodiment of the present disclosure.

[0084] As shown in FIG. 8, the video encoder includes an encoding device 1201, a decoding unit 2402, a selection unit 2403, and a plurality of task processing units 2404A to 2404N. The selection unit 2403 may be implemented in the decoding unit 2402. The video encoder is configured to create a plurality of feature maps A to N based on an input image or feature, generate a bit stream by encoding the created plurality of feature maps A to N, and transmit the generated bit stream to the decoding device 1202. Furthermore, the video encoder may be configured to decode the plurality of feature maps A to N based on the generated bit stream, input the decoded plurality of feature maps A to N to the task processing units 2404A to 2404N, and output an estimation result by the task processing units 2404A to 2404N executing a neural network task.

[0085] In step S2001 of Fig. 2, an image or a feature is input to the encoding device 1201. The encoding device 1201 creates a plurality of feature maps A to N based on the input image or feature. The encoding device 1201 generates a bit stream including encoded data of the feature maps A to N by encoding the created feature maps A to N in units of blocks. The encoding device 1201 transmits the generated bit stream to the decoding device 1202.

[0086] More specifically, the encoding device 1201 encodes a number of feature maps for an input image, each of which indicates a specific attribute for the image, and each feature map is, for example, arithmetically coded, for example, by context-adaptive binary arithmetic coding (CABAC).

[0087] 9 and 10 are diagrams showing a first example of the creation process of a feature map. The feature map is created using a convolutional neural network having multiple convolution layers, multiple pooling layers, and a fully connected layer. The feature map includes multiple feature images F1 to F108 related to an input image. The resolution of each feature image and the number of feature images may be different for each layer of the neural network. For example, the horizontal size X1 and vertical size X2 of the feature images F1 to F12 in the upper convolution layer X and pooling layer X are larger than the horizontal size Y1 and vertical size Y2 of the feature images F13 to F36 in the lower convolution layer Y and pooling layer Y. In addition, the horizontal size Y1 and vertical size Y2 are larger than the horizontal size Z1 and vertical size Z2 of the feature images F37 to F108 in the fully connected layer.

[0088] For example, the feature images F1 to F108 are arranged according to the order of the hierarchy of the neural network, that is, in ascending order (largest size) or descending order (smallest size) of the hierarchy of the neural network.

[0089] 13 and 14 are diagrams showing a second example of the process of creating a feature map, in which an example of a filter process for extracting features from an input image is shown. The extracted features indicate measurable and characteristic attributes of the input image. As shown in FIGS. 13 and 14, a feature image in which a dot component is extracted, a feature image in which a vertical line component is extracted, or a feature image in which a horizontal line component is extracted can be generated by applying a dot filter, a vertical line filter, or a horizontal line filter of a desired filter size to the input image. A feature map can be generated based on the filter process by arranging the generated feature images.

[0090] With reference to FIG. 8, a bit stream including encoded data of a plurality of feature maps A to N is input to the decoding unit 2402. The decoding unit 2402 decodes an image from the input bit stream as necessary, and outputs an image signal for human vision to a display device. The decoding unit 2402 also decodes the plurality of feature maps A to N from the input bit stream, and inputs the decoded feature maps A to N to the selection unit 2403. The plurality of feature maps A to N of the same time instance can be decoded independently. An example of independent decoding is using intra prediction. Also, the plurality of feature maps A to N of the same time instance can be decoded in a correlated manner. An example of correlated decoding is using inter prediction, and a second feature map can be decoded by inter prediction using a first feature map. The selection unit 2403 selects a desired feature map from the plurality of decoded feature maps A to N, and inputs the selected feature map to each of the task processing units 2404A to 2404N.

[0091] FIG. 17 is a diagram showing an example of using both inter prediction and intra prediction. A plurality of feature maps FM01a to FM01f are generated based on an input image I01, a plurality of feature maps FM02a to FM02f are generated based on an input image I02, and a plurality of feature maps FM03a to FM03f are generated based on an input image I03. In FIG. 17, the feature maps or feature images with hatching are coded by intra prediction, and the feature maps or feature images without hatching are coded by inter prediction. Inter prediction may use other feature maps or feature images corresponding to input images at the same time (same time instance), or may use other feature maps or feature images corresponding to input images at different times (different time instances).

[0092] FIG. 11 is a diagram showing a first example of the operation of the selection unit 2403. The selection unit 2403 selects the feature maps A to N based on the index information IA to IN added to each of the feature maps A to N. The index information IA to IN may be an ID, a category, an expression, or any unique expression that distinguishes each of the multiple feature maps A to N. The selection unit 2403 holds table information indicating the correspondence between the index information IA to IN and the task processing units 2404A to 2404N, and selects the feature maps A to N to be input to each of the task processing units 2404A to 2404N based on the index information IA to IN added to the header or the like of the bit stream constituting each of the feature maps A to N and the table information. The table information may also be described in the header or the like of the bit stream.

[0093] FIG. 12 is a diagram showing a second example of the operation of the selection unit 2403. The selection unit 2403 selects the feature maps A to N based on size information SA to SN such as the resolution or number of feature images of each of the feature maps A to N. The resolution is the number of pixels of the feature map, such as 112×112, 56×56, or 14×14. The number of feature images is the number of multiple feature images included in each feature map. The sizes of the feature maps that can be input to each of the task processing units 2404A to 2404N are different from each other, and the setting information thereof is held by the selection unit 2403. The selection unit 2403 selects the feature maps A to N to be input to each of the task processing units 2404A to 2404N based on the size information SA to SN added to the header or the like of the bit stream constituting each of the feature maps A to N and the setting information. The setting information may also be described in the header or the like of the bit stream.

[0094] The selection unit 2403 may select the feature maps A to N based on a combination of the index information IA to IN and the size information SA to SN.

[0095] 2, the task processing unit 2404A executes a first task process, such as a neural network task involving at least estimation, based on the input feature map A. Examples of the neural network task include object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, machine-human hybrid vision, or any combination thereof.

[0096] FIG. 15 is a diagram showing object detection and object segmentation as examples of neural network tasks. In object detection, attributes of objects included in an input image (television and person in this example) are detected. In addition to the attributes of objects included in the input image, the positions and number of objects in the input image may be detected. This may narrow down the positions of objects to be recognized, or remove objects other than the objects to be recognized. Specific applications include face detection in a camera and detection of pedestrians in autonomous driving. In object segmentation, pixels in an area corresponding to an object are segmented (i.e., divided). This may be used, for example, to separate obstacles from roads in autonomous driving, to assist in safe driving of a car, to detect defects in products in a factory, and to identify terrain in satellite images.

[0097] FIG. 16 is a diagram showing object tracking, action recognition, and pose estimation as examples of neural network tasks. In object tracking, the movement of an object included in an input image is tracked. Possible applications include, for example, counting the number of users of a facility such as a store, and analyzing the movements of athletes. If the processing speed is further increased, it becomes possible to track objects in real time, and it can also be applied to camera processing such as autofocus. In action recognition, the type of object movement (in this example, "riding a bicycle" or "walking") is detected. For example, by using it in a security camera, it can be applied to applications such as preventing and detecting criminal behavior such as robbery and shoplifting, and preventing forgetting of work in a factory. In pose estimation, the posture of an object is detected by detecting key points and joints. For example, it can be used in industrial fields such as improving work efficiency in factories, security fields such as detecting abnormal behavior, and fields such as health care and sports.

[0098] The task processor 2404A outputs a signal indicative of the execution result of the neural network task, which may include at least one of the number of detected objects, the confidence level of the detected objects, the boundary information or position information of the detected objects, and the classification category of the detected objects.

[0099] 2, the task processor 2404B executes a second task process, such as a neural network task involving at least estimation, based on the input feature map B. As with the first task process, examples of the neural network task include object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, machine-human hybrid vision, or any combination thereof. The task processor 2404B outputs a signal indicating the execution result of the neural network task.

[0100] In the configuration shown in Fig. 8, the decoder 2402, the selector 2403, and the task processors 2404A-2404N are provided to enable the execution of neural network tasks to output estimation results, but if there is no need to execute neural network tasks in the video decoder, the decoder 2402, the selector 2403, and the task processors 2404A-2404N may be omitted. Similarly, in the processing procedure 2000 shown in Fig. 2, if there is no need to execute neural network tasks, steps S2002 and S2003 may be omitted.

[0101] Fig. 7 is a block diagram showing the configuration of a video decoder according to the first embodiment of the present disclosure, and Fig. 1 is a flowchart showing a processing procedure 1000 of an image decoding method according to the first embodiment of the present disclosure.

[0102] 7, the video decoder includes a decoding device 1202, a selection unit 1400, and a plurality of task processing units 1203A to 1203N. The selection unit 1400 may be implemented in the decoding device 1202. The video decoder is configured to decode a plurality of feature maps A to N based on a received bitstream, input the decoded plurality of feature maps A to N to the task processing units 1203A to 1203N, and output an estimation result by the task processing units 1203A to 1203N executing a neural network task.

[0103] A bit stream including encoded data of a plurality of feature maps A to N is input to the decoding device 1202. The decoding device 1202 decodes an image from the input bit stream as necessary, and outputs an image signal for human vision to a display device. The decoding device 1202 also decodes the plurality of feature maps A to N from the input bit stream, and inputs the decoded feature maps A to N to the selection unit 1400. The plurality of feature maps A to N of the same time instance can be decoded independently. An example of independent decoding is using intra prediction. The plurality of feature maps A to N of the same time instance can be decoded in a correlated manner. An example of correlated decoding is using inter prediction, and a second feature map can be decoded by inter prediction using a first feature map. The selection unit 1400 selects a desired feature map from the plurality of decoded feature maps A to N, and inputs the selected feature map to each of the task processing units 1203A to 1203N.

[0104] FIG. 17 is a diagram showing an example of using both inter prediction and intra prediction. A plurality of feature maps FM01a to FM01f are generated based on an input image I01, a plurality of feature maps FM02a to FM02f are generated based on an input image I02, and a plurality of feature maps FM03a to FM03f are generated based on an input image I03. In FIG. 17, the feature maps or feature images with hatching are coded by intra prediction, and the feature maps or feature images without hatching are coded by inter prediction. Inter prediction may use other feature maps or feature images corresponding to input images at the same time (same time instance), or may use other feature maps or feature images corresponding to input images at different times (different time instances).

[0105] FIG. 11 is a diagram showing a first example of the operation of the selection unit 1400. The selection unit 1400 selects the feature maps A to N based on the index information IA to IN added to each of the feature maps A to N. The index information IA to IN may be an ID, a category, an expression, or any unique expression that distinguishes each of the multiple feature maps A to N. The selection unit 1400 holds table information indicating the correspondence between the index information IA to IN and the task processing units 1203A to 1203N, and selects the feature maps A to N to be input to each of the task processing units 1203A to 1203N based on the index information IA to IN added to the header or the like of the bit stream constituting each of the feature maps A to N and the table information. The table information may also be described in the header or the like of the bit stream.

[0106] FIG. 12 is a diagram showing a second example of the operation of the selection unit 1400. The selection unit 1400 selects the feature maps A to N based on size information SA to SN such as the resolution or number of feature images of each of the feature maps A to N. The resolution is the number of pixels of the feature map, such as 112×112, 56×56, or 14×14. The number of feature images is the number of multiple feature images included in each feature map. The feature maps that can be input to each of the task processing units 1203A to 1203N have different sizes, and the selection unit 1400 holds the setting information. The selection unit 1400 selects the feature maps A to N to be input to each of the task processing units 1203A to 1203N based on the size information SA to SN added to the header of the bit stream constituting each of the feature maps A to N and the setting information. The setting information may also be described in the header of the bit stream.

[0107] The selection unit 1400 may select the feature maps A to N based on a combination of the index information IA to IN and the size information SA to SN.

[0108] In step S1002 in Fig. 1, the task processing unit 1203A executes a first task process such as a neural network task involving at least estimation based on the input feature map A. An example of the neural network task is object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, machine-human hybrid vision, or any combination thereof. An example of the neural network task is similar to Fig. 15 and Fig. 16.

[0109] The task processor 1203A outputs a signal indicating the execution result of the neural network task, which may include at least one of the number of detected objects, the confidence level of the detected objects, the boundary information or position information of the detected objects, and the classification category of the detected objects.

[0110] 1, the task processor 1203B executes a second task process, such as a neural network task involving at least estimation, based on the input feature map B. As with the first task process, examples of the neural network task include object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, machine-human hybrid vision, or any combination thereof. The task processor 1203B outputs a signal indicating the execution result of the neural network task.

[0111] According to this embodiment, the encoding device 1201 transmits a bit stream including encoded data of a first feature map A and a second feature map B to the decoding device 1202. Furthermore, the decoding device 1202 selects the first feature map A from the multiple decoded feature maps A to N and outputs it to the first task processing unit 1203A, and selects the second feature map B from the multiple decoded feature maps A to N and outputs it to the second task processing unit 1203B. This makes it unnecessary to implement multiple sets of encoding devices and decoding devices corresponding to each of the multiple task processing units 1203A to 1203N, thereby simplifying the system configuration.

[0112] Second embodiment Generally, video codecs have memory capacity limitations, so image encoding is often performed in Z scan order. However, when building a system using a GPU with a large memory capacity, it is possible to perform faster processing by continuously loading input images or features into the GPU memory using raster scan order rather than Z scan order. Therefore, in this embodiment, a system capable of switching between a general Z scan order and a high-speed raster scan order in a process of arranging multiple feature images in a predetermined scan order to build a feature map will be described. This embodiment is applicable to an image processing system having at least one task processing unit.

[0113] 20 is a diagram showing a configuration example of an image processing system 2100 according to the second embodiment of the present disclosure. The image processing system 2100 includes an encoding device 2101 as an image encoding device, a decoding device 2102 as an image decoding device, and a task processing unit 2103 as a task processing device. As in the first embodiment, a plurality of task processing units 2103 may be provided.

[0114] The encoding device 2101 creates a feature map based on an input image or feature. The encoding device 2101 generates a bit stream including encoded data of the feature map by encoding the created feature map. The encoding device 2101 transmits the generated bit stream to the decoding device 2102. The decoding device 2102 decodes the feature map based on the received bit stream. The decoding device 2102 inputs the decoded feature map to the task processing unit 2103. The task processing unit 2103 executes a predetermined task processing such as a neural network task based on the input feature map, and outputs an estimation result.

[0115] Fig. 22 is a block diagram showing a configuration of an encoding device 2101 according to the second embodiment of the present disclosure. Also, Fig. 19 is a flowchart showing a processing procedure 4000 of an image encoding method according to the second embodiment of the present disclosure.

[0116] 22, the encoding device 2101 includes a scanning order setting unit 3201, a scanning unit 3202, and an entropy encoding unit 3203. The encoding device 2101 may also include a reconstruction unit 3204 and a task processing unit 3205.

[0117] A feature map is input to the scanning order setting unit 3201. As shown in Fig. 10, the feature map is constructed by arranging a plurality of feature images F1 to F108 in a predetermined scanning order.

[0118] 23 is a diagram showing another example of the feature map. The feature map includes a plurality of feature images F1 to F36 related to the input image. The resolution of each feature image and the number of feature images may be the same for all layers of the neural network. All of the feature images F1 to F36 have the same horizontal size X1 and vertical size X2.

[0119] 19, the scanning order setting unit 3201 sets a scanning order for dividing a feature map into multiple feature images in accordance with a rule determined in advance between the encoding device 2101 and the decoding device 2102. Note that the scanning order setting unit 3201 may arbitrarily set a scanning order for dividing a feature map into multiple feature images, and add setting information indicating the scanning order to a header of a bitstream to transmit the bitstream to the decoding device 2102. In this case, the decoding device 2102 can construct a feature map by arranging the multiple decoded feature images in the scanning order indicated by the setting information.

[0120] 26 is a diagram showing a first example of a scanning order. The scanning order setting unit 3201 sets the raster scanning order as the scanning order.

[0121] 27 is a diagram showing a second example of the scanning order. The scanning order setting unit 3201 sets the Z scanning order as the scanning order.

[0122] The scanning unit 3202 divides the feature map into a plurality of segments in the scanning order set by the scanning order setting unit 3201, and divides each segment into a plurality of feature images.

[0123] 28 to 30 are diagrams showing examples of division into a plurality of segments. In the example shown in FIG. 28, the feature map is divided into three segments SG1 to SG3. In the example shown in FIG. 29, the feature map is divided into seven segments SG1 to SG7. In the example shown in FIG. 30, the feature map is divided into six segments SG1 to SG6. Feature images are scanned for each segment, and multiple feature images belonging to the same segment are always coded consecutively in the bit stream. Each segment may be a unit that can be coded and decoded independently, for example, called a slice.

[0124] In the example shown in FIG. 22, scanning order setting section 3201 and scanning section 3202 are configured as separate processing blocks, but they may be configured to perform processing collectively as a single processing block.

[0125] The scanning unit 3202 sequentially inputs the divided feature images to the entropy coding unit 3203. The entropy coding unit 3203 generates a bitstream by coding and arithmetically coding each feature image with a coding block size. The arithmetic coding is, for example, context-adaptive binary arithmetic coding (CABAC). The coding device 2101 transmits the bitstream generated by the entropy coding unit 3203 to the decoding device 2102.

[0126] 24 and 25 are diagrams showing the relationship between the size of a feature image and the size of a coding block. A feature map is constructed from a plurality of types of feature images with different sizes.

[0127] As shown in Fig. 24, the entropy encoding unit 3203 encodes a plurality of feature images with a fixed encoding block size corresponding to the smallest feature image size among a plurality of sizes of a plurality of types of feature images (hereinafter referred to as "feature image sizes"). Alternatively, as shown in Fig. 25, the entropy encoding unit 3203 may encode a plurality of feature images with a plurality of encoding block sizes corresponding to the plurality of feature image sizes.

[0128] Figures 31 and 32 are diagrams showing scanning orders when one feature image is divided into multiple coding blocks and coded. The entropy coding unit 3203 may perform coding in a raster scan order for each feature image as shown in Figure 31, or may perform coding in a raster scan order for each row of coding blocks across multiple feature images as shown in Figure 32.

[0129] Furthermore, the encoding device 2101 may be configured to reconstruct the divided feature map, input the reconstructed feature map to the task processing unit 3205, and output an estimation result by the task processing unit 3205 executing a neural network task.

[0130] 19, the reconstruction unit 3204 receives a plurality of feature images divided into a plurality of segments from the scanning unit 3202. The reconstruction unit 3204 reconstructs each of the plurality of segments by arranging the input plurality of feature images in a predetermined scanning order, and reconstructs a feature map by arranging the plurality of segments in a predetermined order. Note that the reconstruction unit 3204 may be configured to receive the output of the entropy encoding unit 3203 as input and execute a process similar to that executed by the decoding device 2102, in order to reconstruct a feature map identical to the feature map generated by the decoding device 2102.

[0131] For example, the feature images are arranged according to the order of the layers of the neural network, that is, in ascending order (largest size) or descending order (smallest size) of the layers of the neural network.

[0132] The scanning order setting unit 3201 sets the ascending or descending order of the scanning order based on the size of each of the input feature images, and the reconstruction unit 3204 switches between ascending and descending orders according to the scanning order set by the scanning order setting unit 3201. For example, the reconstruction unit 3204 switches to ascending order when the multiple feature images are input in descending order of size, and switches to descending order when the multiple feature images are input in descending order of size. Alternatively, order information for setting a predetermined ascending or descending order of the scanning order may be added to a header of a bit stream or the like, and the reconstruction unit 3204 may switch between ascending and descending order of the scanning order based on the order information. The reconstruction unit 3204 inputs a feature map reconstructed by arranging the multiple feature images in a predetermined scanning order to the task processing unit 3205.

[0133] 19, the task processing unit 3205 executes a predetermined task process, such as a neural network task involving at least estimation, based on the input feature map. Examples of neural network tasks include object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, machine-human hybrid vision, or any combination thereof.

[0134] The task processor 3205 outputs a signal indicating the execution result of the neural network task, which may include at least one of the number of detected objects, the confidence level of the detected objects, boundary information or position information of the detected objects, and the classification category of the detected objects.

[0135] In the configuration shown in Fig. 22, the reconstruction unit 3204 and the task processing unit 3205 are provided, which makes it possible to output an estimation result by executing a neural network task, but if there is no need to execute a neural network task in the video encoder, the configuration may omit the reconstruction unit 3204 and the task processing unit 3205. Similarly, in the processing procedure 4000 shown in Fig. 19, if there is no need to execute a neural network task, the configuration may omit steps S4002 and S4003.

[0136] Fig. 21 is a block diagram showing a configuration of a decoding device 2102 according to the second embodiment of the present disclosure. Also, Fig. 18 is a flowchart showing a processing procedure 3000 of an image decoding method according to the second embodiment of the present disclosure.

[0137] As shown in FIG. 21, the decoding device 2102 includes an entropy decoding unit 2201, a scanning order setting unit 2202, and a scanning unit 2203.

[0138] In step S3001 in FIG. 18, the entropy decoding unit 2201 decodes a plurality of feature images from the bit stream received from the encoding device 2101 in units of decoding blocks.

[0139] 24 and 25 are diagrams showing the relationship between the size of a feature image and the size of a decoding block. A feature map is constructed from a plurality of types of feature images with different sizes.

[0140] As shown in Fig. 24, the entropy decoding unit 2201 decodes multiple feature images with a fixed decoding block size corresponding to the smallest feature image size among multiple feature image sizes of multiple types of feature images. Alternatively, as shown in Fig. 25, the entropy decoding unit 2201 may decode multiple feature images with multiple decoding block sizes corresponding to the multiple feature image sizes.

[0141] Figures 31 and 32 are diagrams showing scanning orders when one feature image is divided into multiple coding blocks and coded. The entropy decoding unit 2201 may perform decoding in a raster scan order for each feature image as shown in Figure 31, or may perform decoding in a raster scan order for each row of coding blocks across multiple feature images as shown in Figure 32.

[0142] The scanning order setting unit 2202 receives as input a plurality of decoded blocks or a plurality of feature images from the entropy decoding unit 2201 .

[0143] 18, the scanning order setting unit 2202 sets a scanning order for constructing a feature map from multiple feature images, in accordance with a rule determined in advance between the encoding device 2101 and the decoding device 2102. Note that, when the above-mentioned setting information indicating an arbitrary scanning order is added to the header of the bitstream, the decoding device 2102 can construct a feature map by arranging the multiple decoded feature images in the scanning order indicated by the setting information.

[0144] 26 is a diagram showing a first example of the scanning order. The scanning order setting unit 2202 sets the raster scanning order as the scanning order.

[0145] 27 is a diagram showing a second example of the scanning order. The scanning order setting unit 2202 sets the Z scanning order as the scanning order.

[0146] A plurality of feature images divided into a plurality of segments are input to the scanning unit 2203. The scanning unit 2203 arranges the plurality of feature images in the scanning order set by the scanning order setting unit 2202, thereby constructing a feature map.

[0147] For example, the feature images are arranged according to the order of the layers of the neural network, that is, in ascending order (largest size) or descending order (smallest size) of the layers of the neural network.

[0148] The scanning order setting unit 2202 sets the scanning order in ascending or descending order based on the size of each of the input feature images, and the scanning unit 2203 switches between ascending and descending order according to the scanning order set by the scanning order setting unit 2202. For example, the scanning unit 2203 switches between ascending and descending order when the input feature images are in descending order of size, and switches between ascending and descending order when the input feature images are in descending order of size. Alternatively, order information for setting a predetermined ascending or descending order of the scanning order may be decoded from a header of a bit stream or the like, and the scanning unit 2203 may switch between ascending and descending order of the scanning order based on the order information. The scanning unit 2203 inputs a feature map constructed by arranging the plurality of feature images in a predetermined scanning order to the task processing unit 2103.

[0149] In the example shown in FIG. 21, the scanning sequence setting unit 2202 and the scanning unit 2203 are configured as separate processing blocks, but they may be configured to perform processing collectively as a single processing block.

[0150] 18, the task processing unit 2103 executes a predetermined task process, such as a neural network task involving at least estimation, based on the input feature map. Examples of the neural network task include object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, machine-human hybrid vision, or any combination thereof.

[0151] The task processor 2103 outputs a signal indicating the execution result of the neural network task, which may include at least one of the number of detected objects, the confidence level of the detected objects, boundary information or position information of the detected objects, and the classification category of the detected objects.

[0152] According to this embodiment, a feature map can be appropriately constructed by arranging a plurality of feature images in a predetermined scanning order. [Industrial Applicability]

[0153] The present disclosure is particularly useful when applied to an image processing system that includes an encoder for transmitting images and a decoder for receiving images.

Claims

1. Decoding the plurality of feature images; constructing a feature map by arranging the plurality of feature images in a raster scan order; Image decoding device.

2. the feature map includes a plurality of segments; each of the plurality of segments includes the plurality of feature images; constructing each of the plurality of segments by arranging the plurality of feature images in a raster scan order; constructing the feature map by arranging the plurality of segments in a raster scan order; The image decoding device according to claim 1 .

3. Further decoding configuration information indicative of a scanning order; arranging the plurality of feature images in the raster scan order based on the setting information; The image decoding device according to claim 1 .

4. the feature map includes a plurality of segments; Arranging the plurality of feature images for each of the segments; The image decoding device according to claim 1 .

5. the plurality of characteristic images include a plurality of types of characteristic images having different sizes; decoding the plurality of characteristic images with a decoding block size corresponding to a smallest size among a plurality of sizes of the plurality of types of characteristic images; The image decoding device according to claim 1 .

6. the plurality of characteristic images include a plurality of types of characteristic images having different sizes; decoding the plurality of feature images with a plurality of decoding block sizes corresponding to a plurality of sizes of the plurality of types of feature images; The image decoding device according to claim 1 .

7. performing a neural network task involving estimation based on the feature map; outputting the result of said estimation; The image decoding device according to claim 1 .

8. The neural network task includes at least one of object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, and hybrid vision. The image decoding device according to claim 7.

9. Generate a feature map based on the input image; determining a scanning order for dividing the feature map into a plurality of feature images; encoding the feature map and configuration information indicating the scanning order; The setting information indicates a raster scan order as the scan order. Image encoding device.

10. the feature map includes a plurality of segments; each of the plurality of segments includes the plurality of feature images; encoding the plurality of feature images belonging to the same segment consecutively; The image encoding device according to claim 9.

11. generating a bitstream by encoding and arithmetically encoding each of the plurality of feature images with a coding block size; The image encoding device according to claim 9.

12. The arithmetic coding is a context-adaptive binary arithmetic coding (CABAC). An image encoding device as described in claim 11.

13. Reconstructing the feature map; performing a neural network task involving estimation based on the feature map; outputting the result of said estimation; The image encoding device according to claim 9.

14. The neural network task includes at least one of object detection, object segmentation, object tracking, action recognition, pose estimation, pose tracking, and hybrid vision. The image encoding device according to claim 13.

15. An image decoding device, Decoding the plurality of feature images; constructing a feature map by arranging the plurality of feature images in a raster scan order; Image decoding method.

16. An image encoding device, Generate a feature map based on the input image; determining a scanning order for dividing the feature map into a plurality of feature images; encoding the feature map and configuration information indicating the scanning order; The setting information indicates a raster scan order as the scan order. Image encoding method.

Images