Decoding device, encoding device, decoding method, and encoding method

Abstract

This decoding device is provided with: a circuit that generates an image by decoding a bit stream, generates a plurality of intermediate feature maps having a uniform size on the basis of the image, and generates a plurality of intermediate feature maps having a uniform size on the basis of the plurality of intermediate feature maps; in the present invention, a plurality of feature maps having different sizes for each of a plurality of levels included in a machine task-oriented neural network are generated, and in the generation of the feature maps, either one of a first method in which the feature maps are selected for at least one of the plurality of levels or a second method in which the feature maps are selected for at least one of the plurality of levels, and the first method is selected for at least one of the plurality of levels, and the second method is selected for at least one of the plurality of levels. In the first method, the plurality of feature maps for the hierarchy are generated using the plurality of intermediate feature maps, and in the second method, the plurality of feature maps for the hierarchy are generated using the plurality of feature maps generated in a hierarchy different from the hierarchy among the plurality of hierarchy without using the plurality of intermediate feature maps.

Description

Technical Field

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method. Background Art

[0002] Faster-RCNN consists of a first neural network (Feature Pyramid Network) that generates multiple feature maps, and a second neural network (Region Proposal Network) that extracts ROI (Region Of Interest) areas from the feature maps.

[0003] Patent Documents 1 and 2 disclose object detection methods using Faster-RCNN.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Chinese Patent Application Publication No. 201811150901

[0007] Patent Document 2: Chinese Patent Application Publication No. 201811509164 Summary of the invention

[0008] The purpose of the present disclosure is to reduce the amount of data of a bit stream sent from an encoding device having a first neural network to a decoding device having a second neural network.

[0009] A decoding device involved in one embodiment of the present invention comprises: a circuit; and a memory connected to the circuit, wherein the circuit performs the following processing: generating an image by decoding a bit stream, generating a plurality of intermediate feature maps having uniform sizes based on the image, generating a plurality of feature maps having different sizes for each level in each of a plurality of levels of a neural network for machine tasks based on the plurality of intermediate feature maps, in generating the feature maps, selecting either a first method or a second method for at least one level of the plurality of levels, wherein in the first method, the plurality of intermediate feature maps are used to generate the plurality of feature maps of the level, and in the second method, the plurality of feature maps of the level are generated using the plurality of feature maps generated in a level different from the level among the plurality of levels without using the plurality of intermediate feature maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a diagram schematically showing the configuration of an image processing system according to an embodiment of the present disclosure.

[0011] Figure 2A It is a diagram showing a first configuration example of the information processing unit.

[0012] Figure 2B It is a diagram showing a second configuration example of the information processing unit.

[0013] Figure 3 It is a flowchart showing the flow of processing executed by the information processing unit.

[0014] Figure 4 It is a flowchart showing the details of the generation process of the intermediate feature map.

[0015] Figure 5 is a diagram showing a feature map.

[0016] Figure 6 is a diagram showing an intermediate feature map.

[0017] Figure 7 is a diagram showing an intermediate feature map.

[0018] Figure 8 It is a diagram showing the image generation processing by the image generation unit.

[0019] Fig. 9 This is a diagram showing an example of the data structure of a bit stream.

[0020] Fig. 10A This is a diagram showing a first example of the data structure of parameters.

[0021] Fig. 10B This is a diagram showing a first example of the data structure of parameters.

[0022] Fig.11A This is a diagram showing a second example of the parameter data structure.

[0023] Fig. 11B This is a diagram showing a second example of the parameter data structure.

[0024] Fig. 12A It is a diagram showing a third example of the data structure of parameters.

[0025] Fig. 12B It is a diagram showing a third example of the data structure of parameters.

[0026] Fig.13A It is a diagram showing a first configuration example of the information processing unit.

[0027] Fig. 13B It is a diagram showing a second configuration example of the information processing unit.

[0028] Fig.14 It is a flowchart showing the flow of processing executed by the information processing unit.

[0029] Fig.15 It is a flowchart showing the details of the feature map generation process.

[0030] Fig.16A It is a diagram showing a third configuration example of the information processing unit.

[0031] Fig. 16B It is a diagram showing a fourth configuration example of the information processing unit.

[0032] Fig.17 It is a diagram showing a configuration of an information processing unit according to a modification example.

[0033] Fig.18 It is a diagram showing a configuration of an information processing unit according to a modification example. DETAILED DESCRIPTION

[0034] (Knowledge that forms the basis of this disclosure)

[0035] Faster-RCNN is a model that speeds up the region-based object detection model, namely R-CNN (Region based Convolutional Neural Network). In Faster-RCNN, a first neural network (Feature Pyramid Network) with multiple layers is used to perform convolution processing on the input image of the processing object, thereby generating multiple feature maps of different sizes for each layer. In addition, by applying the RP model using the second neural network (Region Proposal Network) to the generated feature map, the ROI area is extracted from the feature map, and image recognition is performed on the extracted ROI area.

[0036] For example, in a surveillance camera system, when the camera side performs processing using the first neural network and the server side performs processing using the second neural network, the encoding device generates a bit stream by encoding the feature map generated using the first neural network, and sends the generated bit stream to the decoding device. The decoding device reconstructs the feature map by decoding the received bit stream, and performs processing using the second neural network on the reconstructed feature map.

[0037] However, the data volume of the feature map is huge compared to the data volume of the input image to be processed, so there is a problem that the data volume of the bit stream sent from the encoding device to the decoding device also increases.

[0038] In order to solve this problem, the inventors have come to the following understanding, which led to the present disclosure: focusing on the fact that multiple feature maps generated in multiple layers of the first neural network have strong correlations and include a large amount of redundancy, the encoding of some feature maps can be omitted during encoding based on the content of the image or machine task, thereby solving the above-mentioned problem.

[0039] Next, each aspect of the present disclosure will be described.

[0040] The decoding device involved in the first embodiment of the present invention comprises: a circuit; and a memory connected to the circuit, wherein the circuit performs the following processing: generating an image by decoding a bit stream, generating a plurality of intermediate feature maps with uniform sizes based on the image, generating a plurality of feature maps with different sizes for each level in each of a plurality of levels of a neural network for machine tasks based on the plurality of intermediate feature maps, in generating the feature maps, selecting either the first method or the second method for at least one level of the plurality of levels, wherein in the first method, the plurality of intermediate feature maps are used to generate the plurality of feature maps of the level, and in the second method, the plurality of feature maps of the level are generated using the plurality of feature maps generated in a level different from the level among the plurality of levels without using the plurality of intermediate feature maps.

[0041] According to the first method, for at least one layer of the multiple layers of the neural network, the first method is selected in the decoding device when the encoding of the feature map is not omitted in the encoding device, and the second method is selected in the decoding device when the encoding of the feature map is omitted in the encoding device, thereby enabling the feature map of the layer to be properly reconstructed in the decoding device.

[0042] The decoding device involved in the second embodiment of the present disclosure may be, in the first embodiment, further, by decoding the bit stream to obtain a parameter indicating which one of the first method and the second method is selected for the at least one layer, and in generating the feature map, selects either the first method or the second method for the at least one layer based on the parameter.

[0043] According to the second aspect, by transmitting parameters from the encoding device to the decoding device, it is possible to dynamically switch between the first method and the second method for selection for each of a plurality of layers.

[0044] The decoding device involved in the third embodiment of the present disclosure may be, in the second embodiment, capable of selecting either the first method or the second method for each of the multiple layers, and the parameter includes multiple flags corresponding to the multiple layers, and the value of each flag of the multiple flags indicates which of the first method and the second method is selected for the corresponding layer.

[0045] According to the third aspect, by referring to the value of each of the plurality of flags included in the parameter, it is possible to appropriately select the first method or the second method for each hierarchy.

[0046] The decoding device involved in the fourth embodiment of the present disclosure may be, in the second embodiment, capable of selecting either the first method or the second method for each of the multiple layers, and the parameter includes multiple bits of data corresponding to the multiple layers, and the value of each bit of the multiple bits indicates which of the first method and the second method is selected for the corresponding layer.

[0047] According to the fourth aspect, by referring to the value of each of the plurality of bits included in the parameter, it is possible to appropriately select the first method or the second method for each layer.

[0048] The decoding device involved in the 5th mode of the present disclosure may be, in the 2nd mode, capable of selecting any one of the first method and the second method for each of the multiple layers, and a plurality of modes are pre-specified regarding the combination of which one of the first method and the second method to select for each of the multiple layers, and a mode is selected from the plurality of modes, and the parameters include information for determining the one mode.

[0049] According to the fifth aspect, by referring to the information identifying one mode included in the parameter, it is possible to appropriately select the first method or the second method for each hierarchy.

[0050] The decoding device according to the sixth aspect of the present disclosure, in any one of the second to fifth aspects, may acquire the parameter by decoding a supplemental enhancement information (SEI) area of ​​the bitstream.

[0051] According to the sixth aspect, by decoding the SEI area of ​​the bit stream, it is possible to easily acquire parameters.

[0052] The decoding device involved in the 7th mode of the present disclosure may be, in any one of the 1st to 6th modes, that, when the 1st method is selected for the at least one level, generates the multiple feature maps of the level using the multiple intermediate feature maps and the multiple feature maps generated in a level different from the level among the multiple levels.

[0053] According to the seventh aspect, when the first method is selected for at least one level, multiple feature maps of the level can be appropriately generated by using multiple intermediate feature maps and multiple feature maps generated in a level different from the level among the multiple levels.

[0054] The decoding device involved in the 8th mode of the present disclosure may be, in any one of the 1st to 7th modes, that the size of the intermediate feature map is equal to the size of the smallest feature map among the multiple feature maps generated in the multiple levels, and when the first method is selected for the at least one level, in the generation of the feature map, the size of the intermediate feature map is enlarged to the size of the feature map generated in the at least one level.

[0055] According to the eighth aspect, when the first method is selected for at least one level, a plurality of feature maps of the level can be appropriately generated by increasing the size of the intermediate feature map.

[0056] The decoding device involved in the 9th mode of the present disclosure may be, in any one of the 1st to 7th modes, that the size of the intermediate feature map is equal to the size of the feature map with the largest size among the multiple feature maps generated in the multiple levels, and when the first method is selected for the at least one level, in the generation of the feature map, the size of the intermediate feature map is reduced to the size of the feature map generated in the at least one level.

[0057] According to the ninth aspect, when the first method is selected for at least one level, multiple feature maps of the level can be appropriately generated by reducing the size of the intermediate feature map.

[0058] The decoding device involved in the tenth embodiment of the present disclosure may be, in any one of the first to ninth embodiments, that, when the first method is selected for the at least one level, generates the feature maps by performing convolution processing on the plurality of intermediate feature maps to generate a plurality of feature maps whose number is greater than the number of the plurality of intermediate feature maps.

[0059] According to the tenth embodiment, when the first method is selected for at least one level, a plurality of feature maps whose number is greater than the number of the plurality of intermediate feature maps can be appropriately generated by performing convolution processing on the plurality of intermediate feature maps.

[0060] The decoding device involved in the 11th mode of the present disclosure may be, in any one of the 1st to 9th modes, that, when the first method is selected for the at least one level, in the generation of the feature map, generates the plurality of feature maps whose number is less than the number of the plurality of intermediate feature maps by extracting two or more intermediate feature maps corresponding to the at least one level from the plurality of intermediate feature maps.

[0061] According to the 11th method, a plurality of intermediate feature maps including two or more intermediate feature maps corresponding to each level of a plurality of levels are sent from an encoding device to a decoding device, and in the decoding device, two or more intermediate feature maps corresponding to each level are extracted from the plurality of intermediate feature maps, thereby being able to appropriately generate feature maps corresponding to each level based on the two or more intermediate feature maps.

[0062] The decoding device involved in the 12th mode of the present disclosure may be, in any one of the 1st to 11th modes, capable of selecting either the first method or the second method for each of the multiple levels, and the greater the number of levels that select the second method among the multiple levels, the smaller the number of the multiple intermediate feature maps.

[0063] According to the twelfth method, since the number of layers that select the second method among multiple layers is larger, the number of multiple intermediate feature maps is smaller, and thus the amount of data of the bit stream sent from the encoding device to the decoding device can be reduced.

[0064] The decoding device involved in the 13th mode of the present disclosure may be, in any one of the 1st to 12th modes, capable of selecting either the 1st method or the 2nd method for each of the multiple levels, and in the generation of the feature map, when the size of the target of the detection object in the machine task is smaller than a threshold value, or when local information is required in the machine task, the 2nd method is selected at least for the level with the smallest size of the feature map generated among the multiple levels.

[0065] According to the 13th method, when the size of the target of the detection object in the machine task is smaller than a threshold value, or when local information is required in the machine task, the second method is selected at least for the layer with the smallest size of the feature map generated among multiple layers, thereby reducing the amount of data in the bit stream sent from the encoding device to the decoding device while retaining the information required for the machine task.

[0066] The decoding device involved in the 14th mode of the present disclosure may be, in any one of the 1st to 12th modes, capable of selecting either the 1st method or the 2nd method for each of the multiple levels, and in the generation of the feature map, when the size of the target of the detection object in the machine task is above a threshold, or when global information is required in the machine task, the 2nd method is selected at least for the level with the largest size of the feature map generated among the multiple levels.

[0067] According to the 14th method, when the size of the target of the detection object in the machine task is above a threshold, or when global information is required in the machine task, the second method is selected at least for the layer with the largest size of the feature map generated among multiple layers, thereby reducing the amount of data in the bit stream sent from the encoding device to the decoding device while retaining the information required for the machine task.

[0068] The encoding device involved in the 15th mode of the present disclosure comprises: a circuit; and a memory connected to the circuit, wherein the circuit performs the following processing: based on an input image of a processing object, generating a plurality of feature maps of different sizes for each layer in each of a plurality of layers of a neural network for machine tasks, generating a plurality of intermediate feature maps of uniform sizes based on the plurality of feature maps generated in the plurality of layers, generating an image based on the plurality of intermediate feature maps, generating a bit stream by encoding the image, and in generating the intermediate feature maps, selecting either the first method or the second method for at least one layer of the plurality of layers, wherein in the first method, the plurality of intermediate feature maps are generated using the plurality of feature maps generated in the layer, and in the second method, the plurality of intermediate feature maps are generated without using the plurality of feature maps generated in the layer.

[0069] According to the fifteenth aspect, for at least one of the multiple layers of the neural network, either the first method or the second method is selected, wherein the first method generates multiple intermediate feature maps using multiple feature maps generated in the layer, and the second method generates multiple intermediate feature maps without using multiple feature maps generated in the layer. When the second method is selected for the layer, encoding of the multiple intermediate feature maps for the layer can be omitted, and thus the amount of data of the bit stream sent from the encoding device to the decoding device can be reduced.

[0070] The encoding device involved in the 16th mode of the present disclosure may be, in the 15th mode, generating the bit stream by further encoding a parameter indicating which of the first method and the second method is selected for the at least one layer.

[0071] According to the sixteenth aspect, by transmitting parameters from the encoding device to the decoding device, it is possible to dynamically switch between the first method and the second method for selection for each of a plurality of layers.

[0072] The encoding device involved in the 17th mode of the present disclosure may be, in the 16th mode, capable of selecting either the first method or the second method for each of the multiple levels, and the parameter includes a plurality of flags corresponding to the multiple levels, and the value of each flag of the plurality of flags indicates which of the first method and the second method is selected for the corresponding level.

[0073] According to the seventeenth aspect, it is possible to clearly indicate which of the first method and the second method is selected for each layer by the value of each of the plurality of flags included in the parameter.

[0074] The encoding device involved in the 18th mode of the present disclosure may be, in the 16th mode, capable of selecting either the first method or the second method for each of the multiple levels, and the parameter includes multiple bits of data corresponding to the multiple levels, and the value of each bit of the multiple bits indicates which of the first method and the second method is selected for the corresponding level.

[0075] According to the eighteenth aspect, it is possible to clearly indicate which of the first method and the second method is selected for each layer by the value of each of the plurality of bits included in the parameter.

[0076] The encoding device involved in the 19th embodiment of the present disclosure may be, in the 16th embodiment, capable of selecting any one of the first method and the second method for each of the multiple layers, and a plurality of modes are pre-specified regarding the combination of which one of the first method and the second method to select for each of the multiple layers, and a mode is selected from the plurality of modes, and the parameters include information for determining the one mode.

[0077] According to the nineteenth aspect, it is possible to clearly indicate which of the first method and the second method is selected for each layer by using information that specifies one mode included in the parameter.

[0078] The encoding device according to the 20th aspect of the present disclosure, in any one of the 16th to 19th aspects, may be configured to store the parameter in a supplemental enhancement information (SEI) area of ​​the bitstream during the generation of the bitstream.

[0079] According to the twentieth aspect, by storing the parameters in the SEI area of ​​the bitstream, the parameters can be easily decoded in the decoding device.

[0080] The encoding device involved in the 21st mode of the present disclosure may be, in any one of the 15th to 20th modes, that the size of the intermediate feature map is equal to the size of the smallest feature map among the multiple feature maps generated in the multiple levels, and when the first method is selected for the at least one level, in the generation of the intermediate feature map, the size of the feature map generated in the at least one level is reduced to the size of the intermediate feature map.

[0081] According to the 21st aspect, when the first method is selected for at least one level, a plurality of intermediate feature maps of the level can be appropriately generated by reducing the size of the feature map.

[0082] The encoding device involved in the 22nd mode of the present disclosure may be, in any one of the 15th to 20th modes, that the size of the intermediate feature map is equal to the size of the feature map with the largest size among the multiple feature maps generated in the multiple levels, and when the first method is selected for the at least one level, in the generation of the intermediate feature map, the size of the feature map generated in the at least one level is enlarged to the size of the intermediate feature map.

[0083] According to the 22nd aspect, when the first method is selected for at least one level, a plurality of intermediate feature maps of the level can be appropriately generated by increasing the size of the feature map.

[0084] The encoding device involved in the 23rd mode of the present disclosure may be, in any one of the 15th to 22nd modes, capable of selecting either the first method or the second method for each of the multiple levels, and the more levels among the multiple levels in which the second method is selected, the smaller the number of the multiple intermediate feature maps.

[0085] According to the 23rd method, the more layers among the multiple layers that select the second method, the smaller the number of the multiple intermediate feature maps, thereby being able to reduce the amount of data in the bit stream sent from the encoding device to the decoding device.

[0086] The encoding device involved in the 24th mode of the present disclosure may be, in any one of the 15th to 23rd modes, capable of selecting either the first method or the second method for each of the multiple levels, and in the generation of the intermediate feature map, when the size of the target of the detection object in the machine task is smaller than a threshold, or when local information is required in the machine task, the second method is selected at least for the level with the smallest size of the feature map generated among the multiple levels.

[0087] According to the 24th method, when the size of the target of the detection object in the machine task is smaller than a threshold value, or when local information is required in the machine task, the second method is selected at least for the layer with the smallest size of the feature map generated among multiple layers, thereby reducing the amount of data in the bit stream sent from the encoding device to the decoding device while retaining the information required for the machine task.

[0088] The encoding device involved in the 25th mode of the present disclosure can select any one of the first method and the second method for each level of the multiple levels in any one of the 15th to 22nd modes, and in the generation of the intermediate feature map, when the size of the target of the detection object in the machine task is greater than a threshold, or when global information is required in the machine task, the second method is selected at least for the level with the largest size of the feature map generated among the multiple levels.

[0089] According to the 25th method, when the size of the target of the detection object in the machine task is above a threshold, or when global information is required in the machine task, the second method is selected at least for the layer with the largest size of the feature map generated among multiple layers, thereby reducing the amount of data in the bit stream sent from the encoding device to the decoding device while retaining the information required for the machine task.

[0090] In the decoding method involved in the 26th embodiment of the present disclosure, the decoding device performs the following processing: generating an image by decoding a bit stream, generating a plurality of intermediate feature maps having uniform sizes based on the image, generating a plurality of feature maps having different sizes for each level in each of a plurality of levels of a neural network for machine tasks based on the plurality of intermediate feature maps, and in generating the feature maps, selecting either the first method or the second method for at least one level of the plurality of levels, wherein in the first method, the plurality of intermediate feature maps are used to generate the plurality of feature maps of the level, and in the second method, the plurality of feature maps of the level are generated using the plurality of feature maps generated in a level different from the level among the plurality of levels without using the plurality of intermediate feature maps.

[0091] According to the 26th method, for at least one layer of the multiple layers of the neural network, the first method is selected in the decoding device when the encoding of the feature map is not omitted in the encoding device, and the second method is selected in the decoding device when the encoding of the feature map is omitted in the encoding device, thereby enabling the feature map of the layer to be properly reconstructed in the decoding device.

[0092] In the encoding method involved in the 27th embodiment of the present disclosure, the encoding device performs the following processing: based on the input image of the processing object, generate a plurality of feature maps of different sizes for each layer in each of the plurality of layers of the neural network for machine tasks, generate a plurality of intermediate feature maps of uniform size based on the plurality of feature maps generated in the plurality of layers, generate an image based on the plurality of intermediate feature maps, generate a bit stream by encoding the image, and in the generation of the intermediate feature maps, select either the first method or the second method for at least one layer of the plurality of layers, wherein in the first method, the plurality of intermediate feature maps are generated using the plurality of feature maps generated in the layer, and in the second method, the plurality of intermediate feature maps are generated without using the plurality of feature maps generated in the layer.

[0093] According to the 27th aspect, for at least one of the multiple layers of the neural network, either the first method or the second method is selected, wherein the first method generates multiple intermediate feature maps using multiple feature maps generated in the layer, and the second method generates multiple intermediate feature maps without using the multiple feature maps generated in the layer. When the second method is selected for the layer, encoding of the multiple intermediate feature maps for the layer can be omitted, and thus the amount of data of the bit stream sent from the encoding device to the decoding device can be reduced.

[0094] (Embodiments of the present disclosure)

[0095] Hereinafter, the embodiments of the present disclosure will be described in detail using the drawings. Elements given the same reference numerals in different drawings represent the same or corresponding elements.

[0096] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are an example and are not intended to limit the present disclosure. In addition, for the structural elements in the following embodiments that are not recorded in the independent technical solutions representing the highest concept, they are described as arbitrary structural elements. In addition, in all embodiments, each content can also be combined.

[0097] Figure 1 1 is a diagram schematically showing a configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an encoding device 1 , a transmission path NW, a decoding device 2 , and a machine task processing unit 3 .

[0098] The encoding device 1 is composed of an information processing unit 11 and a memory 12 connected to the information processing unit 11. However, the memory 12 may be included in the information processing unit 11. The information processing unit 11 is a circuit that performs various information processing, and is composed of a processor such as a CPU or a GPU. The information processing includes processing performed by the machine task processing unit 3 using a neural network 15 for machine tasks. The neural network 15 includes, for example, a first neural network (Feature Pyramid Network) for generating multiple feature maps in Faster-RCNN. The memory 12 includes a semiconductor memory such as ROM or RAM, a magnetic disk or an optical disk, etc. The memory 12 stores information required for the processor to perform processing. For example, the memory 12 stores an input image D1 of a processing object. In addition, the memory 12 stores a program. The processor executes the program read from the memory 12, so that the processor acts as described later. Figure 2A , 2B Each of the processing units shown functions. The encoding device 1 generates a bit stream D2 based on the input image D1, and transmits the generated bit stream D2 to the decoding device 2 via the transmission path NW. The details of the processing performed by the encoding device 1 will be described later.

[0099] The transmission path NW is the Internet, WAN (Wide Area Network), LAN (Local Area Network) or any combination thereof. The transmission path NW can be a public network or a dedicated network that ensures secure communication through access restrictions. In addition, the transmission path NW is not necessarily limited to a two-way communication network, but can also be a one-way communication network that sends broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. In addition, the transmission path NW can also be a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blue-Ray Disc) that records the bit stream D2.

[0100] The decoding device 2 is composed of an information processing unit 21 and a memory 22 connected to the information processing unit 21. However, the memory 22 may be included in the information processing unit 21. The information processing unit 21 is a circuit that performs various information processing, and is composed of a processor such as a CPU or a GPU. The information processing includes processing performed by the machine task processing unit 3 using a neural network 25 for machine tasks. The neural network 25 includes, for example, a second neural network (Region Proposal Network) for extracting ROI (Region Of Interest) areas in Faster-RCNN. The memory 22 includes a semiconductor memory such as ROM or RAM, a magnetic disk or an optical disk, etc. In the memory 22, information required for the processor to perform processing is stored. For example, the bit stream D2 received from the encoding device 1 is stored in the memory 22. In addition, a program is stored in the memory 22. The processor executes the program read from the memory 22, so that the processor acts as described later. Fig.13A , 13B The processing units shown function as follows: The decoding device 2 reconstructs a plurality of feature maps based on the bit stream D2 received from the encoding device 1, and inputs data D3 including the reconstructed plurality of feature maps to the machine task processing unit 3. Details of the processing performed by the decoding device 2 will be described later.

[0101] The machine task processing unit 3 executes machine tasks based on the data D3 input from the decoding device 2, and outputs data D4 including inference results of the machine tasks, etc. The machine tasks include object detection, object segmentation, object tracking, action recognition, or posture estimation, etc.

[0102] (Structure and Processing of Encoding Device 1)

[0103] Figure 2A 1 is a diagram showing a first configuration example of the information processing unit 11. Figure 2A , an example is shown in which the sizes of multiple intermediate feature maps D22 to D25 having uniform sizes are equal to the size of the smallest feature map D15 among the multiple feature maps D12 to D15 generated in multiple levels of the P2 layer to the P5 layer.

[0104] The information processing unit 11 has a neural network 15 composed of four layers, namely, the P2 layer to the P5 layer. However, the number of layers of the neural network 15 is not limited to four layers, as long as it is two or more layers. Specifically, the information processing unit 11 includes: feature map generation units 32 to 35, switching processing units 42 to 45, downsampling units 52D to 54D, convolution processing units 62 to 65, switching control unit 71, image generation unit 72, and encoding processing unit 73. In addition, for example, for the P3 layer and the P4 layer as the intermediate layers, the output of the feature map generation units 33 and 34 and the input of the downsampling units 53D and 54D can be directly connected by omitting the installation of the switching processing units 43 and 44.

[0105] The feature map generation unit 32, the switching processing unit 42, the down-sampling unit 52D, and the convolution processing unit 62 correspond to the P2 layer, which is the lowest layer. The feature map generation unit 33, the switching processing unit 43, the down-sampling unit 53D, and the convolution processing unit 63 correspond to the P3 layer, which is one layer higher than the P2 layer. The feature map generation unit 34, the switching processing unit 44, the down-sampling unit 54D, and the convolution processing unit 64 correspond to the P4 layer, which is one layer higher than the P3 layer. The feature map generation unit 35, the switching processing unit 45, and the convolution processing unit 65 correspond to the P5 layer, which is the highest layer.

[0106] The switching processing units 42 to 45 have a terminal A on the input side, and terminals X and Y on the output side, and select one of the terminal X and the terminal Y to connect to the terminal A based on the parameter D30 input from the switching control unit 71. Figure 2A In the example shown, the switching processing unit 42 connects the terminal A to the terminal X, and the switching processing units 43 to 45 connect the terminal A to the terminal Y. The switching control unit 71 sets the parameter D30 based on the setting information input by the user operation, the setting information obtained based on the analysis result of the input image D1, or the setting information input from the machine task processing unit 3.

[0107] For example, when the size of the target of the detection object in the machine task executed by the machine task processing unit 3 is greater than a given threshold, or when global information is required in the machine task executed by the machine task processing unit 3, the switching control unit 71 connects the terminal A to the terminal X at least for the P2 layer whose feature map size is the largest among the P2 layer to P5 layer. In this case, the feature map D12 generated by the feature map generating unit 32 is discarded, and the feature map D12 is not input to the downsampling unit 52D. In addition, the switching control unit 71 is not limited to the P2 layer, but can also connect the terminal A to the terminal X for the two layers of the P2 layer and the P3 layer or the three layers of the P2 layer to the P4 layer. In addition, as another example, when the processing bit rate of the encoding device 1 or the decoding device 2 is lower than the threshold, when the communication bit rate of the transmission path NW is lower than the threshold, or when the inference accuracy of the machine task executed by the machine task processing unit 3 is lower than the threshold, the terminal A can be connected to the terminal X at least in the P2 layer of the lowest layer.

[0108] Figure 2B 2 is a diagram showing a second configuration example of the information processing unit 11. Figure 2B , an example is shown in which the sizes of multiple intermediate feature maps D22 to D25 having uniform sizes are equal to the size of the largest feature map D12 among the multiple feature maps D12 to D15 generated in multiple levels of the P2 layer to the P5 layer.

[0109] The information processing unit 11 has a neural network 15 composed of four layers, namely, the P2 layer to the P5 layer. However, the number of layers of the neural network 15 is not limited to four layers, as long as it is two or more layers. Specifically, the information processing unit 11 includes: feature map generation units 32 to 35, switching processing units 42 to 45, upsampling units 53U to 55U, convolution processing units 62 to 65, switching control unit 71, image generation unit 72, and encoding processing unit 73. In addition, for example, for the P3 layer and the P4 layer as the intermediate layers, the output of the feature map generation units 33 and 34 and the input of the upsampling units 53U and 54U can be directly connected by omitting the installation of the switching processing units 43 and 44.

[0110] The feature map generation unit 32, the switching processing unit 42, and the convolution processing unit 62 correspond to the P2 layer, which is the lowest layer. The feature map generation unit 33, the switching processing unit 43, the upsampling unit 53U, and the convolution processing unit 63 correspond to the P3 layer, which is one layer higher than the P2 layer. The feature map generation unit 34, the switching processing unit 44, the upsampling unit 54U, and the convolution processing unit 64 correspond to the P4 layer, which is one layer higher than the P3 layer. The feature map generation unit 35, the switching processing unit 45, the upsampling unit 55U, and the convolution processing unit 65 correspond to the P5 layer, which is the highest layer.

[0111] The switching processing units 42 to 45 have a terminal A on the input side, and terminals X and Y on the output side, and select one of the terminal X and the terminal Y to connect to the terminal A based on the parameter D30 input from the switching control unit 71. Figure 2B In the example shown, the switching processing units 42 to 44 connect the terminal A to the terminal Y, and the switching processing unit 45 connects the terminal A to the terminal X. The switching control unit 71 sets the parameter D30 based on the setting information input by the user operation, the setting information obtained based on the analysis result of the input image D1, or the setting information input from the machine task processing unit 3.

[0112] For example, when the size of the target of the detection object in the machine task executed by the machine task processing unit 3 is smaller than a given threshold, or when local information is required in the machine task executed by the machine task processing unit 3, the switching control unit 71 connects terminal A to terminal X for at least the P5 layer, which has the smallest size of the feature map generated among the P2 layer to the P5 layer. In this case, the feature map D15 generated by the feature map generating unit 35 is discarded, and the feature map D15 is not input to the upsampling unit 55U. In addition, the switching control unit 71 is not limited to the P5 layer, but can also connect terminal A to terminal X for the two layers of the P4 layer and the P5 layer, or the three layers of the P3 layer to the P5 layer.

[0113] Figure 3 : is a flowchart showing the flow of processing performed by the information processing unit 11. Figure 4 : is a flowchart showing details of the intermediate feature map generation process (step SP12 ) executed as a subroutine.

[0114] First, in step SP11 , the feature map generation units 32 to 35 generate feature maps D12 to D15 based on the input image D1 .

[0115] Figure 5is a diagram showing feature maps D12 to D15. First, the feature map generation unit 32 performs convolution processing on the input image D1 using a given filter, thereby generating, for example, 256 feature maps D12. Assuming that the horizontal size of the input image D1 is W and the vertical size is H, the horizontal size of the feature map D12 is W / 4 and the vertical size is H / 4. Next, the feature map generation unit 33 performs convolution processing on the feature map D12 using a given filter, thereby generating, for example, 256 feature maps D13. The horizontal size of the feature map D13 is W / 8 and the vertical size is H / 8. Next, the feature map generation unit 34 performs convolution processing on the feature map D13 using a given filter, thereby generating, for example, 256 feature maps D14. The horizontal size of the feature map D14 is W / 16 and the vertical size is H / 16. Next, the feature map generation unit 35 performs convolution processing on the feature map D14 using a given filter, thereby generating, for example, 256 feature maps D15. The horizontal size of the characteristic map D15 is W / 32, and the vertical size is H / 32. The characteristic map generation units 32 to 35 input the generated characteristic maps D12 to D15 to the terminals A of the switching processing units 42 to 45.

[0116] Next, in step SP12, based on the feature graphs D12 to D15 and the parameter D30, use Figure 4 The processing flow shown generates intermediate feature maps D22 to D25.

[0117] Specifically, first, in step SP121, the switching processing unit 42 determines, based on the parameter D30, whether the first method (that is, terminal Y is selected) or the second method (that is, terminal X is selected) is selected for the P2 layer. The first method is a method of using the feature map D12 when generating the intermediate feature maps D22 to D25, and the second method is a method of not using the feature map D12 when generating the intermediate feature maps D22 to D25. When the first method is selected for the P2 layer (step SP121: yes), next, in step SP122, the switching processing unit 42 connects terminal A to terminal Y, thereby inputting the feature map D12 to the subsequent processing unit ( Figure 2A The downsampling section 52D or Figure 2B Convolution processing unit 62). When the second method is selected for the P2 layer (step SP121: No), next, in step SP123, the switching processing unit 42 connects terminal A to terminal X, thereby not inputting the feature map D12 to the subsequent processing unit.

[0118] Next, in step SP124, it is determined whether the processing of steps SP121 to SP123 has been completed for all the layers from the P2 layer to the P5 layer.

[0119] When there is an unprocessed layer (step SP124: No), in step SP125, the layer is updated from the P2 layer to the P3 layer, and then, in step SP121, the switching processing unit 43 determines whether the first method or the second method is selected for the P3 layer based on the parameter D30. The first method is a method of using the feature map D13 when generating the intermediate feature maps D22 to D25, and the second method is a method of not using the feature map D13 when generating the intermediate feature maps D22 to D25. When the first method is selected for the P3 layer (step SP121: Yes), then, in step SP122, the switching processing unit 43 connects terminal A to terminal Y, thereby inputting the feature map D13 to the subsequent processing unit ( Figure 2A The downsampling section 53D or Figure 2B When the second method is selected for the P3 layer (step SP121: No), next, in step SP123, the switching processing unit 43 connects the terminal A to the terminal X, thereby not inputting the feature map D13 to the subsequent processing unit.

[0120] Next, in step SP124, it is determined whether the processing of steps SP121 to SP123 has been completed for all the layers from the P2 layer to the P5 layer.

[0121] When there is an unprocessed layer (step SP124: No), the layer is updated from the P3 layer to the P4 layer in step SP125, and then, in step SP121, the switching processing unit 44 determines whether the first method or the second method is selected for the P4 layer based on the parameter D30. The first method is a method of using the feature map D14 when generating the intermediate feature maps D22 to D25, and the second method is a method of not using the feature map D14 when generating the intermediate feature maps D22 to D25. When the first method is selected for the P4 layer (step SP121: Yes), then, in step SP122, the switching processing unit 44 connects terminal A to terminal Y, thereby inputting the feature map D14 to the subsequent processing unit ( Figure 2A The down sampling unit 54D or Figure 2B When the second method is selected for the P4 layer (step SP121: No), next, in step SP123, the switching processing unit 44 connects the terminal A to the terminal X, thereby not inputting the feature map D14 to the subsequent processing unit.

[0122] Next, in step SP124, it is determined whether the processing of steps SP121 to SP123 has been completed for all the layers from the P2 layer to the P5 layer.

[0123] When there is an unprocessed layer (step SP124: No), in step SP125, the layer is updated from the P4 layer to the P5 layer, and then, in step SP121, the switching processing unit 45 determines whether the first method or the second method is selected for the P5 layer based on the parameter D30. The first method is a method of using the feature map D15 when generating the intermediate feature maps D22 to D25, and the second method is a method of not using the feature map D15 when generating the intermediate feature maps D22 to D25. When the first method is selected for the P5 layer (step SP121: Yes), then, in step SP122, the switching processing unit 45 connects terminal A to terminal Y, thereby inputting the feature map D15 to the subsequent processing unit ( Figure 2A The convolution processing unit 65 or Figure 2B When the second method is selected for the P5 layer (step SP121: No), next, in step SP123, the switching processing unit 45 connects the terminal A to the terminal X, thereby not inputting the feature map D15 to the subsequent processing unit.

[0124] If there is no unprocessed layer (step SP124: Yes), then, in step SP126, a resizing process is performed to make the size of the feature maps D12 to D15 consistent with the size of the intermediate feature maps D22 to D25, and then, in step SP127, a convolution process is performed to reduce the number of intermediate feature maps D22 to D25 compared to the number of feature maps D12 to D15. Figure 2A The P5 layer and the upsampling units 53U to 55U are not provided. Figure 2B For the P2 layer of the image processing unit, the resizing process of step SP126 is omitted. In addition, for the layer for which the second method is selected based on the parameter D30, the resizing process of step SP126 and the convolution process of step SP127 are omitted.

[0125] Figure 6 is with Figure 2A The example shown in the figure shows the intermediate feature maps D22 to D25 input to the convolution processing units 62 to 65 after the size conversion process. The downsampling unit 52D reduces the size of the feature map D12 to 1 / 8 in the horizontal and vertical directions, thereby generating an intermediate feature map D22 with a horizontal size of W / 32 and a vertical size of H / 32. Figure 2AIn the example shown, the second method is selected for the P2 layer, so the downsampling unit 52D does not generate the intermediate feature map D22, and the intermediate feature map D22 is not input to the convolution processing unit 62. The downsampling unit 53D reduces the size of the feature map D13 to 1 / 4 in the horizontal and vertical directions, thereby inputting 256 intermediate feature maps D23 with a horizontal size of W / 32 and a vertical size of H / 32 to the convolution processing unit 63. The downsampling unit 54D reduces the size of the feature map D14 to 1 / 2 in the horizontal and vertical directions, thereby inputting 256 intermediate feature maps D24 with a horizontal size of W / 32 and a vertical size of H / 32 to the convolution processing unit 64. In addition, with respect to the P5 layer, 256 intermediate feature maps D25 that are the same as the feature map D15 are input to the convolution processing unit 65.

[0126] Figure 7 is with Figure 2A The example shown shows the intermediate feature maps D23 to D25 input to the image generation unit 72 after the convolution process. When the first method is selected for all layers of the P2 layer to the P5 layer, the convolution processing units 62 to 65 reduce the total number of intermediate feature maps D22 to D25 from 1024 (=256×4) to 144, for example, through the convolution process. Figure 2A In the example shown, the second method is selected for the P2 layer, and the 256 intermediate feature maps D22 are not input to the convolution processing unit 62. Therefore, the convolution processing units 63 to 65 reduce the total number of intermediate feature maps D23 to D25 from 768 (= 256 × 3) to 108 (= 144 × 3 / 4) through convolution processing. Figure 2A , 2B As shown, in the example of this embodiment, either the first method or the second method can be selected for each layer of the P2 layer to the P5 layer, and the total number of intermediate feature maps D22 to D25 input from the convolution processing units 62 to 65 to the image generation unit 72 is smaller as the number of layers in which the second method is selected increases, and the total number of layers in which the first method is selected increases. In addition, the number of bits of each pixel of the intermediate feature maps D23 to D25 is 32 bits in the example of this embodiment.

[0127] Fig.17 1 is a diagram showing the structure of the information processing unit 11 involved in the modification. In the structure of the embodiment, for each layer of the P2 layer to the P5 layer, the convolution processing is performed on each signal selected from the feature maps D12 to D15 and downsampled or upsampled, thereby generating intermediate feature maps D22 to D25 with a reduced total number of images. The structure of the embodiment can also be replaced by, for example Fig.17As shown, the following structure is set: the signal selected from the feature maps D12 to D15 for each layer of the P2 layer to the P5 layer and down-sampled or up-sampled is combined by the combination processing unit 60, and the combined signal D20 is convoluted by the convolution processing unit 61, thereby generating an intermediate feature map D21 with a reduced total number of sheets. According to the structure involved in the modification example, the convolution processing units 62 to 65 are gathered into one convolution processing unit 61, and the convolution processing can be concentrated while using the signals of all layers, so it is possible to generate a better feature map.

[0128] Reference Figure 3 Next, in step SP13, the image generating unit 72 generates an image D31 based on the intermediate feature maps D22 to D25 input from the convolution processing units 62 to 65.

[0129] Figure 8 is with Figure 2A The example shown is related to a diagram showing a generation process of an image D31 by the image generation unit 72. The image generation unit 72 performs a packing process in which the intermediate feature maps D23 to D25 input from the convolution processing units 63 to 65 are arranged in order in a raster scan order from the upper layer within the frame, thereby generating the image D31. At this time, the image generation unit 72 reduces the number of bits of each pixel of the intermediate feature maps D23 to D25 from 32 bits to 10 bits by a quantization process in which the lower bits are discarded. In addition, the arrangement order of the intermediate feature maps D23 to D25 may be arranged in order from the lower layer, or may be an order different from the raster scan order.

[0130] Next, in step SP14 , the encoding processing unit 73 encodes the image D31 input from the image generating unit 72 using a given compression encoding method such as VVC (Versatile Video Coding) to generate a bit stream D2 . The encoding processing unit 73 transmits the generated bit stream D2 to the decoding device 2 via the transmission path NW.

[0131] Fig. 9: is a diagram showing an example of the data structure of the bitstream D2. The bitstream D2 has a header area R1 for storing management information and the like and a payload area R2 for storing image data. The encoding processing unit 73 stores the encoded data obtained by encoding the image D31 in the payload area R2. In addition, the parameter D30 is input from the switching control unit 71 to the encoding processing unit 73, and the encoding processing unit 73 stores the encoded data 67 obtained by encoding the parameter D30 in a given part of the header area R1. The given part is, for example, a SEI (Supplemental Enhancement Information) area for storing additional information. The given part may also be a VPS, SPS, PPS, PH, SH, APS or a block header. By storing the encoded data 67 of the parameter D30 in the header area R1 of the bitstream D2, the decoding device 2 can easily obtain the parameter D30 by decoding. In addition, by storing the encoded data 67 in the SEI area, the parameter D30 can be easily processed as additional information. Furthermore, the encoding processing unit 73 may store the encoded data 67 of the parameter D30 in the payload region R2.

[0132] Fig. 10A , 10B : is a diagram showing a first example of the data structure of parameter D30. Parameter D30 includes four flags F2 to F5 corresponding to four levels, namely, P2 to P5. The value of each flag F2 to F5 indicates whether the first method or the second method is selected for the corresponding level. When the value of each flag is "1", it indicates that the first method is selected for the corresponding level, and when the value of each flag is "0", it indicates that the second method is selected for the corresponding level. Fig. 10A and Figure 2B Correspondingly, it is shown that the first method is selected for the P2 to P4 layers, and the second method is selected for the P5 layer. Fig. 10B and Figure 2A Correspondingly, it is shown that the first method is selected for the P3 to P5 layers, and the second method is selected for the P2 layer.

[0133] Fig.11A , 11B : is a diagram showing a second example of the data structure of parameter D30. Parameter D30 includes 4-bit data 68 corresponding to four layers, namely, layer P2 to layer P5. The value of each bit of data 68 indicates which of the first method and the second method is selected for the corresponding layer. When the value of each bit is "1", it indicates that the first method is selected for the corresponding layer, and when the value of each bit is "0", it indicates that the second method is selected for the corresponding layer. Fig.11A It is shown that the first method is selected for the P2 layer and the P3 layer, and the second method is selected for the P4 layer and the P5 layer. Fig. 11B This indicates that the first method is selected for all the layers from the P2 layer to the P5 layer.

[0134] Fig. 12A , 12B : is a diagram showing the third example of the data structure of parameter D30. Regarding which combination of the first method and the second method is selected for each layer from the P2 layer to the P5 layer, a plurality of modes are predefined in table information 69, and an index value is assigned as information for identifying each mode. In this example, four modes are defined and a 2-bit index value is assigned. The switching control unit 71 selects one mode from the four modes and includes the index value corresponding to the selected mode in parameter D30. Fig. 12A In the example shown, when the mode with an index value of "11" is selected, the first method is selected for the P2 layer, and the second method is selected for the P3 to P5 layers. When the mode with an index value of "10" is selected, the first method is selected for the P2 and P3 layers, and the second method is selected for the P4 and P5 layers. When the mode with an index value of "01" is selected, the first method is selected for the P2 to P4 layers, and the second method is selected for the P5 layer. When the mode with an index value of "00" is selected, the first method is selected for all layers from the P2 to P5 layers. Fig. 12B In the example shown, when the mode with the index value "11" is selected, the first method is selected for the P5 layer, and the second method is selected for the P2 to P4 layers. When the mode with the index value "10" is selected, the first method is selected for the P4 and P5 layers, and the second method is selected for the P2 and P3 layers. When the mode with the index value "01" is selected, the first method is selected for the P3 to P5 layers, and the second method is selected for the P2 layer. When the mode with the index value "00" is selected, the first method is selected for all layers from the P2 to P5 layers.

[0135] (Structure and Processing of Decoding Device 2)

[0136] Fig.13A 2 is a diagram showing a first configuration example of the information processing unit 21. Fig.13A In, with Figure 2A Correspondingly, an example is shown in which the size of multiple intermediate feature maps D42 with uniform size is equal to the size of the smallest feature map D15A among the multiple feature maps D12A to D15A generated in multiple levels of the P2 layer to the P5 layer.

[0137] The information processing unit 21 has a neural network 25 composed of four layers, namely, the P2 layer to the P5 layer. However, the number of layers of the neural network 25 is not limited to four layers, as long as it is two or more layers. Specifically, the information processing unit 21 includes: a switching control unit 81, a decoding processing unit 82, an intermediate feature map generation unit 83, a convolution processing unit 92 to 95, an upsampling unit 102U to 104U, 113U to 115U, adders 122U to 124U, and a switching processing unit 132 to 134. In addition, as an example of mathematical operation, it is not limited to addition based on adders 122U to 124U, and it can also be subtraction, shift, convolution, or any combination thereof.

[0138] The convolution processing unit 92, the upsampling unit 102U, the adder 122U, and the switching processing unit 132 correspond to the P2 layer, which is the lowest layer. The convolution processing unit 93, the upsampling unit 103U, the adder 123U, and the switching processing unit 133 correspond to the P3 layer, which is one layer higher than the P2 layer. The convolution processing unit 94, the upsampling unit 104U, the adder 124U, and the switching processing unit 134 correspond to the P4 layer, which is one layer higher than the P3 layer. The convolution processing unit 95 corresponds to the P5 layer, which is the highest layer.

[0139] The switching processing units 132 to 134 have terminals X and Y on the input side and terminal B on the output side, and select one of the terminal X and the terminal Y to connect to the terminal B based on the parameter D30 input from the switching control unit 81. Fig.13A In the example shown, Figure 2A Correspondingly, the switching processing unit 132 connects the terminal B to the terminal X, and the switching processing units 133 and 134 connect the terminal B to the terminal Y. The switching control unit 81 inputs the parameter D30 decoded from the bit stream D2 by the decoding processing unit 82 to the switching processing units 132 to 134 .

[0140] The switching control unit 81 controls the switching processing units 132 to 134 based on the parameter D30 decoded from the bit stream D2 by the decoding processing unit 82. For example, when the size of the target of the detection object in the machine task executed by the machine task processing unit 3 is greater than a given threshold, or when global information is required in the machine task executed by the machine task processing unit 3, the switching control unit 81 connects the terminal B to the terminal X based on the parameter D30 at least for the P2 layer whose feature map size is the largest among the P2 layer to P5 layer. In this case, at least the feature map D12A of the P2 layer is generated not using the intermediate feature map D432 of the P2 layer, but using the feature map D13A generated in the P3 layer of the upper layer. In addition, the switching control unit 81 is not limited to the P2 layer, but can also connect the terminal B to the terminal X for the two layers of the P2 layer and the P3 layer or the three layers of the P2 layer to the P4 layer.

[0141] Fig. 13B 2 is a diagram showing a second configuration example of the information processing unit 21. Fig. 13B In, with Figure 2B Correspondingly, an example is shown in which the sizes of multiple intermediate feature maps D42 with uniform sizes are equal to the size of the largest feature map D12A among the multiple feature maps D12A to D15A generated in multiple levels of the P2 layer to the P5 layer.

[0142] The information processing unit 21 has a neural network 25 composed of four layers, namely, the P2 layer to the P5 layer. However, the number of layers of the neural network 25 is not limited to four layers, as long as it is two or more layers. Specifically, the information processing unit 21 includes: a switching control unit 81, a decoding processing unit 82, an intermediate feature map generation unit 83, convolution processing units 92 to 95, downsampling units 103D to 105D, 112D to 114D, adders 123D to 125D, and switching processing units 143 to 145.

[0143] The convolution processing unit 92 corresponds to the P2 layer, which is the lowest layer. The convolution processing unit 93, the downsampling unit 103D, the adder 123D, and the switching processing unit 143 correspond to the P3 layer, which is one layer higher than the P2 layer. The convolution processing unit 94, the downsampling unit 104D, the adder 124D, and the switching processing unit 144 correspond to the P4 layer, which is one layer higher than the P3 layer. The convolution processing unit 95, the downsampling unit 105D, the adder 125D, and the switching processing unit 145 correspond to the P5 layer, which is the highest layer.

[0144] The switching processing units 143 to 145 have terminals X and Y on the input side and terminal B on the output side, and select one of the terminals X and Y to connect to the terminal B based on the parameter D30 input from the switching control unit 81. Fig. 13B In the example shown, Figure 2B Correspondingly, the switching processing units 143 and 144 connect the terminal B to the terminal Y, and the switching processing unit 145 connects the terminal B to the terminal X. The switching control unit 81 inputs the parameter D30 decoded from the bit stream D2 by the decoding processing unit 82 to the switching processing units 143 to 145 .

[0145] The switching control unit 81 controls the switching processing units 132 to 134 based on the parameter D30 decoded from the bit stream D2 by the decoding processing unit 82. For example, when the size of the target of the detection object in the machine task executed by the machine task processing unit 3 is smaller than a given threshold, or when local information is required in the machine task executed by the machine task processing unit 3, the switching control unit 81 connects the terminal B to the terminal X based on the parameter D30 at least for the P5 layer whose feature map is generated with the smallest size among the P2 layer to the P5 layer. In this case, at least the feature map D15A of the P5 layer is generated not using the intermediate feature map D435 of the P5 layer, but using the feature map D14A generated in the P4 layer of the lower layer. In addition, the switching control unit 81 is not limited to the P5 layer, but can also connect the terminal B to the terminal X for the two layers of the P4 layer and the P5 layer or the three layers of the P3 layer to the P5 layer.

[0146] Fig.14 : is a flowchart showing the flow of processing performed by the information processing unit 21. Fig.15 : is a flowchart showing the details of the feature map generation process (step SP23 ) executed as a subroutine.

[0147] First, in step SP21, the decoding processing unit 82 generates an image D41 equivalent to the image D31 by decoding the bit stream D2 received from the encoding device 1. In addition, the decoding processing unit 82 obtains the parameter D30 by decoding the encoded data 67 stored in the SEI area of ​​the bit stream D2, for example, and inputs the obtained parameter D30 to the switching control unit 81.

[0148] Next, in step SP22 , the intermediate feature map generation unit 83 generates an intermediate feature map D42 corresponding to the intermediate feature maps D22 to D25 based on the image D41 input from the decoding processing unit 82 .

[0149] Specifically, the intermediate feature map generation unit 83 performs an unpacking process of expanding the intermediate feature maps D22 to D25 included in the image D41 input from the decoding processing unit 82 in order from the upper layer in a raster scan order, thereby generating the intermediate feature map D42. At this time, the intermediate feature map generation unit 83 increases the number of bits of each pixel of the intermediate feature map D42 from 10 bits to 32 bits through an inverse quantization process. In addition, the expansion order of the intermediate feature maps D22 to D25 can be expanded in order from the lower layer, or it can be an order different from the raster scan order. The intermediate feature map generation unit 83 inputs the generated intermediate feature map D42 to the convolution processing units 92 to 95. In addition, regarding the total number of intermediate feature maps D42, the more the number of layers in the encoding device 1 that select the second method among the multiple layers, the smaller the total number, and the more the number of layers that select the first method, the larger the total number.

[0150] Next, in step SP23, based on the intermediate feature map D42 and the parameter D30, use Fig.15 The processing flow shown generates feature graphs D12A to D15A corresponding to feature graphs D12 to D15.

[0151] Specifically, first, in step SP231, a convolution process is performed to restore the number of intermediate feature maps D42 to the number of feature maps D12 to D15. The convolution processing units 92 to 95 generate 256 intermediate feature maps D432 to D435 for each level of the P2 to P5 layers, for example, based on the 108 intermediate feature maps D42, by performing a convolution process using 256 filters. Fig.13A In the example shown, 256 feature maps D15A of the P5 layer are obtained as the intermediate feature map D435 output from the convolution processing unit 95. Fig. 13B In the example shown, 256 feature maps D12A of the P2 layer are obtained as the intermediate feature map D432 output from the convolution processing unit 92.

[0152] Fig.18 2 is a diagram showing the structure of the information processing unit 21 involved in the modified example. In the structure of the embodiment, the convolution processing units 92 to 94 perform convolution processing on the intermediate feature map D42 whose total number of images is reduced, and generate intermediate feature maps D432 to D435 whose number of images is restored for each layer of the P2 layer to the P5 layer. Alternatively, the structure of the embodiment may be replaced by, for example, Fig.18As shown, the following structure is set: the segmentation processing unit 84 segments the intermediate feature map D42 whose total number is reduced toward each layer of the P2 layer to the P5 layer, and the convolution processing units 92 to 95 perform convolution processing on each segmented intermediate feature map D42, thereby generating intermediate feature maps D432 to D435 whose number is restored. At this time, if it is known based on the parameter D30 that the intermediate feature map D432 is not used by the switching processing unit 132, for example, the segmentation processing unit 84 may also segment the intermediate feature map D42 toward each layer of the P3 layer to the P5 layer. According to the structure involved in the modified example, it is possible to reduce the processing amount required by each convolution processing unit 92 to 95.

[0153] Next, in step SP232, a size conversion process is performed to restore the size of the intermediate feature graphs D432 to D435 to the size of the feature graphs D12 to D15. Fig.13A The P5 layer and the downsampling units 103D to 105D are not provided. Fig. 13B The P2 layer of step SP232 is omitted. The up-sampling unit 102U enlarges the size of the intermediate feature map D432 to 8 times in the horizontal and vertical directions, the up-sampling unit 103U enlarges the size of the intermediate feature map D433 to 4 times in the horizontal and vertical directions, and the up-sampling unit 104U enlarges the size of the intermediate feature map D434 to 2 times in the horizontal and vertical directions. The down-sampling unit 103D reduces the size of the intermediate feature map D433 to 1 / 2 times in the horizontal and vertical directions, the down-sampling unit 104D reduces the size of the intermediate feature map D434 to 1 / 4 times in the horizontal and vertical directions, and the down-sampling unit 105D reduces the size of the intermediate feature map D435 to 1 / 8 times in the horizontal and vertical directions.

[0154] As mentioned above, in Fig.13A In the example shown, 256 feature maps D15A of the P5 layer are obtained as the intermediate feature map D435 output from the convolution processing unit 95. The size of the feature map D15A is W / 32 in the horizontal dimension and H / 32 in the vertical dimension.

[0155] Next, in step SP233, the switching processing unit 134 determines whether the first method or the second method is selected for the P4 layer based on the parameter D30. The first method is a method of using the intermediate feature map D434 when generating the feature map D14A, and the second method is a method of not using the intermediate feature map D434 when generating the feature map D14A.

[0156] When the first method is selected for the P4 layer (step SP233: Yes), next, in step SP234, the switching processing unit 134 connects terminal B to terminal Y, thereby using the intermediate feature map D434 of the P4 layer and the feature map D15A of the P5 layer one level above to generate a feature map D14A of the P4 layer. The upsampling unit 115U increases the size of the feature map D15A by 2 times in both the horizontal and vertical directions. The adder 124U adds the intermediate feature map D434 output from the upsampling unit 104U and the feature map D15A output from the upsampling unit 115U, thereby generating a feature map D14A. The size of the feature map D14A is W / 16 in horizontal dimension and H / 16 in vertical dimension.

[0157] When the second method is selected for the P4 layer (step SP233: No), next, in step SP235, the switching processing unit 134 connects terminal B to terminal X, thereby generating a feature map D14A of the P4 layer using the feature map D15A of the P5 layer one level above instead of using the intermediate feature map D434 of the P4 layer. The feature map D15A output from the upsampling unit 115U is used to obtain the feature map D14A.

[0158] Next, in step SP236, it is determined whether the feature map generation process is completed for all the layers from the P2 layer to the P5 layer.

[0159] If there is an unprocessed layer (step SP236: No), in step SP237, the layer is updated from the P4 layer to the P3 layer, and then in step SP233, the switching processing unit 133 determines whether the first method or the second method is selected for the P3 layer based on the parameter D30. The first method is a method of using the intermediate feature map D433 when generating the feature map D13A, and the second method is a method of not using the intermediate feature map D433 when generating the feature map D13A.

[0160] When the first method is selected for the P3 layer (step SP233: Yes), next, in step SP234, the switching processing unit 133 connects terminal B to terminal Y, thereby using the intermediate feature map D433 of the P3 layer and the feature map D14A of the P4 layer one level above to generate a feature map D13A of the P3 layer. The upsampling unit 114U increases the size of the feature map D14A by 2 times in both the horizontal and vertical directions. The adder 123U adds the intermediate feature map D433 output from the upsampling unit 103U and the feature map D14A output from the upsampling unit 114U, thereby generating a feature map D13A. The size of the feature map D13A is W / 8 in horizontal dimension and H / 8 in vertical dimension.

[0161] When the second method is selected for the P3 layer (step SP233: No), next, in step SP235, the switching processing unit 133 connects terminal B to terminal X, thereby generating a feature map D13A of the P3 layer using the feature map D14A of the P4 layer one level higher instead of the intermediate feature map D433 of the P3 layer. The feature map D14A output from the upsampling unit 114U is used to obtain the feature map D13A.

[0162] Next, in step SP236, it is determined whether the feature map generation process is completed for all the layers from the P2 layer to the P5 layer.

[0163] If there is an unprocessed layer (step SP236: No), in step SP237, the layer is updated from the P3 layer to the P2 layer, and then in step SP233, the switching processing unit 132 determines whether the first method or the second method is selected for the P2 layer based on the parameter D30. The first method is a method of using the intermediate feature map D432 when generating the feature map D12A, and the second method is a method of not using the intermediate feature map D432 when generating the feature map D12A.

[0164] When the first method is selected for the P2 layer (step SP233: Yes), next, in step SP234, the switching processing unit 132 connects terminal B to terminal Y, thereby generating a feature map D12A of the P2 layer using the intermediate feature map D432 of the P2 layer and the feature map D13A of the P3 layer one level above. The upsampling unit 113U increases the size of the feature map D13A by 2 times in both the horizontal and vertical directions. The adder 122U adds the intermediate feature map D432 output from the upsampling unit 102U and the feature map D13A output from the upsampling unit 113U to generate a feature map D12A. The horizontal size of the feature map D12A is W / 4, and the vertical size is H / 4.

[0165] When the second method is selected for the P2 layer (step SP233: No), next, in step SP235, the switching processing unit 132 connects terminal B to terminal X, thereby generating a feature map D12A of the P2 layer using the feature map D13A of the P3 layer one level higher instead of using the intermediate feature map D432 of the P2 layer. The feature map D13A output from the upsampling unit 113U is used to obtain the feature map D12A.

[0166] When there is no unprocessed layer (step SP236: Yes), the feature map generation process is terminated.

[0167] In addition, as mentioned above, Fig. 13BIn the example shown, 256 feature maps D12A of the P2 layer are obtained as the intermediate feature map D432 output from the convolution processing unit 92. The feature map D12A has a horizontal size of W / 4 and a vertical size of H / 4.

[0168] In step SP233 , the switching processing unit 143 determines whether the first method or the second method is selected for the P3 layer based on the parameter D30 .

[0169] When the first method is selected for the P3 layer (step SP233: Yes), next, in step SP234, the switching processing unit 143 connects terminal B to terminal Y, thereby generating a feature map D13A of the P3 layer using the intermediate feature map D433 of the P3 layer and the feature map D12A of the P2 layer at one level below. The downsampling unit 112D reduces the size of the feature map D12A to 1 / 2 times in both the horizontal and vertical directions. The adder 123D adds the intermediate feature map D433 output from the downsampling unit 103D and the feature map D12A output from the downsampling unit 112D to generate the feature map D13A.

[0170] When the second method is selected for the P3 layer (step SP233: No), next, in step SP235, the switching processing unit 143 connects terminal B to terminal X, thereby generating a feature map D13A for the P3 layer using the feature map D12A of the P2 layer one level lower, without using the intermediate feature map D433 of the P3 layer. The feature map D12A output from the downsampling unit 112D is used to obtain the feature map D13A.

[0171] Next, in step SP236, it is determined whether the feature map generation process is completed for all the layers from the P2 layer to the P5 layer.

[0172] If there is an unprocessed layer (step SP236: No), in step SP237, the layer is updated from the P3 layer to the P4 layer, and then in step SP233, the switching processing unit 144 determines whether the first method or the second method is selected for the P4 layer based on the parameter D30.

[0173] When the first method is selected for the P4 layer (step SP233: Yes), next, in step SP234, the switching processing unit 144 connects terminal B to terminal Y, thereby generating a feature map D14A of the P4 layer using the intermediate feature map D434 of the P4 layer and the feature map D13A of the P3 layer one level lower. The downsampling unit 113D reduces the size of the feature map D13A to 1 / 2 times in both the horizontal and vertical directions. The adder 124D adds the intermediate feature map D434 output from the downsampling unit 104D and the feature map D13A output from the downsampling unit 113D, thereby generating a feature map D14A.

[0174] When the second method is selected for the P4 layer (step SP233: No), next, in step SP235, the switching processing unit 144 connects terminal B to terminal X, thereby generating a feature map D14A for the P4 layer using the feature map D13A of the P3 layer one level lower, instead of using the intermediate feature map D434 of the P4 layer. The feature map D13A output from the downsampling unit 113D is used to obtain the feature map D14A.

[0175] Next, in step SP236, it is determined whether the feature map generation process is completed for all the layers from the P2 layer to the P5 layer.

[0176] If there is an unprocessed layer (step SP236: No), in step SP237, the layer is updated from the P4 layer to the P5 layer, and then in step SP233, the switching processing unit 145 determines whether the first method or the second method is selected for the P5 layer based on the parameter D30.

[0177] When the first method is selected for the P5 layer (step SP233: Yes), next, in step SP234, the switching processing unit 145 connects terminal B to terminal Y, thereby generating a feature map D15A of the P5 layer using the intermediate feature map D435 of the P5 layer and the feature map D14A of the P4 layer at one level below. The downsampling unit 114D reduces the size of the feature map D14A to 1 / 2 times in both the horizontal and vertical directions. The adder 125D adds the intermediate feature map D435 output from the downsampling unit 105D and the feature map D14A output from the downsampling unit 114D to generate a feature map D15A.

[0178] When the second method is selected for the P5 layer (step SP233: No), next, in step SP235, the switching processing unit 145 connects terminal B to terminal X, thereby generating a feature map D15A for the P5 layer using the feature map D14A of the P4 layer one level lower, instead of using the intermediate feature map D435 of the P5 layer. The feature map D14A output from the downsampling unit 114D is used to obtain the feature map D15A.

[0179] When there is no unprocessed layer (step SP236: Yes), the feature map generation process is terminated.

[0180] Fig.16A 2 is a diagram showing a third configuration example of the information processing unit 21. Fig.13A The first configuration example shown omits the convolution processing units 92 to 95. In this case, Figure 2A The convolution processing units 62 to 65 shown in the figure are also omitted. For each layer P3 to P5 where the first method is selected, 256 intermediate feature maps D23 to D25 with a horizontal size of W / 32 and a vertical size of H / 32 are encoded and sent to the decoding device 2 as a bit stream D2. Figure 2A , 16A In the example shown, a bit stream D2 including a total of 768 (=256×3) intermediate feature maps D23 to D25 of the P3 layer to the P5 layer is transmitted. The intermediate feature map generation unit 83A extracts 256 intermediate feature maps D423 corresponding to the P3 layer from the total of 768 intermediate feature maps D423 to D425 corresponding to the intermediate feature maps D23 to D25 and outputs them to the P3 layer. As a result, for the P3 layer, 256 feature maps D13A, which are less than the total of 768 intermediate feature maps D423 to D425, are generated using the intermediate feature map D423. In addition, the intermediate feature map generation unit 83A extracts 256 intermediate feature maps D424 corresponding to the P4 layer from the total of 768 intermediate feature maps D423 to D425 and outputs them to the P4 layer. As a result, for the P4 layer, the intermediate feature map D424 is used to generate 256 feature maps D14A, which is less than the total 768 intermediate feature maps D423 to D425. In addition, the intermediate feature map generation unit 83A extracts 256 intermediate feature maps D425 corresponding to the P5 layer from the total 768 intermediate feature maps D423 to D425 and outputs them to the P5 layer. As a result, for the P5 layer, the intermediate feature map D425 is used to generate 256 feature maps D15A, which is less than the total 768 intermediate feature maps D423 to D425. In addition, the second method is selected for the P2 layer, so the intermediate feature map D422 of the P2 layer is not used, and the feature map D13A of the P3 layer is used to generate 256 feature maps D12A. Fig.16A The third structural example shown is not limited to application to Fig.13AThe first structural example shown can also be applied to Fig. 13B The second structural example is shown.

[0181] Fig. 16B 2 is a diagram showing a fourth configuration example of the information processing unit 21. Fig.13A The switching processing units 132 to 134 shown in the figure are provided with switching processing units 132A to 134A further having an input terminal Z. When the terminal Z is selected to be connected to the terminal B, the adders 122U to 124U are bypassed, thereby obtaining the intermediate feature maps D432 to D434 output from the up-sampling units 102U to 104U as feature maps D12A to D14A. Fig. 16B The fourth structural example shown is not limited to application to Fig.13A The first structural example shown can also be applied to Fig. 13B The second structural example is shown.

[0182] (Effect)

[0183] According to the encoding device 1 involved in this embodiment, for at least one of the multiple layers of the neural network 15, either a first method of generating multiple intermediate feature maps using multiple feature maps generated in the layer or a second method of generating multiple intermediate feature maps without using the multiple feature maps generated in the layer is selected. By being able to select either the first method or the second method, it is possible to apply the best encoding processing according to the content of the image or machine task, etc. In addition, when the second method is selected for the layer, the encoding of the multiple intermediate feature maps for the layer can be omitted, so the amount of data of the bit stream D2 sent from the encoding device 1 to the decoding device 2 can be reduced.

[0184] In addition, according to the decoding device 2 involved in this embodiment, for at least one of the multiple layers of the neural network 25, when the encoding of the feature map is not omitted in the encoding device 1, the first method is selected in the decoding device 2, and when the encoding of the feature map is omitted in the encoding device 1, the second method is selected in the decoding device 2. As a result, even if the encoding of the intermediate feature map of a part of the layers is omitted in the encoding device 1, the feature map of the layer can be properly reconstructed in the decoding device 2. As a result, it is possible to prevent the inference accuracy of the machine task from being reduced.

[0185] Industrial Applicability

[0186] The present disclosure is particularly useful for application to object detection systems using neural networks for machine-oriented tasks.

Claims

1. A decoding device, comprising: Circuits; and a memory connected to the circuit, The circuit performs the following processing: Generates an image by decoding the bitstream, Based on the image, generating a plurality of intermediate feature maps having uniform sizes, Based on the plurality of intermediate feature maps, a plurality of feature maps having different sizes for each layer are generated at each of the plurality of layers of the neural network for the machine task, In generating the feature map, either the first method or the second method is selected for at least one of the multiple levels. In the first method, the multiple feature maps of the level are generated using the multiple intermediate feature maps, and in the second method, the multiple feature maps of the level are generated using the multiple feature maps generated in a level different from the level among the multiple levels without using the multiple intermediate feature maps.

2. The decoding device according to claim 1, wherein: Furthermore, by decoding the bit stream, a parameter indicating which of the first method and the second method is selected for the at least one layer is obtained, In generating the feature map, based on the parameter, one of the first method and the second method is selected for the at least one level.

3. The decoding device according to claim 2, wherein: For each of the plurality of levels, either the first method or the second method can be selected. The parameter includes a plurality of flags corresponding to the plurality of levels, The value of each of the plurality of flags indicates which of the first method and the second method is selected for the corresponding layer.

4. The decoding device according to claim 2, wherein: For each of the plurality of levels, either the first method or the second method can be selected. The parameter includes a plurality of bits of data corresponding to the plurality of levels, The value of each of the plurality of bits indicates which of the first method and the second method is selected for the corresponding layer.

5. The decoding device according to claim 2, wherein: For each of the plurality of levels, either the first method or the second method can be selected. A plurality of patterns are predefined regarding which combination of the first method and the second method is selected for each of the plurality of levels. Selecting a mode from among the plurality of modes, The parameters include information for determining the one mode.

6. The decoding device according to claim 2, wherein: The parameters are obtained by decoding a supplemental enhancement information (SEI) area of ​​the bitstream.

7. The decoding device according to claim 1, wherein: When the first method is selected for at least one level, in generating the feature map, the multiple intermediate feature maps and the multiple feature maps generated in a level different from the level among the multiple levels are used to generate the multiple feature maps of the level.

8. The decoding device according to claim 1, wherein: The size of the intermediate feature map is equal to the size of the smallest feature map among the plurality of feature maps generated in the plurality of levels, When the first method is selected for the at least one level, in generating the feature map, the size of the intermediate feature map is enlarged to the size of the feature map generated in the at least one level.

9. The decoding device according to claim 1, wherein: The size of the intermediate feature map is equal to the size of the largest feature map among the plurality of feature maps generated in the plurality of levels, When the first method is selected for the at least one level, in the generation of the feature map, the size of the intermediate feature map is reduced to the size of the feature map generated in the at least one level.

10. The decoding device according to claim 1, wherein: When the first method is selected for the at least one level, in the generation of the feature map, the plurality of feature maps, which is greater in number than the plurality of intermediate feature maps, are generated by performing convolution processing on the plurality of intermediate feature maps.

11. The decoding device according to claim 1, wherein: In the case where the first method is selected for the at least one level, in the generation of the feature map, by extracting two or more intermediate feature maps corresponding to the at least one level from the multiple intermediate feature maps, the multiple feature maps, which are less in number than the multiple intermediate feature maps, are generated.

12. The decoding device according to claim 1, wherein: For each of the plurality of levels, either the first method or the second method can be selected. The greater the number of levels that select the second method among the multiple levels, the smaller the number of the multiple intermediate feature maps.

13. The decoding device according to claim 1, wherein: For each of the plurality of levels, either the first method or the second method can be selected. In the generation of the feature map, when the size of the target of the detection object in the machine task is smaller than a threshold, or when local information is required in the machine task, the second method is selected at least for the level with the smallest size of the feature map generated among the multiple levels.

14. The decoding device according to claim 1, wherein: For each of the plurality of levels, either the first method or the second method can be selected. In the generation of the feature map, when the size of the target of the detection object in the machine task is above a threshold, or when global information is required in the machine task, the second method is selected at least for the level with the largest size of the feature map generated among the multiple levels.

15. An encoding device comprising: Circuits; and a memory connected to the circuit, The circuit performs the following processing: Based on the input image to be processed, a plurality of feature maps having different sizes for each layer are generated at each of the plurality of layers of the neural network for machine tasks. Based on the plurality of feature maps generated in the plurality of levels, generating a plurality of intermediate feature maps having uniform sizes, generating an image based on the plurality of intermediate feature maps, Generate a bitstream by encoding the image, In the generation of the intermediate feature map, either the first method or the second method is selected for at least one of the multiple levels. In the first method, the multiple feature maps generated in the level are used to generate the multiple intermediate feature maps, and in the second method, the multiple intermediate feature maps are generated without using the multiple feature maps generated in the level.

16. The encoding device according to claim 15, wherein: In the generation of the bitstream, the bitstream is generated by further encoding a parameter indicating which of the first method and the second method is selected for the at least one layer.

17. The encoding device according to claim 16, wherein: For each of the plurality of levels, either the first method or the second method can be selected. The parameter includes a plurality of flags corresponding to the plurality of levels, The value of each of the plurality of flags indicates which of the first method and the second method is selected for the corresponding layer.

18. The encoding device according to claim 16, wherein: For each of the plurality of levels, either the first method or the second method can be selected. The parameter includes a plurality of bits of data corresponding to the plurality of levels, The value of each of the plurality of bits indicates which of the first method and the second method is selected for the corresponding layer.

19. The encoding device according to claim 16, wherein: For each of the plurality of levels, either the first method or the second method can be selected. A plurality of patterns are predefined regarding which combination of the first method and the second method is selected for each of the plurality of levels. Selecting a mode from among the plurality of modes, The parameters include information for determining the one mode.

20. The encoding device according to claim 16, wherein: In generating the bitstream, the parameter is stored in a supplemental enhancement information SEI area of ​​the bitstream.

21. The encoding device according to claim 15, wherein: The size of the intermediate feature map is equal to the size of the smallest feature map among the plurality of feature maps generated in the plurality of levels, When the first method is selected for the at least one level, in generating the intermediate feature map, the size of the feature map generated in the at least one level is reduced to the size of the intermediate feature map.

22. The encoding device according to claim 15, wherein: The size of the intermediate feature map is equal to the size of the largest feature map among the plurality of feature maps generated in the plurality of levels, When the first method is selected for the at least one level, in generating the intermediate feature map, the size of the feature map generated in the at least one level is enlarged to the size of the intermediate feature map.

23. The encoding device according to claim 15, wherein: For each of the plurality of levels, either the first method or the second method can be selected. The greater the number of levels in which the second method is selected among the multiple levels, the smaller the number of the multiple intermediate feature maps.

24. The encoding device according to claim 15, wherein: For each of the plurality of levels, either the first method or the second method can be selected. In the generation of the intermediate feature map, when the size of the target of the detection object in the machine task is smaller than a threshold, or when local information is required in the machine task, the second method is selected at least for the level with the smallest size of the feature map generated among the multiple levels.

25. The encoding device according to claim 15, wherein: For each of the plurality of levels, either the first method or the second method can be selected. In the generation of the intermediate feature map, when the size of the target of the detection object in the machine task is above a threshold, or when global information is required in the machine task, the second method is selected at least for the level with the largest size of the feature map generated among the multiple levels.

26. A decoding method, The decoding device performs the following processing: Generates an image by decoding the bitstream, Based on the image, generating a plurality of intermediate feature maps having uniform sizes, Based on the plurality of intermediate feature maps, a plurality of feature maps having different sizes for each layer are generated at each of the plurality of layers of the neural network for the machine task, In generating the feature map, either the first method or the second method is selected for at least one of the multiple levels. In the first method, the multiple feature maps of the level are generated using the multiple intermediate feature maps, and in the second method, the multiple feature maps of the level are generated using the multiple feature maps generated in a level different from the level among the multiple levels without using the multiple intermediate feature maps.

27. A method of encoding, The encoding device performs the following processing: Based on the input image to be processed, a plurality of feature maps having different sizes for each layer are generated at each of the plurality of layers of the neural network for machine tasks. Based on the plurality of feature maps generated in the plurality of levels, generating a plurality of intermediate feature maps having uniform sizes, generating an image based on the plurality of intermediate feature maps, Generate a bitstream by encoding the image, In the generation of the intermediate feature map, either the first method or the second method is selected for at least one of the multiple levels. In the first method, the multiple feature maps generated in the level are used to generate the multiple intermediate feature maps, and in the second method, the multiple intermediate feature maps are generated without using the multiple feature maps generated in the level.

Images