Pyramid pooling module and method for performing pooling process

Abstract

To provide a module that identifies, with high accuracy, the types, ranges, and positions of elements photographed in an image, and an image processing system and an image processing method using the module.SOLUTION: A module receives input of a feature map extracted from an image, sets the pooling area number of two or more for an image, infers pooling areas in the image corresponding to the pooling area number, performs pooling processing for each of the pooling areas, and causes a convolutional neural network to extract the feature map including area information. The module performs a plurality of times processing of setting the pooling area number larger than before, inferring the pooling areas for each of the set pooling area number, performing pooling processing, and causing the convolutional neural network to extract and output the feature map of the image including the area information for each of the pooling areas. The feature map obtained includes multi-scale context information having a hierarchical structure.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention inputs an image into a computer, performs machine learning, and outputs a feature map that is effective in identifying the type, range, and position of elements captured in the image. Module The present invention relates to an image processing system and method that utilizes this module, and in particular to a technology for improving classification accuracy in semantic segmentation by considering the relationships between elements captured in an image in a hierarchical structure. [Background technology]

[0002] Semantic segmentation is an image recognition analysis method characterized by assigning labels to every pixel in an image and outputting an image with regions segmented by label. The labels here refer to information assigned by classifying objects in the input image. For example, if an image containing three types of objects - people, furniture, and background - is input, and semantic segmentation is used to identify them, different labels are assigned to the people, furniture, and background, and the contours of each object are defined, resulting in an output image in which the objects are classified.

[0003] Semantic segmentation models typically use convolutional neural networks to identify objects in images. Semantic segmentation models, using convolutional neural networks, are becoming increasingly capable of accurately identifying diverse objects with similar appearances. Analytical techniques that collect "contextual information" have been proposed to improve the accuracy of identifying diverse objects. In the field of image recognition, contextual information refers to information related to each element in a photograph within the scene or the image that captures that scene. For example, suppose we are trying to predict a segmentation map for an image containing the sky, sea, boat, and airplane. At a pixel-by-pixel level, the sky and sea have similar luminance and chromaticity, making them difficult to distinguish. However, if an object identified as a yacht is located nearby, we can infer that the pixel represents the sea. Similarly, if an object identified as an airplane is located nearby, we can infer that the pixel represents the sky. Thus, contextual information is information related to each element and improves identification performance.

[0004] Known technologies for collecting contextual information include the Pyramid Pooling Module (PPM) disclosed in Non-Patent Document 1 and the Atrous Spatial Pyramid Pooling Module (ASPPM) disclosed in Non-Patent Document 2.

[0005] Pooling here refers to the process of combining information within a region into a single vector. Generally, max pooling is used to extract the maximum value within a region, and average pooling is used to extract the average value within a region.

[0006] The pyramid pooling module (PPM) in Non-Patent Document 1 is a model that applies multiple pooling processes in parallel to an extracted feature map, each with a different pooling size, which is the area where pooling is performed. The Atros Spatial Pyramid Pooling Module (ASPPM) in Non-Patent Document 2 is a model that applies dilated convolution and global average pooling with different dilated rates in parallel to extracted feature maps. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Pyramid Scene Parsing Network" by Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia, 2017, published on the website https: / / arxiv.org / pdf / 1612.01105 [Non-patent document 2] Liang-Chieh, Chen George, Papandreou Florian, Schroff Hartwig Adam, (Google Inc.) "Rethinking Atrous Convolution for Semantic Image Segmentation," 2017, available at https: / / arxiv.org / pdf / 1706.05587 Summary of the Invention [Problem to be solved by the invention]

[0008] In conventional image processing using PPM and ASPPM, pooling is performed using fixed shapes and sizes without considering the shape and scale (size, the ratio of enlargement to the real thing) of objects in the input image. This can result in a lack of meaningful correlation between sampled features, and insufficient correlation between features of different sizes. As a result, it can be difficult to capture the confusion between similar hierarchies and the dependency relationships when similar objects are located far apart.

[0009] In semantic segmentation, multi-scale context information, i.e., "different ranges of surrounding information for each pixel," is essentially important. By using multi-scale context information in predicting the segmentation map, it is possible to take into account multiple ranges of surrounding information for each pixel, resulting in a more accurate prediction result. [Means for solving the problem]

[0010] The present invention provides a hierarchical pyramid pooling module (HPPM) that performs pooling processing taking into account the shape and size of objects in an input image and sufficiently associates sampled features to output contextual information for highly accurate image recognition. Specifically, the hierarchical pyramid pooling module of the present invention sets two or more pooling regions for a base feature map extracted from an image, infers a number of pooling regions corresponding to the number of pooling regions and corresponding to the shapes and / or sizes of objects in the image, performs dynamic region pooling (DRP) using the inferred pooling regions, and outputs a feature map containing context information for image recognition. Furthermore, the hierarchical pyramid pooling module sets a new number of pooling regions greater than the number already set for the feature map output using the dynamic region pooling process, infers pooling regions corresponding to the newly set number of pooling regions, performs dynamic region pooling, and repeats this process one or more times to output a feature map containing context information for image recognition.

[0011] The hierarchical pyramid pooling module of the present invention preferably has a configuration in which multiple dynamic region pooling units are arranged in parallel. Each dynamic region pooling unit uses a convolutional neural network to estimate pooling units whose number corresponds to a set number of pooling units and to perform pooling on the estimated pooling units. When feature maps obtained by other dynamic region pooling units exist, the dynamic region pooling unit combines the base feature map with the feature maps obtained by the other dynamic region pooling units to output a final image.

[0012] The present invention also provides an image processing system for identifying the type, extent, and location of an element captured in an image, comprising a backbone network, a hierarchical pyramid pooling module, and a decoder. The backbone network inputs an image, extracts features for each pixel using a convolutional neural network, and outputs a feature map. The hierarchical pyramid pooling module receives a feature map, sets two or more pooling regions for the feature map, infers pooling regions of the feature map corresponding to the number of pooling regions, performs pooling processing to combine features of the feature map for each pooling region into one vector, and extracts and outputs a feature map of the image including region information for each pooling region using a convolutional neural network.The hierarchical pyramid pooling module further sets a new number of pooling regions greater than the already set number of pooling regions, infers pooling regions corresponding to the newly set number of pooling regions, performs pooling processing, and extracts and outputs a feature map of the image including region information for each pooling region using a convolutional neural network, repeating this processing one or more times, and outputs the resulting feature map to a decoder. The decoder decodes the feature map output from the hierarchical pyramid pooling module using a convolutional neural network and outputs the result as the element identification result.

[0013] The hierarchical pyramid pooling module of the image processing system preferably has a configuration in which multiple dynamic region pooling processors with different numbers of regions are arranged in parallel. A dynamic region pooling processor with a number of regions corresponding to the set number of pooling regions causes a convolutional neural network to estimate pooling regions, perform pooling processing, and output a feature map including pooling region information. When feature maps obtained by other dynamic region pooling processors exist, the dynamic region pooling processor can combine the feature map output from the backbone network with the feature maps obtained by the other dynamic region pooling processors to output a final image. [Effects of the Invention]

[0014] Our hierarchical pyramid pooling module can build a semantic hierarchical structure within an image and consider the relationships between layers, thereby enabling us to correctly identify similar classes and capture the dependencies between multiple similar objects located far apart.

[0015] The image processing system of the present invention including the hierarchical pyramid pooling module and the image processing method of the present invention using the hierarchical pyramid pooling module have improved ability to identify objects in an image compared to conventional methods. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram showing a conceptual configuration of an image processing system according to the present invention. [Figure 2] FIG. 2 is a flowchart showing the processing contents of the dynamic region pooling processing unit of the present invention. [Figure 3] FIG. 3 is a diagram showing an example of a semantic hierarchical structure within an image. [Figure 4] Fig. 4 is a diagram comparing the pooling area set by the conventional technology with the pooling area estimated by the pyramid pooling module of the present invention. The lower part of Fig. 4 shows an example of the pooling area of ​​the hierarchical pyramid pooling module of this embodiment, indicated by the hatched area. The upper part of Fig. 4 shows the pooling area obtained by the pyramid pooling module of the conventional technology. The middle part of Fig. 4 shows the pooling area estimated by the Atros spatial pyramid pooling module of the conventional technology. [Figure 5] Figure 5 shows an example of the classification results output by the image processing system and image processing method of the present invention. In the figure, Input represents the input image, Ground Truth represents the ground truth data, HSPNet represents the prediction results of the hierarchical scene parsing network, which is the image processing system of the present invention, and r represents the number of regions. DETAILED DESCRIPTION OF THE INVENTION

[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an image processing system and an image processing method according to the present invention will be described in detail below with reference to the accompanying drawings.

[0018] The image processing system and method of this embodiment use a hierarchical pyramid pooling module to identify and capture the semantic hierarchical structure of the scene captured in the image to collect multi-scale contextual information. The "semantic hierarchical structure" here refers to a pyramid-shaped hierarchical structure obtained by subdividing the objects captured in the image into more specific categories, from broad categories with comprehensive meanings to smaller categories with increasingly more specific meanings, each of which has its own meaning.

[0019] The convolutional neural network used in this embodiment is a mathematical model that mimics the system of neurons, which are human nerve cells, and is embodied as a network structure on one or more computers. In this embodiment, various processes are performed by the convolutional neural network using Python (registered trademark), which is a scripting language, but the language used is not limited to Python (registered trademark), and any other scripting language can be used.

[0020] 1 is a block diagram showing a conceptual configuration of an image processing system according to the present invention, which includes a backbone network, a hierarchical pyramid pooling module, and a decoder.

[0021] The backbone network is a network that extracts features from an image. In this embodiment, the following feature quantities are output from an input image:

number

[0022] As an example, the backbone network of this embodiment uses ResNet101 (Mathworks (registered trademark)), a convolutional neural network with a depth of 101 layers, but other neural networks can also be used. Preferably, the pooling process for the last two downsampling steps is replaced with dilated convolution, which can reduce loss of spatial information due to a decrease in resolution.

[0023] The hierarchical pyramid pooling module inputs a feature map, which is a collection of pixel-by-pixel features extracted by the backbone network, and collects multi-scale contextual information that captures the semantic hierarchical structure of the scene captured in the image, and outputs it as a feature map.

[0024] Figure 3 shows an example of capturing the semantic hierarchical structure of a scene. The scene captured in an image can be given different meanings by changing the number of regions to be decomposed. For example, if the image of the room in the top row of Figure 3 is divided into three by setting the number of regions to three, it is possible to roughly classify the ceiling, walls, and floor, as shown in the middle row of Figure 3. If a larger number of regions is set and the image is decomposed, it is possible to identify and classify the chairs, desks, and people included in the floor area in more detail. In this way, by decomposing an image into multiple different numbers of regions, it is possible to form a "semantic hierarchical structure" in which each layer contains items whose individual contents are classified with meaning, and there are relationships between the layers.

[0025] To obtain a semantic hierarchical structure, the hierarchical pyramid pooling module sets two or more pooling regions for a feature map, infers pooling regions of the feature map corresponding to the number of pooling regions, performs pooling processing to combine features of the feature map for each pooling region into one vector, and causes a convolutional neural network to extract a feature map of the image including region information for each pooling region. This processing involves setting multiple different numbers of pooling regions, inferring pooling regions for each different number of pooling regions, performing pooling processing, and causing a convolutional neural network to extract and output a feature map of the image including region information for each pooling region. By performing this processing multiple times, it is possible to output multi-scale context information with a hierarchical structure that has high classification accuracy.

[0026] The hierarchical pyramid pooling module has a configuration in which multiple dynamic region pooling units with different numbers of regions are arranged in parallel. When the number of pooling regions is set to a different value, the dynamic region pooling unit with the number of regions corresponding to the set number of pooling regions causes the convolutional neural network to estimate pooling regions, perform pooling processing, and output a feature map including the pooling region information. The dynamic region pooling unit infers pooling regions to perform pooling according to the shape and size of the object in the input image.

[0027] In the hierarchical pyramid pooling module of this embodiment, the number of regions is inferred from a pooling region where the number of regions is set to 2, and then the number of regions is increased to 4, 8, and 16. Using the corresponding dynamic region pooling processor, more detailed pooling regions are repeatedly inferred, taking into account the relationship between layers, to construct a feature map that captures the hierarchical structure of the scene. During multiple inferences, features including region information inferred by the dynamic region pooling processor of each layer are combined to predict and learn a segmentation map, and each inferred pooling region learns the semantic hierarchical structure.

[0028] The processing contents of the dynamic region pooling processor are shown in Figure 2. The dynamic region pooling processor outputs the feature values ​​input from the backbone network,

number

number

number

[0029] The dynamic region pooling processing unit TIFF0007737127000015.tif2099 and The pooling area is inferred by applying a convolution and softmax function to the combination of TIFF0007737127000016.tif2099. The number of pooling areas set during inference is set to "2" for the first inference. If an inference has already been made, by setting the number of areas to be larger than the previous inference, it is possible to use the features of the pooling area, which is the result of the previous inference, as context information and infer area information of a hierarchy classified into more detailed meanings. Next, the array is reshaped and transposed to determine the area,

number

[0030] The dynamic region pooling unit performs pooling on the inferred pooling region, and outputs image features by performing normalization, convolution, and upsampling. The process is described below.

[0031] The dynamic region pooling processor uses the features from the backbone network. Reshape the TIFF0007737127000019.tif2099.

number

[0032] The pooling process is performed in the following order: The dynamic region pooling unit calculates the feature values ​​included in each region as shown in Equation (1). Collect TIFF0007737127000023.tif24125.

number

[0033] Next, as a normalization process, Each area of ​​TIFF0007737127000025.tif24125 Feature vector of TIFF0007737127000026.tif24125 TIFF0007737127000027.tif24125 is the inferred region Each area of ​​TIFF0007737127000028.tif24125 Each element of TIFF0007737127000029.tif24125 Sum of TIFF0007737127000030.tif24125

number

number

[0034] The region information P is transposed TIFF0007737127000037.tif27133 and By upsampling using TIFF0007737127000038.tif27133, it is possible to construct features with region information. First, TIFF0007737127000039.tif24133 and By multiplying TIFF0007737127000040.tif24133, we can get the average feature vector for each region as a feature map. Revert to TIFF0007737127000041.tif23144.

number

[0035] lastly, Reshape TIFF0007737127000043.tif22144,

number

[0036] In this way, by arranging dynamic region pooling processing units with different numbers of regions in parallel, the feature map obtained can be combined with the output of the backbone network to collect multi-scale contextual information that captures the semantic hierarchical structure of the scene captured in the image, label each pixel, and output as a feature map.

[0037] The hierarchical pyramid pooling module combines the features output by the dynamic region pooling processor and inputs them to the decoder, which decodes and outputs the element identification results using a convolutional neural network.

[0038] Figure 5 shows the results of identifying and visualizing elements in an image using the image processing system of this embodiment. The PASCAL-Context Dataset (https: / / cs.stanford.edu / ~roozbeh / pascal-context / ) was used as the evaluation dataset. As shown in Figure 5, as the number of pooling regions increases from 2 to 4, 8 to 16, the elements are more accurately identified.

[0039] Table 1 shows the results of a comparison of the accuracy of a conventional image recognition system and the image processing system of this embodiment.

[0040] [Table 1]

[0041] As shown in Table 1, it was confirmed that the image classification accuracy obtained by the image processing system and image processing method of this embodiment is capable of collecting feature representations that are higher than those obtained conventionally.

[0042] Table 2 shows the results of a comparison of the computational efficiency of a conventional image recognition system and the image processing system of this embodiment. The items compared are the number of module parameters, GPU memory consumption, and computational complexity (FLOPs) when a 1 x 2048 x 128 x 128 tensor is used as input.

[0043] [Table 2]

[0044] It has been confirmed that the image processing system and image processing method of this embodiment can extract features with high accuracy and efficiency equivalent to that of conventional image recognition systems.

[0045] The image processing system, image processing method, and module configuration described in this embodiment can be modified as appropriate. For example, the model and configuration of the convolutional neural network used are not particularly limited, and any language for using the convolutional neural network can be used. In the embodiment, the processing procedure of the dynamic region pooling processor is described as setting the number of regions small and gradually increasing them. However, since learning can be performed by changing the number of regions when constructing a hierarchy, it is possible to construct a hierarchy according to the number of objects and the number of classes included in the images of the training data. [Industrial Applicability]

[0046] The module, image processing system, and image processing method of the present invention can be widely applied to fields such as autonomous driving, robot vision, and virtual reality.

Claims

1. A hierarchical pyramid pooling module that causes a computer to perform a dynamic region pooling process, comprising: The hierarchical pyramid pooling module Extract a base feature map from the image, a module that causes the computer to execute a dynamic region pooling process on the base feature map, thereby outputting a feature map including context information for image recognition; The dynamic region pooling process Set two or more pooling regions for the base feature map; inferring a number of pooling regions corresponding to the shape and / or size of objects in the image, the number of pooling regions corresponding to the number of pooling regions; A process of causing a computer to execute a pooling process using the inferred pooling area, a pyramid pooling module that causes a computer to execute a process of setting a number of pooling regions that is greater than a previously set number of pooling regions each time the number of repetitions increases, when repeating a process of outputting a feature map including context information for the output feature map including context information one or more times.

2. A method for causing a computer to perform a pooling process using a hierarchical pyramid pooling module, comprising: Extract a base feature map from the image, A method for outputting a feature map including context information for image recognition by performing a dynamic region pooling process on the base feature map, comprising: The dynamic region pooling process includes: Set two or more pooling regions for the base feature map; inferring a number of pooling regions corresponding to the shape and / or size of objects in the image, the number of pooling regions corresponding to the number of pooling regions; A step of performing a pooling process using the inferred pooling region, A method characterized by, when repeating a processing step of outputting a feature map including context information for the output feature map including context information one or more times, setting a number of pooling regions greater than the number of pooling regions that has already been set each time the number of repetitions increases.

Images