IMAGE IDENTIFICATION DEVICE, IMAGE IDENTIFICATION METHOD, AND PROGRAM

Abstract

To provide an image identification device capable of computing significance more accurately, a method of performing semantic segmentation, and a program.SOLUTION: An image identification device is provided, comprising an image acquisition unit for acquiring an image, a feature value extraction unit configured to extract multiple feature values of the acquired image, a feature map generation unit configured to generate a feature map for each of the multiple feature values, and a multiplication unit configured to multiply each feature map by a weight coefficient which is an arbitrary positive value representing significance of the feature.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image recognition device, a method for performing semantic segmentation, and a program. [Background technology]

[0002] Semantic segmentation is a fundamental yet difficult problem that aims to identify the category of each pixel, and high accuracy is required for building systems such as autonomous mobile robots and self-driving cars. In a real environment, in addition to robustness against factors such as the scale of the object, the lighting environment, and occlusion, the ability to distinguish different categories with similar appearances is required. Therefore, in order to achieve high-precision recognition, it is necessary to acquire and select features with high discriminative properties (see, for example, Patent Documents 1 and 2).

[0003] In addition, deep learning-based semantic segmentation has been greatly improved by combining context information. Recently, a technique for understanding context has been proposed that modifies the features obtained from a feature extractor (backbone) using pixel-level or category-level similarity. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-128804 A [Patent Document 2] Republished WO2008 / 129881 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, the features input to the network responsible for the final classification are treated equally for each feature map, which makes it difficult to distinguish between feature maps. Also, in the conventional technology that enhances feature maps using residual structures, the structure only allows enhancement, which makes it difficult to distinguish between feature maps.

[0006] The present invention has been made in consideration of the above problems, and has an object to provide an image recognition device, a method for performing semantic segmentation, and a program that can calculate importance more accurately than conventional methods. [Means for solving the problem]

[0007] (1) In order to achieve the above object, an image recognition device according to one aspect of the present invention includes an image acquisition unit that acquires an image (X), a feature amount extraction unit that extracts a plurality of feature amounts of the acquired image, and a feature map (X i A feature map generating unit generates a feature map of the feature map, and a weighting coefficient (a i ) and a multiplication unit for multiplying the

[0008] (2) In the image recognition device according to one aspect of the present invention, the weighting coefficient (a i ) may be calculated by a process of convolving the image (X) to create a convolutional layer, a process of applying a ReLU function to the convolutional layer to calculate a feature F, and a process of applying a Global Average Pooling (GAP) layer to the feature F.

[0009] (3) In order to achieve the above object, an image recognition device according to one embodiment of the present invention includes an image acquisition unit that acquires an image, a feature extraction unit that extracts multiple features of the acquired image, a creation unit that creates a feature map by convolution processing for each of the multiple features, and a weighted feature generation unit that calculates modified features by convolution processing for the feature map, performs a global average pooling process on the calculated modified features to aggregate contexts, generates attention, which is a weighting coefficient for each channel, and multiplies the generated attention by the feature map to apply enhancement and attenuation weighting to the multiple feature maps to generate weighted features.

[0010] (4) In addition, the image recognition device according to one aspect of the present invention may further include a first loss calculation unit that performs convolution and upsampling processing on the weighted features to calculate an output and compares the calculated output with teacher data to calculate a first loss, and a second loss calculation unit that performs convolution and upsampling processing on the feature map to calculate an output and compares the calculated output with teacher data to calculate a second loss, and may calculate an overall loss function from the first loss and the second loss, and may learn the weighting coefficients using the calculated loss function.

[0011] (5) In order to achieve the above object, a method for performing semantic segmentation according to one aspect of the present invention is a method for performing semantic segmentation of an image (X) using a neural network system, the method including a process of inputting the image, a process of extracting a plurality of feature amounts of the acquired image, and a process of generating a feature map (X) for each of the plurality of feature amounts of the image. i ) and weighting coefficients (a i )

[0012] (6) In order to achieve the above object, a program according to one aspect of the present invention is a program for causing a computer to acquire an image, extract a plurality of feature amounts of the acquired image, and generate a feature map (X i ) is created, and a weighting coefficient (a i ) is multiplied. Effect of the Invention

[0013] According to (1) to (6), it is possible to calculate the importance more accurately than before. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of an image recognition device including a semantic segmentation device according to an embodiment. [Diagram 2] FIG. 1 illustrates an example of a context-incorporating network structure. [Diagram 3] 1 is a schematic configuration diagram of a CFANet according to an embodiment. [Figure 4] 1 is a computation graph that simplifies the CFANet according to the present embodiment. [Diagram 5] This figure shows the evaluation results on the PASACL VOC 2012 validation set. [Figure 6] This figure shows the evaluation results on the PASACL VOC 2012 test set. [Figure 7] FIG. 13 is a diagram showing an example of a visualization result of cosine similarity. [Figure 8] 1 is a flowchart of an example of a processing procedure of an image identification device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0016] [Outline of the embodiment] By using features multiplied by the importance of the feature maps, a mechanism is provided to increase or decrease the influence of each feature map, making it easier to distinguish between feature maps that contribute to the output. When calculating the importance of feature maps, a Global Average Pooling layer is used to capture the context of the entire image. A head network that performs auxiliary inference is installed in the branch that calculates the importance, resulting in a structure that calculates more accurate importance.

[0017] In this embodiment, we use the ContextawareFeatureAttentionNetwork(CFANet) to solve the problem. In CFANet, we aggregate contexts using Global Average Pooling(GAP) to generate attention for each channel. The attention obtained is directly multiplied to the feature maps, and each feature map is weighted in both directions, that is, boosting and attenuating. Therefore, it is possible to obtain more discriminability than conventional methods.

[0018] [Configuration example of image identification device 1] FIG. 1 is a block diagram showing a configuration of an image recognition device 1 including a semantic segmentation device 10 according to the present embodiment. As shown in FIG. 1, the image recognition device 1 includes an image acquisition unit 11, a feature extraction unit 12, a semantic segmentation device 10, and a visualization unit 30. The semantic segmentation device 10 includes a feature acquisition unit 21, a multiplication unit 22 (feature map creation unit, weighted feature generation unit), a first convolution layer 23 (first loss calculation unit), a second convolution layer 24 (feature map creation unit, creation unit), a third convolution layer 25 (feature map creation unit, weighted feature generation unit), a GAP unit 26 (feature map creation unit, weighted feature generation unit), and a fourth convolution layer 27 (second loss calculation unit). The visualization unit 30 includes a Head 31 (first loss calculation unit), an auxiliary Head 32 (second loss calculation unit), a teacher label provision unit 33, and a similarity map creation unit 34.

[0019] [Example of a network structure that incorporates context] An example of a network structure that incorporates a context will now be described. Figure 2 is a diagram showing an example of a network structure that incorporates a context. Image g110 in Fig. 2 is an example of a structure in which features are combined in the channel direction. Image g120 in Fig. 2 is a structure in which a context is taken in as a residual feature. Image g130 in Fig. 2 is an example of a structure in which a feature map is modulated in both directions, enhancement and attenuation, taking into account the context of this embodiment.

[0020] [Network structure] FIG. 3 is a schematic diagram of the CFANet according to this embodiment. In this embodiment, ResNet is used as the backbone. As in Reference 1 (Zhao, H., Shi, J., Qi, X., Wang, X. and Jia, J., “Pyramid Scene Parsing Network”, CVPR, 2017), dilated convolution is applied to the last two blocks of ResNet to suppress the reduction in resolution to 1 / 8 of the input image. The feature F0 obtained from the backbone is propagated to the CFA module indicated by the dashed line. F0 is converted to F1 through the convolution layer and then propagated in two directions. The first is a network for generating channel-level attention. In this way, the feature map obtained by the Backbone is propagated to the CFA module, where it is weighted for each channel and then imported into the Head.

[0021] After passing through the Convolution-BatchNorm-ReLU layer, F1 aggregates the global context by GAP (Global Average Pooling) and converts it into attention a for each channel. Specifically, attention ac for a certain channel c is expressed as follows:

[0022]

number

[0023] Here, H is the height, W is the width, α∈R is the scale factor adapted by learning, and F1' is the feature after ReLU (modified feature). Also, u=(c,v,u), v is the position in the row direction, and u is the position in the column direction, and the sum of them is taken over the entire feature map. The feature F2, which is weighted taking into account the context, is expressed by the following equation (2).

[0024]

number

[0025] The obtained F2 is input to the Head to obtain the output Y. The other propagation direction of F1 is the auxiliary head, which obtains the auxiliary output Y'. Each output is compared with the teacher label T, and the loss Lmain (first loss), and L aux (2nd loss) is obtained. The overall loss function is defined as follows:

[0026]

number

[0027] [Characteristics of CFA Module] Next, the equivalence of CFANet and the mixed expert model, and the effect of the auxiliary head branching from the CFA module will be described. To simplify the description, consider the computation graph of the simplified CFANet shown in FIG. 4. FIG. 4 is a computation graph of the simplified CFANet according to this embodiment. In FIG. 4, all activation functions are linear functions. Also, the convolution layer represented by (c) in FIG. 3 is omitted. These are assumptions that do not lose generality in the discussion. In FIG. 4, each F represents a feature amount, and each W represents a weight matrix of the convolution. Also, a is the attention for each channel, and X, Y, and T are the correct label corresponding to the input image, the estimation result, and the input, respectively. Variables for the auxiliary head are represented with '.

[0028] [Mixed Expert Model] A mixed experts model consists of C experts (E,…,E C―1 ) and a gating network G that generates C-dimensional weights. For input x, the output y is given by the following equation (4).

[0029]

number

[0030] Here, G(x) i is the expert E with index i i is the weight assigned to To confirm the equivalence of the mixed expert model and CFANet, we first consider the weight matrix in the following equation (5) that appears in the Head of Figure 4.

[0031]

number

[0032] In equation (5), k2 is the kernel size, C is the number of input channels, and Cout is the number of output channels. The final output Y can be transformed into the following equation (6) by using equation (2).

[0033]

number

[0034] Here, we define the position dependence in the kernel as k=(k v ,k u ) as shown. Comparing equation (4) and equation (6), we can see that CFANet is equivalent to a mixture of experts model, which enables CFANet to focus on features specific to the object contained in the input image for classification.

[0035] [Effects of auxiliary head] We show that the presence of an auxiliary head in the CFANet promotes the learning of the weight W2 in the CFA module. We define the gradient backpropagated from the output Y to node F2 as G F2 , the gradient propagating back from the auxiliary output Y' to node F1 is G F1 (Dotted arrow in Figure 4). The total gradient propagated to node F1 is given by the following equation (7).

[0036]

number

[0037] As a result of generating attention for each channel using GAP, the dependency of u = (c, u, v) appears only in the second term. If we use the chain rule to find the gradient for weight W1, we get the following equation (8).

[0038]

number

[0039] The first term of equation (8) can be approximately written as the following equation (9).

[0040]

number

[0041] In other words, when the auxiliary head is not used (G F1 =0), and from equation (8), k = (k v ,k u ) completely disappears, which hinders the learning of the weight W1. The addition of an auxiliary head restores the dependency, leading to the acquisition of more discriminative features.

[0042] [Experimental Results] In the following explanation, we will explain the results of evaluating CFANet on the PASCAL VOC 2012 dataset (Reference 2; Everingham, M., Eslami, SMA, Van Gool, L., Williams, CKI, Winn, J. and Zisserman, A., “The Pascal Visual Object Classes”, (VOC) Challenge, International Journal of Computer Vision, 2010, p303-338). The evaluation metric used is the average IoU of each class (mIoU).

[0043] PASCAL VOC 2012 is a dataset consisting of 1,464 training images, 1,449 validation images, and 1,456 test images, and contains 21 categories including background classes. In addition to the PASCAL VOC 2012 dataset, the validation also used the SBD dataset (Reference 3; Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S. and Malik, J., “Semantic Contours from Inverse Detectors”, ICCV, 2011), which is annotated images extracted from the PASCAL VOC 2011 dataset.

[0044] We used SGD for optimization, with momentum set to 0.9 and weight decay set to 0.0001. For the learning rate schedule, we followed the example of Reference 1 and set the initial learning rate to (1-iter / (total-iter) 0.9 We used the method of multiplying by . We performed 50 epochs of pre-training on the SBD dataset, and then performed 50 epochs of fine-tuning on the PASCAL VOC 2012 dataset using the weights as the initial values. The learning rates for pre-training and fine-tuning were 0.0015 and 0.00015, respectively. To augment the data, we applied random horizontal flipping, random scaling in the range [0.5,2.0], and random cropping with a size of 513 × 513. In addition to evaluation at a single scale, we also evaluated the results obtained from left-right flipped (Flip) and multi-scaled (MS) input images.

[0045] The effect of the three elements incorporated into CFANet is separated using the PASCAL VOC 2012 validation set. First, we evaluated FCN with ResNet50 as the backbone as a baseline, and obtained an accuracy of 71.38%. In contrast, we applied Multi-Grid (Reference 4; Chen, L.-C., Papandreou, G., Schroff, F. and Adam, H., “Rethinking Atrous Convolution for Semantic Image Segmentation”, arXiv:1706.05587, 2017), which sets the dilation of the three 3×3 convolution layers in the final block of ResNet to (4,8,16), and the accuracy improved to 77.90%. This can be interpreted as an effect obtained by expanding the receptive field.

[0046] Adding the CFA module, which is the method of this embodiment, raises the performance to 78.90%. This is considered to be the result of the CFA module being able to properly estimate the importance of each channel based on the global context. Furthermore, by adding the above-mentioned auxiliary head, the weights in the CFA module are effectively learned, and the performance improves to 79.46%. For further improvement, by changing the backbone to ResNet101, the performance improved to 81.54%, and by performing inference using Flip and MS, a performance of 82.33% was finally achieved.

[0047] Figure 5 shows the evaluation results on the PASACL VOC 2012 validation set. In Figure 5, MG stands for Multi-Grid, CFA stands for CFA module, Aux stands for auxiliary head, and MS+Flip stands for multi-scale and left-right flipped input.

[0048] [Visualization of feature similarity] To make it easier to understand the effect of the CFA module, the visualization unit 30 visualizes the Cosine similarity in the feature space between the target pixel and other pixels. As an object of visualization, we focus on the feature input to the Head (F2 in FIG. 3). In addition, as a comparison object, we also visualize the similarity to the feature in the FCN just before the Head.

[0049] FIG. 6 is a diagram showing an example of a visualization result of the cosine similarity. A square g600 (g601 to g603 from the top) in FIG. 6 represents an input image, and pixels that are the subject of similarity calculations are marked with crosses (g641 to g643). The square g610 in FIG. 6 (g611 to g613 from the top) is the correct label corresponding to the input image. A rectangle g620 in FIG. 6 (g621 to g623 from the top) and a rectangle g640 in FIG. 6 (g631 to g633 from the top) respectively indicate similarity maps of the baseline FCN and the method CFANet of the embodiment.

[0050] The similarity map shows that pixels closer to red (g651~g656) have high similarity, and pixels closer to blue (g661~g663) have low similarity. Although FCN shows high similarity to the same object region as the target pixel, it also shows relatively high similarity (green~yellow) (g671~g673) to regions other than the same object, such as the background. In contrast, it can be seen that the CFANet of the embodiment is more discriminative of regions unrelated to the target region. This shows that while FCN treats unimportant feature maps with equal contribution to others, CFANet reduces the contribution of such feature maps and increases the contribution of important channels. This effect leads to improved performance and improved discrimination.

[0051] [Performance comparison with existing methods] The performance of CFANet was also compared with other existing methods. The comparison was performed using the test set of PASCAL VOC 2012. This test set was given only the input image, and the results of inference made by the model were sent to the evaluation server for evaluation, making it a fair evaluation method. For evaluation on the test set, the model trained on the SBD dataset was fine-tuned using the train+validation set of PASCALVOC 2012. The results are shown in Figure 7. Figure 7 shows the evaluation results on the PASACL VOC 2012 test set. The CFANet of the embodiment achieved an accuracy of 84.5%, which exceeds that of existing methods.

[0052] As described above, in this embodiment, we used CFANet, which has a mechanism for weighting feature maps using channel-level attention. From the experimental results, we were able to confirm by visualizing the feature maps that CFANet can handle feature maps more discriminatively than conventional methods. In addition to improving discriminability, we also achieved a significant improvement in performance, achieving accuracy that exceeds that of existing methods on the PASCAL VOC 2012 test set.

[0053] An example of a network structure incorporating context will now be further described with reference to Figure 2, where we focus on channel dimension dependency. In a structure in which the features of an image g110 are combined in the channel direction, the features extracted from the image by a backbone g111 are input to a context module g112. In this structure, the features are combined in the channel direction regardless of their importance.

[0054] The backbone is a ResNet (Reference 1) pre-trained on ImageNet (Reference 5; Deng, J., Dong, W., Socher, R., Li, LJ, Li, K., Fei-Fei, L., “Imagenet: A large-scale hierarchical image database”, In: CVPR09, 2009). In this embodiment, for example, following PSPNet, the first 7×7 convolution is replaced with a 3×3 convolution layer, and the last two blocks of ResNet use dilated convolutions, so the output stride of the feature map is 8.

[0055] In a structure that incorporates the context of an image g120 as a residual feature, the features extracted from the image by the backbone g121 are input to a context module g122 and a calculation unit g123. In this structure, important features are emphasized and combined in the channel direction. The calculation unit g123 (element-wise summation) performs element-wise summation of the features obtained by the backbone g121 and the context module g122. In other words, the residual format expands and enhances the representation of each pixel by aggregating pixel-by-pixel context.

[0056] In a structure that modulates the feature map in both directions, strengthening and attenuating, taking into account the context of the image g130, the features extracted from the image by the backbone g131 are input to the context module g132 and the calculation unit g133. In this structure, the features with high importance are strengthened and the features with low importance are attenuated, and then combined in the channel direction. The calculation unit g133 (Channel-wise multiplication) calculates the product for each channel. In this configuration, the weight of each feature map can be adjusted by focusing on the features. Therefore, in this configuration, it is possible to make related features easier to distinguish.

[0057] The goal of semantic segmentation is to assign a semantic category to each pixel. In semantic segmentation, the more categories there are, the more ambiguous the classes become. Therefore, models need to be trained to select more discriminative features for higher quality image segmentation. However, in the structure (g110) that combines features in the channel direction, these aggregated features are treated as equally important by the head network, making it difficult to identify a specific feature that is more distinctive.

[0058] State-of-the-art context modeling approaches use pixel-level similarity maps to refine backbone features. A residual formalism is adopted in the architecture that captures context as residual features (g120). However, this architecture limits the discriminatory power of features, since they can only be enhanced by selected features.

[0059] For this reason, in this embodiment, a CFA module (Context-aware Feature Attention Network (CFANet)) having the configuration shown in FIG. 3 is used. In CFANet, we introduce a Context-aware Feature Attention (CFA) module that adaptively adjusts the importance of individual features while taking into account the context. Taking advantage of global context is essential for accurate segmentation. Therefore, in this embodiment, we use Global Average Pooling (GAP) to aggregate global features and directly generate channel-wise attention. According to this configuration, each feature map can be strengthened or weakened by the corresponding attention weight, as shown in image g130 in Fig. 2. As a result, according to this embodiment, the attention weight can be a positive value, so that each feature can be treated more distinctly than with other methods.

[0060] Next, the schematic configuration of CFANet will be further explained with reference to FIG. Each feature map in the backbone network represents certain features of objects or objects present in the input image. In order to distinguish characteristic patterns that correspond to the target object category, it is necessary to give more weight to the corresponding features based on the scene context. Therefore, in this embodiment, we introduce a CFA module to perform such reprioritization.

[0061] The dimensions of the acquired image X (g210) are 3 × H0 × W0, where 3 is the number of channels, H is the height of the feature map, and W is the width of the feature map. As mentioned above, for example, ResNet is used for the backbone g211. The backbone g211 is a backbone feature F0(∈R (R is a double underline character, which is the set of all real numbers) C×H×W )(g221), where C is the number of channels.

[0062] The CFA module g220 converts the backbone feature F0 into a feature map F1(∈R C×H×W )g223. Next, the CFA module g220 performs 1×1 convolution (g224) on the feature map F1 to calculate a corrected feature F1′.

[0063] Next, the CFA module g220 performs a global average pooling (GAP) process on the corrected feature F1' (g225) to generate attention a(C×1×1) (g226) for each channel.

[0064] Next, the CFA module g220 uses the backbone feature F0 and the attention a to aggregate global features and generate channel-wise attention (g227). In this process, attention a and the backbone feature F0 are multiplied for each channel to generate weighted features F2(∈R C×H×W )(g228).

[0065] The auxiliary head 32 (g232) performs, for example, convolution and upsampling on the corrected feature F1' to output Y'(C out × H0 × W0) (g244) and output it.

[0066] Head31 (g231) performs, for example, convolution and upsampling on the weighted feature F2 to produce the output Y(C out × H0 × W0) (g241) and output it.

[0067] The image recognition device 1 uses the teacher label T(C out ×H0×W0) (g242) and output Y (g241) to determine the loss. Lmain In addition, the image recognition device 1 calculates the teacher label T(C out × H0 × W0) (g242) and the output Y' (g244) to calculate the loss L aux Calculate.

[0068] Next, the simplified computation graph of CFANet will be further explained with reference to Figure 4, which shows a simplified representation of CFANet. Note that in Figure 4, the convolutional layers (g224, etc.) in Figure 3 are omitted, assuming that all activation functions are linear. Note that in Figure 4, the convolutional layers (g224, etc.) in Figure 3 are omitted for simplification, but the generality is not lost in the explanation.

[0069] An input image X is input to the backbone g310. The backbone g310 performs convolution g341 using a weighting W0 to calculate a backbone feature F0.

[0070] The CFA module g320 performs convolution g344 on the feature map F0 using the weighting W1 to calculate a modified feature F1. The CFA module g320 calculates the attention a by the GAP process (g344). The CFA module g320 calculates a weighted feature F2 by calculating a product for each channel using the attention a and the backbone feature F0 (g345). Note that the CFA module g320 does not perform downsampling in order to maintain the resolution of the feature maps.

[0071] The head 31 (g330) performs convolution (g346) on the weighted feature amount F2 using the weighting W2 to calculate and output the output Y. The auxiliary head 32 (g340) performs convolution (g347) on the corrected feature F1 using the weighting W2' to calculate and output the output Y'.

[0072] The formulas and calculation methods used to calculate the feature amounts, attention, and the like described with reference to Figures 3 and 4 are as described above.

[0073] In this embodiment, a network consisting of a convolutional layer and a dropout layer is used for Head 31 and auxiliary Head 32. In this embodiment, in order to facilitate optimization, for example, a deep supervision head proposed in PSPNet is used in the penultimate block of ResNet. In this way, in this embodiment, main , L aux , L ds The three losses are calculated, and the net loss is calculated, for example, as shown in the following formula (1). ds is a loss for segmentation output using features from the hidden layer of the backbone, and was proposed in PSPNet.

[0074]

number

[0075] In addition, in equation (10), the loss L ds The weight is set to 0.4, but the weight is not limited to this. The image classification device 1 uses a loss function or a total loss to learn attention, which is a weighting coefficient. The image classification device 1 may use a loss function or a total loss to learn the weights used by the second convolutional layer 24, the weights used by the third convolutional layer 25, the weights used by the first convolutional layer 23 and the Head 31, and the weights used by the fourth convolutional layer 27 and the auxiliary Head 32.

[0076] [Processing Procedure] Next, an example of a processing procedure of the image identification device 1 will be described. FIG. 8 is a flowchart of an example of a processing procedure of the image identification device 1 according to this embodiment.

[0077] (Step S1) The image acquisition unit 11 acquires an image. (Step S2) The feature extraction unit 12 performs convolution on the acquired image using the weighting W0 to extract a backbone feature F0.

[0078] (Step S3) The second convolutional layer 24 performs convolution on the backbone feature F0 using the weighting W1 to calculate a feature map F1. (Step S4) The third convolutional layer 25 performs convolution on the feature map F1 to calculate a corrected feature F1′.

[0079] (Step S5) The GAP unit 26 performs overall average pooling processing on the corrected feature F1' to calculate attention a for each channel.

[0080] (Step S6) The multiplication unit 22 multiplies the attention a by the backbone feature F0 for each channel to calculate a weighted feature F2.

[0081] (Step S7) The first convolutional layer 23 and the Head 31 perform, for example, convolution and upsampling on the weighted feature F2 to calculate and output the output Y. The Head 31 uses the output Y to calculate the loss L main Calculate.

[0082] (Step S8) The fourth convolutional layer 27 and the auxiliary head 32 perform, for example, convolution and upsampling on the corrected feature F' to calculate and output Y'. Next, the auxiliary head 32 uses the output Y' to calculate the loss L aux Calculate.

[0083] (Step S9) The image recognition device 1 calculates the loss L main and loss L aux From,the overall loss function is calculated and attention is learned using,the calculated loss function L.

[0084] The above-mentioned processing procedure is an example, and is not limited to this. For example, some processes may be performed in a balanced manner, or the order of the processes may be reversed. In addition, if learning has been completed, the calculation of the loss and the loss function, and the learning process may not be performed.

[0085] As described above, in this embodiment, we consider the idea of ​​feature attention for semantic segmentation and provide a CFA (Context-aware Feature Attention) module that adjusts the importance of corresponding feature maps based on global context. Note that we have constructed CFANet by combining FCN and CFA modules. As a result, according to this embodiment, the use of the CFA module makes it possible to improve the discrimination between feature maps.

[0086] Moreover, according to this embodiment, it is possible to improve the accuracy of semantic segmentation, and it has been experimentally demonstrated that the estimated weights indicate the importance of the feature maps. Furthermore, according to this embodiment, it is possible to improve the discrimination between feature maps, and when comparing the similarity of the feature amounts between pixels, it is possible to distinguish between regions more clearly than in the past.

[0087] In addition, according to this embodiment, the discrimination between pixels is improved, making visual judgment easier. According to this embodiment, it can be applied to understanding the basis for judgment, analyzing the cause of false detection, etc. According to this embodiment, since the importance of the feature map can be known, calculation time can be shortened by extracting only maps with high importance. According to this embodiment, it can be applied to channel pruning by making the importance sparse. According to this embodiment, the distribution of importance tends to become sharper as the amount of learning data increases (the more data, the higher the certainty of importance). According to this embodiment, by utilizing this tendency, new data can be input and the distribution can be confirmed, and it can be determined whether the data should be taught as training data. As a result, according to this embodiment, it is possible to reduce the cost of teaching.

[0088] The network structure for calculating the importance of each feature map (channel) is not limited to the above-mentioned configuration, and may be other configurations. In the above-mentioned embodiment, the auxiliary classification network is trained on the exact same task as the main task, but an appropriate importance can be calculated even when it is combined with other tasks (scene classification, edge detection, caption generation, etc.). In addition, the position at which the CFA module and each functional unit of the CFA module are inserted is not limited to the above-mentioned positions, and may be other positions. In the above-mentioned example, the importance is limited to positive values ​​to improve interpretability. However, it may have negative values.

[0089] In addition, a program for realizing all or part of the functions of the image identification device 1 of the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the image identification device 1. In addition, the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also refers to a storage device that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0090] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0091] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0092] Reference Signs List 1...image recognition device, 11...image acquisition unit, 12...feature extraction unit, 10...semantic segmentation device, 30...visualization unit, 21...feature acquisition unit, 22...multiplication unit, 23...first convolution layer, 24...second convolution layer, 25...third convolution layer, 26...GAP unit, 27...fourth convolution layer, 31...head, 32...auxiliary head, 33...teacher label providing unit, 34...similarity map creation unit

Claims

1. an image acquisition unit that acquires an image; a feature extraction unit that extracts a plurality of feature amounts of the acquired image; a creation unit that creates a feature map by performing a convolution process on each of the plurality of feature amounts; a weighted feature generation unit that calculates modified features by convolution processing on the feature maps, performs overall average pooling processing on the calculated modified features to aggregate contexts, generates attention as a weighting coefficient for each channel, and multiplies the generated attention by the feature maps to generate weighted features by applying enhancement and attenuation weighting to the multiple feature maps; a first loss calculation unit that calculates an output by performing convolution and upsampling processing on the weighted feature amount, and compares the calculated output with teacher data to calculate a first loss; a second loss calculation unit that calculates an output by performing convolution and upsampling processing on the feature map, and calculates a second loss by comparing the calculated output with teacher data; Equipped with Calculating an overall loss function from the first loss and the second loss, and learning the weighting coefficients using the calculated loss function. Image recognition device.

2. An image acquisition unit acquires an image, A feature extraction unit extracts a plurality of feature amounts of the acquired image, a creation unit that creates a feature map by performing a convolution process on each of the plurality of feature amounts; a weighted feature generation unit calculates modified features by performing a convolution process on the feature maps, performs a global average pooling process on the calculated modified features to aggregate contexts, generates attention, which is a weighting coefficient for each channel, and multiplies the generated attention by the feature maps to generate weighted features by performing enhancement and attenuation weighting on the multiple feature maps; a first loss calculation unit performs convolution and upsampling processing on the weighted features to calculate an output, and compares the calculated output with teacher data to calculate a first loss; a second loss calculation unit performs convolution and upsampling processing on the feature map to calculate an output, and compares the calculated output with teacher data to calculate a second loss; Calculating an overall loss function from the first loss and the second loss, and learning the weighting coefficients using the calculated loss function. Image identification methods.

3. On the computer, Acquire an image, Extracting a plurality of feature amounts of the acquired image; creating a feature map by convolution processing for each of the plurality of feature amounts; A convolution process is performed on the feature map to calculate a corrected feature, a global average pooling process is performed on the calculated corrected feature to aggregate contexts, attention is generated as a weighting coefficient for each channel, and the generated attention is multiplied by the feature map to perform weighting of enhancement and attenuation on the multiple feature maps to generate weighted features; performing convolution and upsampling processing on the weighted features to calculate an output, and comparing the calculated output with training data to calculate a first loss; performing convolution and upsampling processing on the feature map to calculate an output, and comparing the calculated output with training data to calculate a second loss; An overall loss function is calculated from the first loss and the second loss, and the weighting coefficient is learned using the calculated loss function. program.

Images