Image processing device and image processing method

Abstract

To provide a technique for performing recognition processing at high speed robustly on, for example, a size and a posture, without performing processing involving a large operation cost as done in the related art.SOLUTION: Outputs of a plurality of layers of a hierarchical neural network obtained by processing an input image using the hierarchical neural network are connected to generate connected hierarchical features. An attribute score map representing an attribute of an area of the input image using the connected hierarchical features is generated for each attribute. The generated attribute score maps for the respective attributes are integrated to generate and output a recognition result for a recognition target.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to recognition technology. [Background technology]

[0002] There are known technologies that extract features from video data and use a classifier to identify the subject in the video data. One such technology is the Convolutional Neural Network (CNN), a type of neural network. CNNs have the property of sequentially performing local convolutional calculations in multiple stages.

[0003] The technology described in Non-Patent Document 1 applies CNN to image data to detect objects. Specifically, (1) the image is processed using CNN. Furthermore, (2) the features of the final layer of CNN are aggregated for each region of interest (ROI) to determine whether or not it is an object. This process is performed for all ROIs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent application 2015-115171 [Non-patent literature]

[0005] [Non-Patent Document 1] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015 [Non-patent document 2] MD Zeiler, R. Fergus, Visualizing and Understanding Convolutional Networks, ECCV 2014 [Non-patent document 3] Hyeonwoo Noh, Seunghoon Hong, Bohyung Han, Learning Deconvolution Network for Semantic Segmentation, ICCV 2015 [Non-patent document 4] A. Krizhevsky et al., ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012 [Non-patent document 5] Emmanouil Z. Psarakis and Georgios D. Evangelidis, An Enhanced Correlation-Based Method for Stereo Correspondence with Sub-Pixel Accuracy, ICCV 2005 [Non-patent document 6] Wei Luo, XiaogangWang, Xiaoou Tang, Content-Based Photo Quality Assessment, ICCV 2011 [Non-Patent Document 7] Lazebnik, Schmid, Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, CVPR 2006 Summary of the Invention [Problem to be solved by the invention]

[0006] Conventional methods such as those in Non-Patent Document 1 aggregate features for each region of interest, which poses a problem of high computational costs when there are many candidate object regions or when multiple recognition tasks are performed simultaneously.

[0007] The present invention has been made in view of such problems, and provides a technique for performing recognition processing at high speed and robustly with respect to, for example, size and orientation, without performing processing that requires high computational costs as in the past. [Means for solving the problem]

[0008] One aspect of the present invention is to provide an input unit for inputting an input image to a hierarchical neural network having a plurality of layers; a map generating means for generating, based on a concatenated feature map in which feature maps output from the plurality of layers are concatenated, a likelihood map indicating the likelihood that a predetermined position in an object region when an object is detected exists at coordinates indicating a position in the concatenated feature map; an output means for outputting a result of detecting the object in the input image based on the likelihood map; With death, The map generating means generates, for each of the plurality of attributes, a likelihood map indicating the likelihood that a predetermined position of an object region when an object of the attribute is detected exists at coordinates indicating a position in the linked feature map, based on input of the linked feature map to a plurality of classifiers corresponding to the plurality of attributes, and the output means outputs a detection result of the object in the input image based on an integration result obtained by integrating for each coordinate of the likelihood map having a common coordinate axis, which corresponds to the plurality of attributes generated by the map generating means. It is characterized by: [Effects of the Invention]

[0009] According to the configuration of the present invention, it is possible to perform recognition processing robustly against size, orientation, and the like at high speed without performing processing that requires high computational costs as in the past. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 3] 10 is a flowchart of a recognition process performed by the image processing device. [Figure 4] 10 is a flowchart of the processing of steps S1 to S4. [Figure 5] FIG. [Figure 6] FIG. [Figure 7] A diagram showing an example of training data [Figure 8]10 is a flowchart of a procedure for integrating attributes related to region determination. [Figure 9] 1 is a flowchart of the attribute integration procedure for object detection. [Figure 10] FIG. 10 is a diagram for explaining a modification of the first embodiment. [Figure 11] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 12] 1 is a flowchart of the integration procedure for person detection. [Figure 13] 10A and 10B are diagrams showing an example of attribute determination and an example of result output. [Figure 14] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 15] 10 is a flowchart of the operation of the image processing device. [Figure 16] 4A and 4B are diagrams showing horizon information values, focus information values, and photometry information values. [Figure 17] FIG. 1 is a diagram showing an example of the functional configuration of an image processing apparatus. [Figure 18] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 19] 10 is a flowchart of the operation of the image processing device. [Figure 20] FIG. 10 is a diagram for explaining a derivative form of the fourth embodiment. [Figure 21] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the embodiment described below shows an example of a specific implementation of the present invention, and is one of the specific examples of the configuration described in the claims.

[0012] [First embodiment] First, an example of the functional configuration of an image processing device that performs image recognition on an input image will be described with reference to the block diagram of FIG. 1. An image input unit 101 inputs an input image containing one or more objects as a recognition target (detection target). A neural network 102 is a hierarchical neural network having first to n-th layers (n is a natural number of 2 or more), and processes the input image input by the image input unit 101. A linked feature generation unit 104 performs feature generation to generate linked hierarchical features 103 by linking outputs (feature maps) of specified layers in the neural network 102. The attribute determination unit 105 has first attribute determination units 105a to n-th attribute determination units 105c, and each of the first attribute determination units 105a to n-th attribute determination units 105c uses the linked hierarchical features 103 to generate and output a likelihood map (attribute score map 108) of the corresponding attribute. The attribute integration unit 106 integrates the attribute score map 108 to generate a predetermined recognition result such as object detection or scene recognition for the input image. The result output unit 107 outputs the recognition result by the attribute integration unit 106.

[0013] A specific example of image recognition processing using such an image processing device will be described below. In the following, seven types of attributes are used to calculate the attribute score map 108: pedestrian, small face, large face, sky, ground, building, and main subject. In this case, as shown in Fig. 2, the attribute determination unit 105 in Fig. 1 is configured as an attribute determination unit 205 that generates the attribute score map 108 for each of these seven types of attributes. In other words, in the following, an image processing device having the functional configuration example shown in Fig. 2 is used.

[0014] 2, n=7 in FIG. 1, and the first attribute determination unit 105a to the seventh attribute determination unit 105c are respectively referred to as likelihood determination units 205a to 205g. Likelihood determination unit 205a generates an attribute score map 208 for the attribute "pedestrian". Likelihood determination unit 205b generates an attribute score map 208 for the attribute "small face". Likelihood determination unit 205c generates an attribute score map 208 for the attribute "large face". Likelihood determination unit 205d generates an attribute score map 208 for the attribute "sky". Likelihood determination unit 205e generates an attribute score map 208 for the attribute "ground". Likelihood determination unit 205f generates an attribute score map 208 for the attribute "building". Likelihood determination unit 205g generates an attribute score map 208 for the attribute "main subject".

[0015] Next, the recognition process performed by the image processing device of FIG. 2 on one input image will be described with reference to the flowchart of FIG.

[0016] In step S1, image input unit 101 inputs an input image. In step S2, neural network 102 processes the input image input by image input unit 101 in step S1, and linked feature generation unit 104 extracts outputs (feature maps) of specified layers in neural network 102. In step S3, linked feature generation unit 104 generates linked hierarchical features 103 by linking the outputs (feature maps) of the specified layers extracted in step S3. In step S4, attribute determination units 205a to 205g in attribute determination unit 205 each generate attribute score maps 208 for pedestrians, small faces, large faces, sky, ground, buildings, and main subjects using linked hierarchical features 103.

[0017] In step S5, the attribute integration unit 106 integrates the attribute score maps 208 generated by the attribute determination units 205a to 205g, respectively, to determine the areas of sky, ground, and buildings in the input image. In step S6, the attribute integration unit 106 integrates the attribute score maps 208 generated by the attribute determination units 205a to 205g, respectively, to determine the area of the main subject in the input image. In step S7, the attribute integration unit 106 integrates the attribute score maps 208 generated by the attribute determination units 205a to 205g, respectively, to detect pedestrians and faces in the input image. In step S8, the result output unit 107 outputs the determination results and detection results obtained by the attribute integration unit 106 in steps S5 to S7. The output destination of the result output unit 107 is not limited to a specific output destination.

[0018] Next, the details of the processing in the above steps S1 to S3, that is, the processing from inputting the input image to generating the linked hierarchical feature 103, will be described with reference to the flowchart in FIG. 4(A).

[0019] In step S101, the image input unit 101 inputs an input image. In step S102, the connected feature generation unit 104 initializes an array (array of connected hierarchical features) for registering the output (feature map) of a specified layer in the neural network 102 (an empty array is prepared by setting a value such as NULL to each element of the array). Then, the processes of steps S103 to S108 are performed for each of the first to n-th layers of the neural network 102. In the following, a CNN is used as the neural network 102. The CNN here may not use a layer called a fully connected layer, but may use only a type of layer called a convolutional layer.

[0020] In step S104, the neural network 102 performs arithmetic processing on the layer of interest. In the first step S104, arithmetic processing is performed on the first layer, in the second step S104, arithmetic processing is performed on the second layer, and in the nth step S104, arithmetic processing is performed on the nth layer. By performing the arithmetic processing in step S104 for each layer, the neural network 102 realizes the arithmetic processing for each layer as shown in FIG.

[0021] The neural network 102 according to this embodiment performs arithmetic processing for each layer as shown in Fig. 5. Although Fig. 5 illustrates an example of a three-layer neural network 102, the number of layers is not limited to three.

[0022] The neural network 102 performs convolutional filter processing 402a, 402b, and 402c at each layer to perform multiple convolutional filter processing on the input image 401. Furthermore, the neural network 102 performs activation function processing after the convolutional filter processing at each layer (omitted in Figure 5). The neural network 102 also performs pooling processing 404a and 404b after the activation function processing at each layer except the first layer. The convolutional filter processing and activation function processing at the Lth layer (L=1, 2, 3 in Figure 5) are performed according to the following equations:

[0023]

number

[0024] where f L (x, y, z) is the feature map output by the Lth layer, and consists of z maps. The numbers in the unit ch in Figure 5 indicate the number of feature maps. For example, in Figure 5, feature map 403a consists of 64 maps, feature map 403b consists of 128 maps, and feature map 403c consists of 256 maps.

[0025] θ(·) is the activation function consisting of half-wave rectification. L(Δx,Δy,CHIN,CHOUT) (where Δx,Δy∈{-K,...,0,...,K}) are the convolution weight parameters of the Lth layer, B L CHOUT is the bias term for the Lth layer. CHIN represents the number of the feature map output by the (L-1)th layer, and CHOUT represents the number of the feature map output by the Lth layer. In the above equation, the input image I(x,y) consisting of RGB3 channels is 0 It shall be treated as (x,y,z).

[0026] Note that before the convolution process, the feature map f L-1 The pixels around the pixel are filled with 0 values before the convolution (this is called padding).

[0027] Pooling processes 404a and 404b are processes that reduce the size of the feature map by representing it with a representative value for each specified local region. Pooling has the effect of making the recognition performance of the CNN more robust, but it also has the disadvantage of reducing the resolution of the results. In the example shown in Figure 5, pooling processes 404a and 404b are processes that integrate feature maps in 2x2 pixel units to reduce them to feature maps with half the resolution.

[0028] Through the computational processing up to this point, feature maps 403a, 403b, and 403c are generated at each layer of the CNN. All of the above is common technology related to CNN and is widely known in Non-Patent Documents 1 to 3, etc., so further detailed explanation will be omitted.

[0029] It is assumed that the CNN learns weight parameters in advance through a classification task using a large-scale database, as in Non-Patent Document 4. As a result, a map that responds well to simple patterns such as the inclination of line segments in an image is generated as a low-level feature map 403a that is close to the CNN's input image. Then, a map that aggregates a wider range of image patterns and responds to complex pattern shapes is generated as a later-stage high-level feature map 403c. The above-mentioned properties of CNN are widely known, as in Non-Patent Document 2, etc.

[0030] Returning to FIG. 4(A), next, in step S105, the connection feature generation unit 104 determines whether the layer of interest is a predetermined specified layer (a layer that outputs a feature map) among the layers in the neural network 102. In other words, the connection feature generation unit 104 determines whether the feature map resulting from the arithmetic processing by the neural network 102 is the result of the specified layer. If the result of this determination is that the layer of interest is the specified layer, the process proceeds to step S106; if the layer of interest is not the specified layer, the process returns to step S103 with the next layer as the layer of interest.

[0031] In step S107, the linked feature generation unit 104 adds and registers the feature map, which is the output of the layer of interest, to the above array. By repeating this process, a high-dimensional linked hierarchical feature (array) in which multiple feature maps are linked together is finally obtained. In the example of FIG. 5, the feature map f 1 (x,y),f 2 (x,y),f 3 The (x, y) are connected to obtain a 448-channel connected hierarchical feature 406 (F(x, y)). The connected hierarchical feature is generally expressed by the following formula:

[0032]

number

[0033] where F is the connected hierarchical feature. 1 ,f 2 ,···,f nis a feature map extracted from the neural network 102, and the linked hierarchical feature F is obtained by linking the above feature maps in the Z-dimensional direction. x, y are the coordinates of a pixel in the feature map.

[0034] Generally, since CNNs perform pooling processing, the resolution of the feature maps differs depending on the layer. Therefore, in step S106 before step S107, the linked feature generation unit 104 adjusts the resolution of the feature maps to be registered in the above array to a specified resolution. In the example of FIG. 5, the vertical and horizontal resolution of feature map 403b is half that of feature map 403a (input image 401), and the vertical and horizontal resolution of feature map 403c is half that of feature map 403b. In this case, the linked feature generation unit 104 does not perform resolution conversion on feature map 403a, but performs resolution conversion on feature map f 1 (x, y). The linked feature generation unit 104 also performs upsampling 405a on the feature map 403b, doubling the vertical and horizontal size (resolution) thereof, thereby aligning the vertical and horizontal size (resolution) of the feature map 403b with that of the feature map 403a (input image 401). This "vertical and horizontal size (resolution) of the feature map 403b aligned with that of the feature map 403a (input image 401)" is the feature map f 2 (x, y). Size conversion (resolution conversion) by upsampling can be achieved by adopting a common method such as copying pixel values or linear interpolation. Furthermore, the linked feature generation unit 104 performs upsampling 405b on the feature map 403c, which increases the vertical and horizontal size (resolution) by four times, thereby aligning the vertical and horizontal size (resolution) of the feature map 403c to that of the feature map 403a (input image 401). This "vertical and horizontal size (resolution) of the feature map 403c aligned to that of the feature map 403a (input image 401)" is the feature map f 3 (x, y).

[0035] Depending on the size to be aligned, downsampling may be performed instead of upsampling. 1(x,y),f 2 (x,y),f 3 (x,y) has the same resolution as the input image I(x,y). And such a feature map f 1 (x,y),f 2 (x,y),f 3 The concatenation of (x, y) is the concatenated hierarchical feature 406 (F(x, y)).

[0036] In this embodiment, the input image I(x,y) and the linked hierarchical feature F(x,y) have the same resolution. However, if the magnification of the upsampling (downsampling) process is changed, it is possible to generate a linked hierarchical feature F'(x',y') with a coarser or finer resolution than the input image I(x,y). Therefore, in the following description, the pixels of the linked hierarchical feature F(x,y) will be referred to as a "region block" as a generalized name.

[0037] The connected hierarchical features obtained in this way are features that integrate multiple feature maps from the lower to the higher layers of the neural network 102. Therefore, information on various scales and variations of various objects is included in the features. Furthermore, each region block (x, y) corresponds to a high-dimensional feature vector. Due to these two properties, the connected hierarchical features can be effectively used for various recognition tasks (several variations of the types of recognition tasks that are possible are shown in each embodiment).

[0038] Next, the details of the process in step S4 above will be explained with reference to the flowchart in Fig. 4(B). The processes in steps S109 to S113 are performed for each region block in the linked hierarchical feature. The processes in steps S110 to S112 are performed for each attribute (each of the seven types of attributes in this embodiment). The attributes determined in this embodiment are the presence or absence of three types of objects: pedestrian, small-sized face, and large-sized face, and the likelihood for four types of regions: sky region, ground region, building region, and main subject region (the method of learning the classifier will be described later).

[0039] The attribute determination unit 205 has likelihood determination units 205a to 205g as classifiers that have been trained in advance to determine the likelihood of each of the above attributes. A linear support vector machine (SVM) can be used as the classifier. In step S111, the attribute determination unit 205 generates an attribute score map 208 for the j (1≦j≦7) attribute for the i (1≦i≦I: I is the number of region blocks) region block in the linked hierarchical feature. For example, when j=1, the likelihood determination unit 205a generates the attribute score map 208 for the attribute "pedestrian" for the i region block. When j=2, the likelihood determination unit 205b generates the attribute score map 208 for the attribute "small face" for the i region block. When j=3, the likelihood determination unit 205c generates the attribute score map 208 for the attribute "large face" for the i region block. When j=4, likelihood determination unit 205d generates an attribute score map 208 for the attribute "sky" for the i-th region block. When j=5, likelihood determination unit 205e generates an attribute score map 208 for the attribute "ground" for the i-th region block. When j=6, likelihood determination unit 205f generates an attribute score map 208 for the attribute "building" for the i-th region block. When j=7, likelihood determination unit 205g generates an attribute score map 208 for the attribute "main subject" for the i-th region block. The attribute score map 208 is generated according to the following formula:

[0040]

number

[0041] However, L j (x,y) is the likelihood score of the jth attribute for the region block at coordinates (x,y), and v jk is the kth support vector of the jth attribute classifier, α jk is the weight coefficient of the support vector, b jare bias terms, which are acquired in advance through training. Furthermore, F(x,y) is the feature vector of the region block at coordinates (x,y), and R(·) is the normalization function used to convert the SVM output into likelihood. However, to reduce the processing load, the normalization function R(·) is approximated by the following function, which normalizes the SVM score to 0 to 1:

[0042]

number

[0043] Here, τ is an appropriate constant. Since the attribute determination unit 205 performs determination processing for all region blocks and all attributes, it is preferable that the attribute determination unit 205 be a classifier with light processing, such as the linear SVM in this embodiment. In other words, any classifier, such as a decision tree or multilayer perceptron, may be used as long as the processing is light.

[0044] In this embodiment, the attribute determination is performed using the feature F(x, y) of each region block (x, y) as the input feature of the attribute determination unit 205. As another example, all feature F(x+Δx, y+Δy) (where Δx, Δy∈{-w,...-1,0,1,...,w}) in the surrounding range w are concatenated and used as the feature for determination. As such, various types of discriminators for the attribute determination unit 205 and features used for determination are conceivable, and the present invention is not limited to any one of them.

[0045] The result of the above processing is shown in Fig. 6. As shown in Fig. 6, attribute determination is performed by the likelihood determination units 205a to 205g of the attribute determination unit 205 using the linked hierarchical feature 602 of the input image 601, thereby generating an attribute score map 604 for each attribute. The attribute score map 604 includes an attribute score map in which the likelihood scores (pedestrian likelihood scores) of the attribute "pedestrian" determined for each region block by the likelihood determination unit 205a are mapped. The attribute score map 604 also includes an attribute score map in which the likelihood scores (small face likelihood scores) of the attribute "small face" determined for each region block by the likelihood determination unit 205b are mapped. The attribute score map 604 also includes an attribute score map in which the likelihood scores (large face likelihood scores) of the attribute "large face" determined for each region block by the likelihood determination unit 205c are mapped. The attribute score map 604 also includes an attribute score map in which the likelihood score of the attribute "sky" (sky likelihood score) determined by the likelihood determination unit 205d for each region block is mapped. The attribute score map 604 also includes an attribute score map in which the likelihood score of the attribute "ground" (ground likelihood score) determined by the likelihood determination unit 205e for each region block is mapped. The attribute score map 604 also includes an attribute score map in which the likelihood score of the attribute "building" (building likelihood score) determined by the likelihood determination unit 205f for each region block is mapped. The attribute score map 604 also includes an attribute score map in which the likelihood score of the attribute "main subject" (main subject likelihood score) determined by the likelihood determination unit 205g for each region block is mapped. In FIG. 6, region blocks with high likelihood scores for each attribute are represented in gray shades, with black representing region blocks with low likelihood scores and white representing region blocks with high likelihood scores.

[0046] Here, in order to obtain a determination result such as the attribute score map 604 shown in Fig. 6, the attribute determination unit 205 is trained in the following manner. An example of training data used for training the attribute determination unit 205 is shown in Fig. 7.

[0047] Each training data set consists of an image as shown in Figure 7(A) and (B) and training values as shown in Figure 7(C) to (F). The training values consist of two types: training values for object attributes (Figure 7(C) and (D)), and training values for region attributes (Figure 7(E) and (F)).

[0048] The teacher value of the object attribute is a value indicating whether each region block contains one of the following objects: a pedestrian, a small-sized face, or a large-sized face. Specifically, a region block containing the reference point of the object is designated as a positive case (1), and other region blocks are designated as negative cases (0), and a value of 0 or 1 is assigned manually for each object category (in this embodiment, the center of gravity of the object region is designated as the reference point of the object).

[0049] The teacher values for region attributes are assigned values of 0 or 1 for each region category: sky, ground, building, and main subject. Region blocks whose area is equal to or greater than a predetermined value are considered positive examples (1), and other region blocks are considered negative examples (0). Specific examples are shown in Figures 7(E) and 7(F). Region blocks that are positive examples of sky, ground, building, and main subject regions are assigned symbols S, G, B, and M, respectively. For convenience, only one region category symbol is shown for each region block here, but multiple attributes may be associated with one region block, such as a building being the main subject.

[0050] The training values of each attribute given above (0, 1) are used as the target variables, and the linked hierarchical feature F(x, y) of the corresponding region block is used as the explanatory variable. The SVM is trained based on a large number of training examples so that it can distinguish between positive and negative examples. Training is performed for each attribute j, and the SVM parameters ([v jk ,α jk ,b j ]) is obtained. The above is the learning procedure of the attribute determination unit 205.

[0051] Next, we will explain the detailed procedure for integrating attribute score maps and outputting the results of the target recognition task (details of the processing in steps S5 to S8 above). Here, attribute score map integration refers to the process of using multiple attributes as direct and indirect clues to determine the target recognition task. Integration can be done rule-based, by machine learning using a classifier or regression, or by combining these methods. Here, we will explain the rule-based method.

[0052] In the case of a human body detection task, attributes of the human body, facial attributes, and building attributes (which are easily mistakenly detected as human bodies) are attributes that provide clues for determining whether a human body is present. Furthermore, since there is a co-occurrence relationship between faces and human bodies, and an exclusive relationship between human bodies and buildings in that they do not exist in the same location, it is possible to perform integration that takes into account the relationship between these attributes. Specifically, this is done using the procedure described below.

[0053] The detailed procedure for integrating attribute score maps and outputting the results of the target recognition task will be explained using Figures 8 and 9. Figure 8 is a flowchart of the attribute integration procedure for region determination, and Figure 9 is a flowchart of the attribute integration procedure for object detection.

[0054] 8A is a flowchart of the integration procedure for sky, ground, and building regions. The processes of steps S201 to S207 are performed for each region block in the linked hierarchical feature. The processes of steps S202 to S206 are performed for each attribute (in this embodiment, "sky," "ground," and "building").

[0055] In step S203, the attribute integration unit 106 determines whether the likelihood score of the j-th attribute of the i-th (1≦i≦I: I is the number of region blocks) region block is equal to or greater than a predetermined value. Here, the first attribute (j=1) is assumed to be "sky," the second attribute (j=2) is assumed to be "ground," and the third attribute (j=3) is assumed to be "building." If the result of this determination is that the likelihood score of the j-th attribute of the i-th region block is equal to or greater than the predetermined value, the process proceeds to step S204. On the other hand, if the likelihood score of the j-th attribute of the i-th region block is less than the predetermined value, the process proceeds to step S206. If the processes of steps S202 to S206 have been performed for j=1 to 3, steps S203 and subsequent steps are performed for the (i+1)-th region block. If the processes of steps S201 to S207 have been performed for i=1 to I, the process of FIG. 8A is completed.

[0056] In step S204, the attribute integration unit 106 determines whether the likelihood score of the other attributes (excluding the main subject) of the i-th region block is less than a predetermined value. If the result of this determination is that the likelihood score of the other attributes (excluding the main subject) of the i-th region block is less than the predetermined value, the process proceeds to step S205. On the other hand, if the likelihood score of the other attributes (excluding the main subject) of the i-th region block is equal to or greater than the predetermined value, the process proceeds to step S206.

[0057] In step S205, the attribute integration unit 106 determines that the attribute of the ith region block is the jth attribute.

[0058] Figure 8(B) is a flowchart of the integration procedure for the main subject region. The processing content is almost identical to the processing content according to the flowchart in Figure 8(A). However, unlike the procedure in Figure 8(A), since faces, pedestrians, sky, ground, and buildings can all be the main subject, the likelihood scores of other attributes are not taken into consideration. The processing of steps S210 to S213 is performed for each region block in the connected hierarchical feature.

[0059] In step S211, the attribute integration unit 106 determines whether the likelihood score of the attribute "main subject" of the i-th region block (1≦i≦I: I is the number of region blocks) is equal to or greater than a predetermined value. If the result of this determination is that the likelihood score of the attribute "main subject" of the i-th region block is equal to or greater than the predetermined value, the process proceeds to step S212. On the other hand, if the likelihood score of the attribute "main subject" of the i-th region block is less than the predetermined value, the process proceeds to step S213, where the subsequent processes are performed on the next region block. In step S212, the attribute integration unit 106 determines that the attribute of the i-th region block is "main subject."

[0060] 9 is a flowchart of the integration procedure for detecting pedestrians and faces. In step S301, the attribute integration unit 106 initializes the value of a variable n, which indicates the number of detected objects, to 0. The processes of steps S302 to S310 are performed for each region block in the linked hierarchical feature. The processes of steps S303 to S309 are performed for each attribute (in this embodiment, "pedestrian," "small face," and "large face").

[0061] In step S304, the attribute integration unit 106 determines whether the likelihood score of the jth attribute of the i-th region block (1≦i≦I: I is the number of region blocks) is equal to or greater than a predetermined value. Here, the first attribute (j=1) is assumed to be "pedestrian," the second attribute (j=2) is assumed to be "small face," and the third attribute (j=3) is assumed to be "large face." If the result of this determination is that the likelihood score of the jth attribute of the i-th region block is equal to or greater than the predetermined value, the process proceeds to step S305. On the other hand, if the likelihood score of the jth attribute of the i-th region block is less than the predetermined value, the process proceeds to step S309. If the processes of steps S303 to S309 have been performed for j=1 to 3, steps S304 and subsequent steps are performed for the (i+1)-th region block. If the processes of steps S302 to S310 have been performed for i=1 to I, the process of FIG. 9A is completed.

[0062] In step S305, the attribute integration unit 106 determines whether the likelihood score of the jth attribute of the i-th region block is the largest among the likelihood scores of all attributes (excluding the main subject) of the nine region blocks, including the region block itself and its eight neighboring region blocks. If the result of this determination is that the likelihood score of the jth attribute of the i-th region block is the largest among the likelihood scores of all attributes (excluding the main subject) of these nine region blocks, the process proceeds to step S306. On the other hand, if the likelihood score of the jth attribute of the i-th region block is not the largest among the likelihood scores of all attributes (excluding the main subject) of these nine region blocks, the process proceeds to step S309.

[0063] In step S306, the attribute integration unit 106 determines that the reference point of the object of the j-th attribute (here, the center of gravity of the object) has been detected, and increments the value of the variable n by 1. In step S307, the attribute integration unit 106 associates the position of the reference point of the object with the likelihood score of the j-th attribute of the i-th region block, and stores them in a memory (not shown) in the image processing device.

[0064] In step S308, the attribute integration unit 106 performs position correction using sub-pixel estimation to estimate the object position more precisely. Here, the sub-pixel estimation uses a method such as that described in Non-Patent Document 5. In Non-Patent Document 5, position correction is performed based on the score values around the object position.

[0065] The process then proceeds to step S311 in Fig. 9(B). The processes in steps S311 to S315 are performed for each object detected by the process according to the flowchart in Fig. 9(A).

[0066] In step S312, the attribute integration unit 106 selects an unselected object (the kth object (k=1 to n)) from among the objects detected in the processing according to the flowchart in FIG. 9(A) as the selected object. The attribute integration unit 106 then determines whether the attribute of the selected object is "small face" or "large face." If the result of this determination is that the attribute of the selected object is "small face" or "large face," the processing proceeds to step S313; if the attribute of the selected object is neither "small face" nor "large face," the processing proceeds to step S314.

[0067] In step S313, the attribute integration unit 106 estimates the size of the face by a weighted sum as shown in the following formula using the likelihood scores of two types of faces with different sizes.

[0068]

number

[0069] However, L 小 , L 大 are the likelihood scores for small and large faces, respectively, and the reference size 小 , reference size 大 is a predetermined reference value for the face size.

[0070] By the above processing up to step S315, attribute information that existed separately in each region block, such as the likelihood of a face by size, the likelihood of a pedestrian, and the likelihood of each category in the background region, is organized and integrated into an object detection result with identified position and size. In this embodiment, pedestrians and faces are integrated independently. However, it is also possible to use a co-occurrence relationship, such as the high probability that a small face exists around a region block with a high likelihood of a pedestrian. As a derivative form of this, for example, the following method can be considered: (1) First, blur the pedestrian score map and add it to the small face score map. (2) The obtained map is used as a new small face score map. (3) The attribute integration process (steps S301 to S315) is performed using the same procedure as before.

[0071] In addition, the following method can be considered as a form of attribute integration that can prevent buildings from being mistakenly detected as pedestrians: (1) Subtract the score map of building areas from the score map of pedestrians. (2) Use the obtained map as a new score map of pedestrians. (3) Perform the attribute integration process (steps S301 to S315) using the same procedure as before.

[0072] An example of the results obtained by the above processing by the attribute integration unit 106 is shown in Fig. 6 as integration result 606. The detection results of pedestrians and faces are indicated by dashed frames, and the area blocks determined to be the sky area, ground area, building area, and main subject area are indicated by the symbols S, G, B, and M, respectively.

[0073] In step S314, the result output unit 107 outputs a "rectangular frame of an estimated size" for the position of the detected face. If a pedestrian is detected, a "rectangular frame of a predetermined size" for the position of the pedestrian is output. Similarly, in step S316, the result output unit 107 outputs labels for each region block of the sky region, ground region, and building region. In step S317, the result output unit 107 outputs a label for the region block of the main subject region. These are the final output results. An example of the final output result by the result output unit 107 is shown in FIG. 6 as final output result 607.

[0074] It should be noted that by using the attribute information integration method shown in this embodiment, such as correcting the object position and estimating the face size, the final output result 607 is obtained with a higher resolution than the original attribute score map. Generating detailed connected hierarchical features or a detailed attribute score map requires a large computational cost. On the other hand, the attribute integration method shown in this embodiment makes it possible to obtain detailed results from a low-resolution attribute score map with a relatively low computational cost. This is one of the features of this embodiment, which determines multiple attributes and integrates them to recognize objects, and is therefore particularly emphasized here.

[0075] <Modification of the first embodiment> In the first embodiment, a CNN is used as the neural network, but the applicable neural network is not limited to a CNN. Any neural network or similar method consisting of multiple layers that generates intermediate results as a map can be applied. One such method is, for example, a non-CNN type neural network in which the convolution filters of each layer do not share weights.

[0076] As yet another example, it is possible to use a configuration that employs the same convolution processing as CNN, rather than SVM, as the attribute determination unit 205. In this derivative configuration, it is also possible to simultaneously learn the parameters of the convolution processing of the attribute determination unit 205 and the parameters of the preceding neural network 102 using the backpropagation method. In this case, it is expected that the accuracy of the recognition task will be improved compared to when each is trained independently (this is called end-to-end learning, and is widely known in Non-Patent Document 1, etc., so details will be omitted).

[0077] In the first embodiment, the feature map was upsampled when generating the linked hierarchical features. However, a method called deconvolution, as disclosed in Non-Patent Document 3, may be further considered as a derivative form. An example of this form is shown in FIG. 10. Through processing by the deconvolution filters 407a and 407b, feature maps with double and quadruple resolution are generated from the feature maps 403b and 403c, respectively. Furthermore, these deconvolution filters 407a and 407b can also be trained using backpropagation in conjunction with the attribute determination unit 205 and the neural network 102. This derivative form is expected to improve the accuracy of the recognition task compared to when simple upsampling is used. As yet another form, the attribute integration unit 106 may also be configured to use the same convolution processing as CNN, rather than being rule-based. As such, various derivative forms are possible for each module, and no specific form is required.

[0078] [Second embodiment] In the following embodiments and modifications, including this embodiment, differences from the first embodiment will be mainly described, and unless otherwise specified below, they will be considered to be the same as the first embodiment. In this embodiment, a form will be described in which recognition is performed that is robust against pose changes of an object such as a person.

[0079] In conventional methods such as those described in Non-Patent Document 1, object candidate regions (hereinafter referred to as regions of interest) are sometimes set to improve detection accuracy for objects of various sizes and orientations. However, conventional methods involve setting multiple regions of interest on an image and aggregating neural network features for each region of interest, which results in high computational costs. It is also common to set multiple regions of interest with various sizes and aspect ratios to accommodate variations in the orientation of the object. This embodiment uses a different method from the above and demonstrates one form of robust object detection that is resistant to variations in size and orientation without requiring computationally expensive processing.

[0080] An example of the functional configuration of an image processing device according to this embodiment will be described with reference to the block diagram of FIG. 11. In FIG. 11, the same functional units as those shown in FIG. 1 are assigned the same reference numerals, and descriptions of these functional units will be omitted. The image processing device of FIG. 11 is specialized for detecting people. As shown by the examples denoted by 908a to 908h in FIG. 11, people are classified into eight groups of appearance patterns based on the size, inclination, and aspect ratio of the frame of the person area. Specifically, in FIG. 11, the eight groups are defined by the following combination of parameters, for example:

[0081] Group 1: Small size, tilt 0°, aspect ratio 2 Group 2: Small size, tilt +45°, aspect ratio 2 Group 3: Small size, tilt 0°, aspect ratio 1 Group 4: Small size, tilt -45°, aspect ratio 2 Group 5: Large size, tilt 0°, aspect ratio 2 Group 6: Large size, tilt +45°, aspect ratio 2 Group 7: Large size, tilt 0°, aspect ratio 1 Group 8: Large size, tilt -45°, aspect ratio 2 In this embodiment, the above eight groups are each set to eight attributes of a person (person attribute). The person attribute determination unit 905 determines which of the eight types of person is present in each area block, or whether no person is present. Each of the person attribute determination units 905a to 905h is trained in advance so that when a person of the group it is responsible for is input, it can be determined as such. The learning procedure is as follows.

[0082] (1) Training images are prepared, and detection area frames are attached to people in the images in advance. The frames have three parameters: position, size, and tilt. (2) Person examples are divided into eight groups based on the size and tilt of the detection area frames (unsupervised clustering can be used, or the groups can be divided manually). (3) Eight attribute classifiers are given the person data of their respective groups as positive examples, and the data of areas other than people as negative examples, and the classifiers are trained using the linked hierarchical features as explanatory variables.

[0083] Fig. 12 is a flowchart of an integration procedure for detecting a person. The flowchart in Fig. 12 is similar to the flowchart in Fig. 9, except that the object to be detected is a person and the jth person attribute is used instead of the jth attribute. That is, the processes in steps S400 to S409 are similar to steps S301 to S310 in Fig. 9, except for the above points. Furthermore, the processes in steps S410 to S412 are similar to steps S311, S314, and S315 in Fig. 9. Therefore, a description of each step in the flowchart in Fig. 12 will be omitted.

[0084] An example of attribute determination and an example of the result output in this embodiment are shown in Fig. 13. Here, attribute determination results 1202 are obtained for an input image 1201 for detection area frames of a plurality of sizes, tilts, and aspect ratios (the magnitude of the likelihood score is indicated by the color intensity of the frame). These results are integrated by the attribute integration unit 906, and the maximum likelihood candidate frame (frames 1203a, 1203b) for each area block is superimposed on the input image 1201, and output as output result 1203. As a result, a detection area frame with a tilt and size that matches the posture of the person is output, as shown in Fig. 12.

[0085] Please note that unlike Non-Patent Document 1 and others, features are not aggregated (each time) for each object detection area frame. Also, the examples are divided into eight groups based on the appearance of the pattern. Please note that this simplifies the attribute determination problem, making it possible to determine patterns even with a lightweight linear classifier.

[0086] In this embodiment, only the maximum likelihood frame is left when integrating detection frames, but a weighted sum may be used as a derivative to estimate the detailed size of a person, as was done in the first embodiment. Furthermore, detailed values of the aspect ratio and tilt may also be estimated. This can be expressed as the following formula:

[0087]

number

[0088] Here, Li is the likelihood score of the i-th attribute, and the aspect ratio is the vertical size of the frame divided by the horizontal size.As a further derivative, instead of eight attributes, the person's attributes can be grouped into three types each: size = {large, medium, small}, tilt = {right, medium, left}, and aspect ratio = {vertical, square, horizontal}, and a nine-type attribute judgment can be performed.

[0089] As yet another derivative form, a two-stage attribute determination may be performed to reduce the amount of calculation. For example, the following example may be considered as a procedure for recognition. (1) As the first attribute determination, three types of human sizes are determined: large, medium, and small. (2) For only those area blocks with a high likelihood score in any of the three types in (1), a second attribute judgment is performed, classifying them into eight types based on size, tilt, and aspect ratio.

[0090] As a further derivative, there is also a possibility of not dividing the images into groups, but instead estimating the size, tilt, and aspect ratio values individually using a regression calculator. In addition to the forms listed here, there are various possible attributes to be judged, the order of attribute judgment, and the order of integration, and there are no particular limitations.

[0091] [Third embodiment] This embodiment is centered on the use of camera information, and the recognition task of this embodiment aims to determine the sky area. An example of the functional configuration of the image processing device according to this embodiment will be described with reference to the block diagram in FIG.

[0092] One feature of the image processing device according to this embodiment is that it has a camera information input unit 1308. Another feature is that the attribute determination unit 1305 includes likelihood determination units 1305a and 1305b for regions of detailed subcategories related to the sky, such as blue sky and cloud regions. Yet another feature is that the attribute determination unit 1305 includes likelihood determination units 1305c and 1305d for categories that are easily mistaken for the sky, such as a water surface and a white wall.

[0093] In this way, not only the attributes of the direct recognition target (blue sky, clouds), but also the attributes of indirect targets (water surface, white wall) that serve as clues for the recognition task can be included in the attribute determination unit 1305. This allows for comprehensive pattern discrimination, which is expected to improve the accuracy of the recognition task.

[0094] The operation of the image processing device according to this embodiment will be described with reference to the flowchart in Fig. 15. The processing in each of steps S501 to S504 is the same as steps S1 to S4 in Fig. 3, respectively, and therefore a description of these steps will be omitted. Furthermore, the processing in each of steps S505 to S508 is processing for determining the likelihood of each attribute for each region block, and is the same as that described in the first embodiment, and therefore a description of these steps will be omitted.

[0095] In step S509, the camera information input unit 1308 inputs horizon information values at the time of image capture as camera information. Specifically, the horizon information values here are as shown in FIG. 16(A). The camera information input unit 1308 is equipped with a gravity sensor, and in step S509a, estimates the position 1501 of the horizon on the image. Furthermore, in step S509b, the camera information input unit 1308 generates horizon information values 1502 that indicate how many blocks above or below the horizon each region block is located from the estimated horizon position 1501.

[0096] In step S510, the SVM 1306 makes a final determination of whether each region block is a sky or not based on the four attribute scores of blue sky, clouds, water surface, and white wall, as well as the horizon information value 1502. Here, the SVM 1306 calculates the sky likelihood for each region block using the above five types of maps (attribute score maps 1318 of the attributes from likelihood determination units 1305a to 1305d, and the map of horizon information value 1502) as five-dimensional input features. In this way, by also using camera information related to the horizon, it is possible to prevent, for example, a water surface reflecting the sky at the bottom of the image from being mistakenly identified as the sky.

[0097] In order to be able to determine the likelihood of sky, the SVM parameters are learned in advance by providing multiple training image data 1309, camera information 1310 corresponding to the training image data, and training values 1311 for each area block category, which are training values for whether or not the image is sky.

[0098] Another possible derivative form using camera information is the use of camera focus information values 1504, as shown in FIG. 16(B). This is a map that quantifies how far behind or in front of the subject in each area block is the focal plane (indicated by the symbol x in FIG. 16(B)) in the image, with the focal plane in focus set to 0. The camera focus information is generated by an optical mechanism such as image plane phase difference AF, which is separately provided in the camera information input unit 1308. By using the focus information values 1504 in combination, it is possible to prevent, for example, the erroneous determination of a blurred white artificial object that is far behind the focal plane as the sky.

[0099] Another possible derivative form of using camera information is a method using a camera photometric information value 1505, as shown in Fig. 16(C). This is a numerical value that represents the physical amount of light in an image, and can be obtained from the results of a camera photometric sensor that is separately provided in the camera information input unit 1308. One form of this photometric information value is called a Bv value, and since it is publicly known from Patent Document 1 and the like, details will be omitted. By also using the photometric information value, it is possible to reduce misidentification of a white wall and a cloudy sky, etc., compared to using only an RGB image.

[0100] As another derivation, it is also possible to concatenate and use camera information maps when generating the linked hierarchical feature 103. An example of the functional configuration of an image processing device in this case is shown in FIG. 17. As shown in FIG. 17, three types of camera information (three-dimensional), namely, a "horizon information value map," a "focus information value map," and a "photometric information value map," are concatenated when generating the linked hierarchical feature and sent to likelihood determination units 1305a to 1305d. Each classifier in the likelihood determination units 1305a to 1305d uses the feature quantities of the linked hierarchical feature and the three-dimensional camera information as feature quantities to determine the respective attributes. It is assumed that the likelihood determination units 1305a to 1305d have trained the classifier parameters in advance using multiple training image data 1309, camera information 1310 corresponding to the training image data, and teacher values 1311 for the categories of each region block so that the likelihood of each attribute can be determined. Whether the form in which camera information is used for attribute integration as shown in FIG. 14 or the form in which it is used for attribute determination as shown in FIG. 17 is more suitable depends on the quantity and quality of the learning data, etc.

[0101] In the above, in this embodiment, a form of additional use of camera information has been described. In addition, in this embodiment, the effect of using detailed subcategories and indirect attributes as targets for attribute determination has also been described. The use of subcategories and indirect attributes can also be applied to other recognition tasks. For example, when detecting people, it is possible to classify targets into subcategories based on gender and determine their attributes, or to classify targets into subcategories based on differences in appearance (such as the presence or absence of hats or glasses). Furthermore, in a person detection task, it is also possible to provide attribute determination for dogs to prevent misidentification.

[0102] The subcategories are distinguished based on at least one of the depth rotation of the object, the in-plane rotation of the object, the posture of the object, the shape of the object, the material of the object, the shape of the region of interest of the object, the size of the region of interest of the object, and the aspect ratio of the region of interest of the object.

[0103] [Fourth embodiment] In this embodiment, a form in which various types of recognition tasks are processed in an integrated manner will be described. The recognition tasks performed by the image processing device according to this embodiment are three types: region determination, image scene classification, and aesthetics determination.

[0104] FIG. 18 shows an example of the functional configuration of an image processing device according to this embodiment. The operation of the image processing device according to this embodiment will be described with reference to the flowchart in FIG. 19. Steps S601 to S603 are processes for generating linked hierarchical features by performing the same processes as in the previous embodiments. Step S604 is a process in which the attribute determination unit 1708 determines the scene attributes for each region block (x, y). Here, N scene categories, such as indoor scene, portrait scene, and sports scene, are assumed as scene attributes. The attribute determination unit 1708 uses the linked hierarchical features F(x, y) as input features and determines, for each region block (x, y), to which of the N scenes the image belongs. A general decision tree is used as a classifier for this determination. The hierarchical features F(x, y) are features for region block x and y, but also include higher-level features of a neural network. It should be noted that such image scene classification determination is also possible because the higher-level features include information that aggregates a wide range of image patterns.

[0105] Next, in step S605, the attribute integration unit 1709 integrates the scene attribute results for each region block to determine the scene of the entire image. Here, a linear SVM is used as the classifier. The likelihood scores of the scene attributes for each region block are linked to form features, and the linear SVM is used to learn and determine which scene the input image should be classified into.

[0106] Note that the scene category used in scene attribute determination and the category used in attribute integration do not need to be the same. For example, a more detailed classification such as "soccer," "baseball," "mountain," and "seaside" may be determined in scene attribute determination, and a higher-level classification such as "sports scene" or "natural landscape scene" may be determined in scene attribute integration. As with the previous embodiments, attribute categories can take various forms as long as they provide clues to the target recognition task.

[0107] Next, in step S606, the attribute determination unit 1705 obtains an attribute score map. This is the same method as that used for sky regions in the first embodiment, and so a description thereof will be omitted. Next, in step S607, the attribute integration unit 1706 determines the type of each region block, taking into account the likelihood of the image scene obtained in the previous stage. Specifically, the following equation is used here:

[0108]

number

[0109] Here, P(c|f) is the posterior probability that a region block has a region attribute c, f is the input feature, P(c|s) is the prior distribution that region attribute c appears in scene s, and P(s) is the probabilistic value of the scene likelihood calculated in step S605. This makes it possible to determine the type of region taking into account the prior distribution of cases, for example, that ocean regions rarely appear in indoor scenes.

[0110] Next, in step S608, the attribute determination unit 1710 determines the aesthetic attribute for each region block (x, y). As is known in Non-Patent Document 6 and elsewhere, this is done by preparing learning data in which images correspond to aesthetic evaluation values for the images in advance, using the linked hierarchical features F(x, y) as input features, and learning and estimating aesthetic score values using a regression unit such as logistic regression. At this time, the attribute determination unit 1710 may also receive the likelihood of the scene from the attribute integration unit 1709 and change the weight dictionary of the regression unit depending on the scene. This is a known method as described in Non-Patent Document 6.

[0111] Next, in step S609, the integration unit 1711 integrates the aesthetic attributes to obtain a scalar value representing the aesthetic judgment result. Here, a multi-layer CNN is used as the integration unit 1711. The CNN receives the aesthetic score map for each region block (x, y) as input and is a network that performs regression learning on weight parameters so that the output layer can correctly output the aesthetic evaluation value for the entire image. Since such a regression learning method using a CNN is well known in, for example, Non-Patent Document 1, details will not be provided here. Other possible implementations of the integration unit 1711 include a configuration that uses an ensemble average of each region block or a configuration that uses a linear discriminator. These can be selected based on the balance between computational complexity and accuracy. Finally, in step S610, the result output unit 1712 outputs the region judgment result, scene judgment result, and aesthetic judgment result, thereby completing the operation.

[0112] As described above, a method for executing multiple recognition tasks in an integrated manner has been described. The most computationally expensive process in this image processing device is the CNN calculation process. In this embodiment, this CNN calculation is performed only once. Instead, multiple attribute information is estimated using lightweight attribute determination processing in the subsequent stage, and this information is then interacted and integrated in the attribute integration unit, thereby obtaining highly accurate recognition results. This method is one of the features of the image processing device according to this embodiment.

[0113] As a derivative form, it is also possible to replace part or all of the attribute integration unit with a module called a fully connected layer of a neural network. In this case, as described in the first embodiment, the parameters of each module can be optimally trained in a unified manner using a method such as error backpropagation.

[0114] A further derived form as shown in FIG. 20 is also possible. This derived form further comprises a dictionary holding unit 1707 in addition to the configuration in FIG. 18. After determining the scene type, the attribute integration unit 1709 of this derived form sends the likelihood of the scene to the attribute determination unit 1705. The attribute determination unit 1705 receives scene likelihood information and performs attribute determination on only area categories that appear particularly frequently in scenes with high likelihood. The attribute determination unit 1705 reads a dictionary of attribute determination targets from the dictionary holding unit 1707 and performs attribute determination. By using the results of scene recognition in this way to perform attribute determination, it is expected that the amount of calculation will be reduced and robustness against noise will be improved.

[0115] As yet another derivative form, a method is conceivable in which scene classification is performed without performing attribute determination processing, but instead using a Bag of Words method (Non-Patent Document 7), which is a representative image scene classification method, based on linked hierarchical features.

[0116] In yet another variant, the attribute determination unit 1705 determines region attributes such as "person" or "grass" and generates a region attribute score map 1714. The attribute integration unit 1709 receives this and uses it to determine whether the scene is a "sports scene." In this way, the results of attribute determination can be shared across recognition tasks, potentially reducing the overall processing load of the image processing device. Yet another variant, also shown in FIG. 20, may further include a user instruction unit 1713. The user instruction unit 1713 accepts user instructions to select which of the results of multiple recognition tasks (region determination, scene recognition, and aesthetics evaluation) to output. When performing multitasking recognition, always displaying all results would make the results difficult to see. Therefore, this technique allows the output results to be changed according to the user's preferences and the shooting mode.

[0117] It should be noted that some or all of the above-described embodiments and modified examples may be used in appropriate combination, or some or all of the above-described embodiments and modified examples may be used selectively.

[0118] As described above, according to the above-described embodiments and modifications, multiple attributes are determined using hierarchical features of a neural network and a lightweight classifier, and then the determined attributes are integrated to perform recognition processing. This allows for fast, robust object detection that is independent of size and orientation, without the need for conventional high-cost processing. In yet another embodiment, object detection, region category determination, image scene category classification, and aesthetic evaluation of image information can be simultaneously performed. In yet another embodiment, objects with various different appearances, such as different sizes, aspect ratios, or tilts, can be rapidly detected.

[0119] [Fifth embodiment] The functional units shown in Figures 1, 2, 11, 14, 17, 18, and 20 may be implemented in hardware or software (computer programs). In the latter case, a computer device capable of executing the computer programs is applicable to the image processing devices described in the above embodiments and modifications. An example of the hardware configuration of a computer device applicable to the image processing devices described in the above embodiments and modifications will be described using the block diagram in Figure 21.

[0120] The CPU 2101 executes various processes using computer programs and data stored in the RAM 2102 and the ROM 2103. As a result, the CPU 2101 controls the operation of the entire computer device, and also executes or controls the processes described above as being performed by the image processing device.

[0121] The RAM 2102 has an area for storing computer programs and data loaded from the ROM 2103 or the external storage device 2106, and data received from the outside via the I / F 2107. The RAM 2102 also has a work area used by the CPU 2101 when executing various processes. In this way, the RAM 2102 can provide various areas as needed. The ROM 2103 stores computer programs and data that do not need to be rewritten.

[0122] The operation unit 2104 is configured with a user interface such as a keyboard and a mouse, and allows the user to input various instructions to the CPU 2101 by operating it.

[0123] The display unit 2105 is configured with a CRT or liquid crystal screen, and can display the processing results of the CPU 2101 as images, text, etc. The display unit 2105 may also be a projection device, such as a projector device, that projects images and text onto a projection surface.

[0124] The external storage device 2106 is a large-capacity information storage device, such as a hard disk drive. The external storage device 2106 stores an operating system (OS), computer programs, and data for causing the CPU 2101 to execute or control the various processes described above as being performed by the image processing device. The computer programs stored in the external storage device 2106 include computer programs for causing the CPU 2101 to implement the functions of the various functional units shown in FIGS. 1, 2, 11, 14, 17, 18, and 20. The data stored in the external storage device 2106 also includes the information described above as known information. The computer programs and data stored in the external storage device 2106 are loaded into the RAM 2102 as appropriate under the control of the CPU 2101, and are processed by the CPU 2101.

[0125] The I / F 2107 is for performing data communication with external devices. For example, an image capturing device may be connected to the I / F 2107, and a captured image to be recognized may be input from the image capturing device.

[0126] The CPU 2101 , RAM 2102 , ROM 2103 , operation unit 2104 , display unit 2105 , external storage device 2106 , and I / F 2107 are all connected to a bus 2108 .

[0127] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0128] 101: Image input unit 102: Neural network 104: Linked feature generation unit 205: Attribute determination unit 106: Attribute integration unit 107: Result output unit

Claims

1. an input means for inputting an input image to a hierarchical neural network having a plurality of layers; a map generating means for generating, based on a concatenated feature map in which feature maps output from the plurality of layers are concatenated, a likelihood map indicating the likelihood that a predetermined position in an object region when an object is detected exists at coordinates indicating a position in the concatenated feature map; an output means for outputting a result of detecting the object in the input image based on the likelihood map; and the map generation means generates, for each of the plurality of attributes, a likelihood map indicating the likelihood that a predetermined position of an object region when an object of the attribute is detected will exist at coordinates indicating a position in the connected feature map, based on input of the connected feature map to a plurality of classifiers corresponding to the plurality of attributes; and the output means outputs a detection result of the object in the input image based on an integration result obtained by integrating for each coordinate of the likelihood maps corresponding to the plurality of attributes and having common coordinate axes generated by the map generation means.

2. 2. The image processing device according to claim 1, wherein the map generating means is trained in advance to generate a likelihood map indicating the likelihood that a predetermined position of an object region when an object is detected exists at coordinates indicating a position in the connected feature map.

3. further comprising an estimation means for estimating the size of an object present at the predetermined position; 3. The image processing apparatus according to claim 1, wherein the output means outputs the coordinates of the predetermined position of the object and the size of the object.

4. 4. The image processing apparatus according to claim 3, wherein the estimation means estimates the size of the object based on the likelihood map.

5. 2. The image processing apparatus according to claim 1, wherein the predetermined position is a reference point of an object.

6. 6. The image processing device according to claim 5, wherein the reference point is a center of gravity.

7. an input step of inputting an input image into a hierarchical neural network having a plurality of layers; a map generation step of generating, based on a concatenated feature map in which feature maps output from the plurality of layers are concatenated, a likelihood map indicating the likelihood that a predetermined position in an object region when an object is detected exists at coordinates indicating a position in the concatenated feature map; an output step of outputting a result of detecting the object in the input image based on the likelihood map; and an output step of outputting a detection result of the object in the input image based on an integration result of the likelihood maps corresponding to the plurality of attributes generated in the map generation step, the likelihood maps indicating the likelihood that a predetermined position of an object region when an object of the attribute is detected exists at coordinates indicating a position in the connected feature map, the likelihood maps corresponding to the plurality of attributes generated in the map generation step being integrated for each coordinate of the likelihood maps having a common coordinate axis.

8. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 6.

Images