Image processing device, image processing method, and program

Abstract

To perform collation of an image including a target object with good accuracy, by performing feature extraction with focusing on a part useful for the feature extraction of the target object.SOLUTION: An image processing system has: creation means for creating a feature map indicating a feature of a target object from an image; detection means for detecting an attention degree depending on a location on the image from the image based on the location of the component of the target object; compensation means for compensating the feature map based on the attention degree; and collation means for performing collation processing of multiple images including the target object.SELECTED DRAWING: Figure 4

Description

Technical Field

[0002] , , , , , ,

[0004] , , , , , , , , , , ,

[0003] , , , ,

[0001] The present invention relates to a technique for performing image matching.

Background Art

[0002] In recent years, face authentication has been performed by determining that a face image matches and that it is of the same person. In image processing, methods for feature extraction that focus on portions where features are expected to strongly appear in an image are described in Patent Document 1 and Patent Document 2. In face authentication as well, a method for obtaining portions of interest in a face and using them for feature extraction is described in Non-Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] However, conventional methods, including the method described in Non-Patent Document 1, acquire the area of ​​interest through learning, including finding it within the entire image. Therefore, whether a desirable method for extracting the area of ​​interest can be acquired depends on the content of the training dataset. As a result, the area of ​​interest may not be determined, leading to unstable model training, or the model may learn to focus on inappropriate areas, resulting in poor final model accuracy. In such models, if a face image with blur or obscured parts is input, the blurred or obscured parts may affect the recognition, potentially leading to misrecognition.

[0006] This invention was made to solve the above-mentioned problems, and aims to accurately match images containing a target object by focusing on parts useful for extracting features from the target object and performing feature extraction. [Means for solving the problem]

[0007] Therefore, the image processing apparatus according to the present invention includes a generation means for generating a feature map that shows the characteristics of a target object from an image, Position of the components of the aforementioned object Based on, The degree of relationship between the regions of the multiple constituent elements Attention level according to position on the aforementioned image as The system is characterized by comprising: detection means for detecting from the image; correction means for correcting the feature map based on the degree of attention; and matching means for performing matching processing on a plurality of images including the target object using the corrected feature map. [Effects of the Invention]

[0008] According to the present invention, by focusing on parts useful for extracting features from the target object and performing feature extraction, it is possible to accurately match images containing the target object. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the hardware configuration of the image processing device according to Embodiment 1. [Figure 2] This figure shows the functional configuration of the image processing apparatus according to Embodiment 1. [Figure 3] It is a diagram showing data registered in the database according to Embodiment 1. [Figure 4] It is a flowchart showing the image matching method according to Embodiment 1. [Figure 5] It is a diagram showing the data flow of the image processing apparatus according to Embodiment 1. [Figure 6] It is a diagram showing the structure of the neural network according to Embodiment 1. [Figure 7] It is a diagram showing the functional configuration of the image processing apparatus according to Embodiments 2 and 4. [Figure 8] It is a diagram showing the data flow of the image processing apparatus according to Embodiment 2. [Figure 9] It is a diagram showing the functional configuration of the image processing apparatus according to Embodiment 3. [Figure 10] It is a diagram showing the data flow of the image processing apparatus according to Embodiment 3. [Figure 11] It is a diagram showing data registered in the database according to Embodiment 3. [Figure 12] It is a diagram showing the data flow of the image processing apparatus according to Embodiment 4. [Figure 13] It is a diagram showing the structure of the neural network according to Embodiment 4.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0011] <Embodiment 1> It is empirically known that the features of a human face are strongly expressed in organs such as the eyes, mouth, and nose, and it is expected that focusing on such areas where individual features are strongly expressed will be useful in the extraction of facial features. The image processing device according to this embodiment detects areas of interest from a human face image based on the regions of organs such as the eyes, mouth, and nose, and weights the feature map extracted from the face image based on the detection results. This makes it possible to perform face image matching using a feature map that focuses on areas useful for facial feature extraction.

[0012] This embodiment describes a method for matching human facial images using the features of a human face. A human face is an example of a target object. Facial features are an example of characteristic components of a target object. The target object is not limited to a human face; it may also be an animal face, a car, etc. This embodiment can be applied to image matching of various target objects by detecting the region of the characteristic components of the target object instead of the region of the facial features of a human face, using a similar configuration.

[0013] Figure 1 shows an example of the hardware configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 includes a control device 11, a storage device 12, an arithmetic unit 13, an input device 14, an output device 15, and an I / F (Interface) device 16. These components are interconnected via a bus 17. The control device 11 consists of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the entire image processing apparatus 100. The storage device 12 consists of a recording medium such as a hard disk and stores programs and data necessary for the operation of the control device 11. The arithmetic unit 13 consists of an MPU (Micro Processing Unit), and executes necessary arithmetic processing based on control from the control device 11.

[0014] The input device 14 is a human interface device, etc., which inputs user operation information to the image processing device 100. The output device 15 is a display, etc., which displays the processing results of the image processing device 100 under the control of the control device 11. The I / F device 16 is a wired interface such as a universal serial bus, local area network, or optical cable, or a wireless interface such as Wi-Fi® or Bluetooth®. Under the control of the control device 11, the I / F device 16 inputs captured images from the camera, transmits processing results obtained by the image processing device 100 to an external device, or inputs programs and data necessary for the operation of the image processing device 100 from an external device.

[0015] Figure 2 shows an example of the functional configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 includes an image acquisition unit 201, a feature map generation unit 202, a facial feature detection unit 203, a map merging unit 204, a focus detection unit 205, a feature map correction unit 206, a facial feature extraction unit 207, a facial feature matching unit 208, and an output unit 209. The image processing apparatus 100 functions as each of these functional units when the control device 11 executes a program stored in the storage device 12, etc. In addition, the storage device 12 of the image processing apparatus 100 stores a facial feature database 210, and the data held in the facial feature database 210 is read out and used as appropriate by each functional unit.

[0016] The image acquisition unit 201 acquires an input image that includes a human face. The image acquisition unit 201 may acquire a captured image from a camera connected to the I / F device 16 as the input image, or it may acquire image data from the storage device 12 or an external device connected to the I / F device 16. The feature map generation unit 202 extracts features from the input image and generates a feature map representing the extraction results. Specifically, the feature map generation unit 202 associates the features extracted from each part of the image with their positions on the image, thereby generating a feature map in which the extracted features are arranged in two dimensions. The facial organ detection unit 203 detects regions in the input image where facial organs such as eyes, nose, and mouth are present. Specifically, the facial organ detection unit 203 calculates a score map for each of several organs, representing the likelihood of the organ's presence based on its position in the input image. Hereinafter, the score map calculated by the facial organ detection unit 203 will be referred to as the organ map (first map). The map merging unit 204 merges the organ map groups calculated for each of the multiple organs by the facial organ detection unit 203.

[0017] The attention detection unit 205 detects the level of attention corresponding to the position on the input image from the result of the map merging unit 204. Specifically, the attention detection unit 205 calculates an attention map, which is a map that represents the magnitude of attention according to the position on the input image. The feature map correction unit 206 corrects the feature map generated by the feature map generation unit 202 using the attention level map calculated by the attention level detection unit 205. The facial feature extraction unit 207 extracts feature vectors that represent an individual's facial features from the corrected feature map corrected by the feature map correction unit 206. The facial feature matching unit 208 performs a matching process by comparing the feature vectors extracted by the facial feature extraction unit 207 with the feature vectors registered in the facial feature database 210 (described later) to find out which person is the person in question. The output unit 209 displays the verification result on the output device 15 and presents it to the user.

[0018] The facial feature database 210 is a database that holds information and feature vectors of multiple individuals. Figure 3 shows an example of the facial feature database 210 according to this embodiment. In the example shown in Figure 3, the facial feature database 210 holds facial feature records for multiple individuals, each consisting of the following items: person ID, facial image, person information, image information, and feature quantity. The person ID item contains a number that uniquely identifies the facial feature record in the facial feature database 210. The facial image item contains image data showing the person's face. The person information item contains information about the person, such as their name. The person information item may also contain additional information such as address, date of birth, and job title. The image information item contains information associated with the image data registered in the facial image item. In the example in Figure 3, the image information item contains information about the shooting date and registration date. The image information item may also contain additional information such as camera parameters at the time of shooting, image format, and face position. In the feature quantity field, the feature vectors extracted by the face feature extraction unit 207 are registered using the image data registered in the face image field as the input image, in the same manner as the face image matching method shown in Figure 4. In S308 of Figure 4, the face feature matching unit 208 performs matching using the feature vectors registered in the feature quantity field.

[0019] Next, a method for matching facial images using the image processing device 100 according to this embodiment will be described. Figure 4 is a flowchart of the facial image matching method. Figure 5 is a diagram showing the data flow used when matching facial images. In the following description of the flowchart, each step will be preceded by an S, and the notation of the step will be omitted. The processing of each step in the flowchart is realized by the control device 11 executing a program stored in the storage device 12, etc.

[0020] First, in S301, the image acquisition unit 201 acquires the query face image 401 as the input image. As shown in Figure 5, the query face image 401 contains only one human face. The height of the query face image 401 is H and the width is W. The number of channels is 3 (RGB). Note that data other than color representation, such as an alpha channel, may be added. Also, other color spaces such as HSV may be used. Furthermore, the query face image may be a grayscale image with 1 channel.

[0021] Next, in S302, the feature map generation unit 202 extracts features from the query face image 401 and generates a feature map 402. The feature map 402 is calculated by a CNN (Convolutional Neural Network) that has been trained to extract features that represent an individual's facial features, and the feature vector 407 extracted by the subsequent face feature extraction unit 207 is then calculated. As the CNN (Convolutional Neural Network), for example, the Residual Network (ResNet) shown in Non-Patent Literature 4 is used. The CNN used here is an example of a second neural network.

[0022] Feature map 402 is a map with vertical width H / K, horizontal width W / K, and number of channels D1, as shown in Figure 5. The value (vector value of length D1) at position (x,y) of feature map 402 represents the facial features in the range [|x / K|,|x / K|+K) × [|y / K|,|y / K|+K)] of query face image 401. Here, [·) represents the half-open interval upwards, and |·| represents the floor function. The value of K is determined by the size of the final layer of the CNN used to generate the feature map. Increasing K reduces the size of the feature map and thus the computational cost, but on the other hand, the resolution of the feature map for the query face image decreases, resulting in a decrease in accuracy. Therefore, K is determined to obtain an appropriate balance of accuracy and computational cost, and the CNN is trained accordingly.

[0023] Next, in S303, the facial organ detection unit 203 detects the regions where facial organs such as eyes, nose, and mouth exist from the query facial image 401. Specifically, the facial organ detection unit 203 calculates an organ map for each of the multiple organs from the query facial image 401 and generates an organ map group 403. In this embodiment, an organ map is calculated for each of the five organs: the right eye, left eye, nose, right corner of the mouth, and left corner of the mouth, using an organ point position estimation method as described in Non-Patent Literature 3. Note that the organs are not limited to the above five; other organs such as ears, cheeks, and jaws may be added, or conversely, the number of organs may be reduced by omitting the nose or using the center of the mouth instead of the right and left corners of the mouth.

[0024] As shown in Figure 5, each organ map in the organ map group 403 is a map with a vertical width of H / K, a horizontal width of W / K, and 1 channel, and its correspondence with the query face image 401 is the same as that of the feature map 402 described above. Each organ map is a map in which each element takes a positive value, and the sum of all elements is 1. The value of each element represents the probability that the organ exists at the corresponding position in the query face image 401. For example, in the organ map for the right eye, the value is large in the region corresponding to the position of the right eye in the query face image 401, and small in other regions. This organ map tends to have smaller values ​​in regions where there is organ blurring or occlusion. In other words, the organ map contains information about organ blurring and occlusion.

[0025] Next, in S304, the map merging unit 204 merges each organ map of the organ map group 403 calculated by the facial organ detection unit 203 so that each organ map forms a channel, thereby generating a merged map 404. As shown in Figure 5, the merged map 404 is a map with a vertical width of H / K, a horizontal width of W / K, and a number of channels of D2. D2 is the number of organ maps in the organ map group 403, which is 5 in this embodiment. Next, in S305, the attention detection unit 205 calculates the attention map 405 from the combined map 404 generated by the map combining unit 204. As shown in Figure 5, the attention map 405 is a score map with the same dimensions as the feature map, with a vertical width of H / K, a horizontal width of W / K, and 1 channel. The attention map 405 is calculated by a CNN that has been trained so that the value (score) is large in the range of the feature map that has a large contribution, in the calculation of the feature vector 407 extracted by the subsequent face feature extraction unit 207. The CNN used here is an example of a first neural network.

[0026] The structure of the CNN used by the attention detection unit 205 will be explained using Figure 6. As shown in Figure 6, the CNN used by the attention detection unit 205 comprises a first convolution unit 1001 and a second convolution unit 1002. The first convolution unit 1001 is a network that takes a map with channel count D2 as input, performs a 1x1 convolution, applies an activation function to each element, and outputs a map with channel count L. The activation function used is a normalized linear function (ReLU: Rectified Linear Unit). The second convolution unit 1002 is a network that takes a map with channel count L as input, performs a 1x1 convolution, applies ReLU to each element, and outputs a map with channel count 1.

[0027] The attention detection unit 205 first inputs the combined map 404 to the first convolution unit 1001. Since the combined map 404 has a vertical width of H / K, a horizontal width of W / K, and D2 channels, the intermediate map output from the first convolution unit 1001 is a map with a vertical width of H / K, a horizontal width of W / K, and L channels. Next, the attention detection unit 205 inputs the intermediate map output from the first convolution unit 1001 to the second convolution unit 1002. The map output from the second convolution unit 1002 is a map with vertical width H / K, horizontal width W / K, and 1 channel. This is referred to as the attention map 405. In this embodiment, L is set to 3. As D2 is 5 as described above, the number of channels in the maps output by the CNN of the attention detection unit 205 will be 5, 3, and 1, respectively. The value of L is appropriately selected according to the value of D2.

[0028] Next, in S306, the feature map correction unit 206 corrects the feature map 402 generated by the feature map generation unit 202 in S302 using the attention level map 405 calculated by the attention level detection unit 205 in S305 to generate a corrected feature map 406. As mentioned above, the combined map 404 contains information such as organ blur and occlusion, so by correcting the feature map 402 using the attention level map 405 calculated using the combined map 404, the effects of organ blur and occlusion can be excluded from the feature map 402. In other words, a feature map corrected to focus on the appropriate parts can be generated.

[0029] The corrected feature map 406 is obtained by taking the Hadamard product of each layer of the feature map 402 and the attention map 405. That is, the feature map correction unit 206 multiplies the elements (y, x, d) of the feature map 402 by the values ​​of the elements (y, x) of the attention map 405 as weights. As a result, the corrected feature map 406 is obtained in which the values ​​of the parts of the feature map 402 that correspond to the attention map 405 and have large values, i.e., the parts with high attention, are emphasized. Note that the feature map correction unit 206 does not simply take the Hadamard product to perform the correction, but may, for example, transform the values ​​of each element of the attention map 405 with a sigmoid function f(x) = 1 / (1 + exp(-ax)) before taking the Hadamard product. The gain a of the sigmoid function may be set to, for example, 1, or it may be determined by learning as described later. Also, the correction method is not limited to the method using the sigmoid function.

[0030] Next, in S307, the face feature extraction unit 207 extracts a feature vector 407 from the corrected feature map 406. Specifically, as shown in Figure 5, the face feature extraction unit 207 calculates the feature vector 407 using a fully connected neural network that calculates a vector value of dimension N from a map with vertical width H / K, horizontal width W / K, and number of channels D1. The feature vector 407 is calculated by a neural network trained to extract values ​​that represent face features that contribute to individual identification. The neural network used here is an example of a third neural network. Next, in S308, the face feature matching unit 208 compares the feature vector 407 extracted by the face feature extraction unit 207 in S307 with the feature vectors held in the face feature database 210 to determine whose face the face in the query face image 401 belongs to. The face feature matching unit 208 performs the matching by searching for the face feature record in the face feature database 210 whose registered feature vector in the feature quantity field is considered to be the closest to the feature vector 407 (let's call it F) calculated from the query face image 401. The method is described below.

[0031] First, the face feature matching unit 208 reads a feature vector 408 from the face feature database 210, as shown in Figure 5. Let i be the person ID of the face feature record containing the feature vector 408, and let Fi be the feature vector included in the record where person ID is i. Next, the face feature matching unit 208 calculates the cosine similarity between F and Fi, and the result of this calculation is the similarity score Si. The face feature matching unit 208 calculates the similarity score for all feature vectors included in the face feature database 210. Next, the face feature matching unit 208 finds I = argmax{Si} as the person ID with the maximum similarity score, and determines whether the similarity score SI exceeds the threshold T (for example, 0.8). If the similarity score SI exceeds the threshold T, the face feature matching unit 208 determines the matching result by assuming that the person in the query face image 401 is the same person as the person in the face feature record where person ID is I. If the similarity score SI does not exceed the threshold T, the facial feature matching unit 208 determines that there is no person whose face matches the one shown in the query facial image 401.

[0032] Finally, in S309, the output unit 209 displays the matching result on the output device 15 and presents it to the user. The output unit 209 retrieves the face feature record with person ID I from the face feature database 210 and displays the image data and various information registered in the face image, person information, and image information fields on the output device 15. This allows the user to find out which of the people registered in the face feature database 210 corresponds to the human face shown in the query face image 401. In this way, the series of processes for matching the face image shown in Figure 4 is completed.

[0033] In order to perform the facial image matching described above, the feature vector 407 calculated by the facial feature extraction unit 207 must have the characteristic that the cosine similarity is high for the same person and low for different people. The networks used in the feature map generation unit 202, the attention detection unit 205, and the facial feature extraction unit 207 are pre-trained so that the feature vector 407 with such characteristics is calculated as a result.

[0034] Network training is performed simultaneously and as a single unit using a method known as end-to-end learning, which integrates the networks used in the feature map generation unit 202, the attention detection unit 205, and the face feature extraction unit 207. First, the area enclosed by the dashed line in the configuration of Figure 2 is prepared as an integrated network 250, connected in the same way as the processing flow and input / output order described earlier. That is, the integrated network 250 is configured by connecting the networks so that the outputs of the feature map generation unit 202 and the attention detection unit 205 are corrected by the feature map correction unit 206, and the corrected map becomes the input to the face feature extraction unit 207. The input and output of the integrated network 250 are the query face image 401 shown in Figure 5 and the combined map 404 obtained by combining the organ map obtained by facial organ detection from the query face image 401, and the output is a feature vector 407 of dimension N.

[0035] This integrated network 250 is trained so that the feature vector 407 is calculated with characteristics that capture features effective for face matching. This acquires network parameters that allow the face feature extraction unit 207 to appropriately calculate the feature vector 407. At the same time, the network parameters used by the feature map generation unit 202 and the attention detection unit 205 are also acquired so that the corrected feature map 406, which is the input to the face feature extraction unit 207, is calculated with information useful for calculating a feature vector 407 with effective characteristics. It should be noted that this embodiment is not limited to the method of simultaneously training the network parameters used by the feature map generation unit 202, the attention detection unit 205, and the face feature extraction unit 207 as a whole, but may also be used to train the individual network parameters separately. For example, the network parameters used by the feature map generation unit 202 may be trained first, and then the network parameters used by the attention detection unit 205 and the feature map correction unit 206 may be trained using these trained parameters.

[0036] The corrected feature map 406 is calculated by taking the Hadamard product of each layer of the feature map 402 and the attention map 405. Therefore, the attention map 405 is calculated as a map that gives greater weight to the parts of the feature map 402 that contain information useful for calculating the feature vector 407. In other words, the attention map 405 is a map calculated so that the values ​​of the locations of interest in the feature map 402, and by extension the query face image 401, are larger, and can be interpreted as a map that represents the degree of attention.

[0037] The integrated network 250 is trained as follows. First, a large number of face images are prepared, and classification labels are assigned so that the same person belongs to the same class and different people belong to different classes to prepare a training dataset. Furthermore, a combined map is calculated from each face image by the face organ detection unit 203 and the map combining unit 204. The combined map calculated here and the original face image pair are used as input to the integrated network 250 for subsequent training. Next, an integrated network 250 is prepared with appropriate initial values. The initial values ​​may be constants or random numbers, or, for example, parameters of a pre-trained ResNet may be used. Then, the parameters of the integrated network 250 are updated using the training dataset, for example, by the method described in Non-Patent Document 5. This training proceeds so that the cosine similarity of the output vectors is large for face images of the same classification class and small for face images of different classification classes. After training with a sufficient number of face images, the parameters of the obtained integrated network 250 are stored in the memory device 12 and used as network parameters for the feature map generation unit 202, attention detection unit 205, and face feature extraction unit 207. If the feature map correction unit 206 has parameters such as the gain of the sigmoid function mentioned above, those parameters may also be trained at the same time. As described above, in this embodiment, the organ map score is used as training data by utilizing the characteristic that the organ map score is smaller in areas where organs are hidden or unclear. This makes it possible to generate a learning model that appropriately defines the areas of interest according to the state of organs, such as occlusion or blurring.

[0038] As described above, the image processing apparatus 100 according to this embodiment makes it possible to perform feature extraction that focuses on parts useful for facial feature extraction by using the regions of facial features to define areas of interest in a human face image. This enables accurate matching of images containing target objects such as human faces.

[0039] <Embodiment 2> In Embodiment 1, the attention map was calculated using only the organ map group. However, by adding information from the feature map itself before correction, it may be possible to extract the parts of interest in the feature space more accurately. This embodiment describes a method for calculating the attention map from the organ map group and the feature map. Note that the parts common to Embodiment 1 will be omitted from the explanation, and the explanation will focus on the differences from Embodiment 1.

[0040] Figure 7 shows an example of the functional configuration of the image processing apparatus 100 according to this embodiment. The functional configuration of this embodiment is the same as that of Embodiment 1, but a connection between the feature map generation unit 202 and the map merging unit 204 is added. That is, the map merging unit 204 merges the organ map group calculated by the facial organ detection unit 203 and the feature map generated by the feature map generation unit 202. Figure 8 shows the data flow used in this embodiment. The method of matching facial images performed by the image processing apparatus 100 according to this embodiment is the same as in Figure 4, but the processing performed by the map merging unit 204 in S304 and the processing performed by the attention level detection unit 205 in S305 differ from Embodiment 1, as will be explained below.

[0041] In this embodiment, in S304, the map merging unit 204 generates a merged map 404 by merging each organ map of the organ map group 403 calculated by the facial organ detection unit 203, along with the feature map 402 generated in S302, in the depth direction. As shown in Figure 8, the merged map 404 has a vertical width of H / K and a horizontal width of W / K. The number of channels D2 is the number of organs in the organ map group 403 plus the number of channels D1 of the feature map 402. Furthermore, in this embodiment, in S305, the attention detection unit 205 uses the combined map 404, which is combined as described above, to calculate the attention map 405.

[0042] In this embodiment, as in Embodiment 1, the parameters of the networks used by the feature map generation unit 202, the attention level detection unit 205, and the face feature extraction unit 207 are those learned by an integrated network connecting them. In the configuration of Figure 7, the area enclosed by the dashed line is the integrated network 1250 according to this embodiment. The integrated network 1250 includes the functions of the map merging unit 204 according to this embodiment. That is, the input to the network used by the attention level detection unit 205 is a combined map 404 that combines the organ map and the output of the network used by the feature map generation unit 202. The method for learning the integrated network 1250 is the same as in Embodiment 1.

[0043] As described above, the image processing apparatus 100 according to this embodiment makes it possible to detect the degree of attention considering the information in the feature map. Since the feature map itself may contain information for determining the location of attention, by correcting the feature map using this degree of attention, it is possible to accurately match images including target objects such as human faces.

[0044] <Embodiment 3> In Embodiments 1 and 2, the feature map was corrected in advance based on the level of attention; however, it is also possible to correct the feature map at the time of matching. This embodiment describes a method of performing correction using the level of attention map at the time of matching. Note that the explanation will be omitted for parts that are common with Embodiment 1, and the explanation will focus on the differences from Embodiment 1.

[0045] Figure 9 shows an example of the functional configuration of the image processing apparatus 100 according to this embodiment. The functional configuration of this embodiment is the same as that of Embodiment 1, but a connection between the feature map correction unit 206 and the face feature database 210 is added. That is, the feature map correction unit 206 also corrects the feature maps stored in the face feature database 210 using the attention level maps detected by the attention level detection unit 205. Figure 10 shows the data flow used in this embodiment. In this embodiment, some of the data held in the face feature database 210 differs from that of Embodiment 1, as will be explained below. Also, the method of matching face images performed by the image processing apparatus 100 according to this embodiment is the same as that in Figure 4, but the content of the processing performed by the face feature matching unit 208 in S308 differs from that of Embodiment 1, as will be explained below.

[0046] Figure 11 shows an example of a face feature database 210 according to this embodiment. In the face feature database 210 according to this embodiment, instead of the feature vectors extracted by the face feature extraction unit 207, a feature map 701 generated by the feature map generation unit 202 in the same manner as in Embodiment 1 is registered in the feature quantity field. In this embodiment, in S308, the feature map correction unit 206 first reads the feature map 701 from the face feature database 210, as shown in Figure 10. The person ID of the face feature record containing the feature map 701 is denoted as i, and the feature map included in the record where the person ID is i is denoted as Mi. Next, the feature map correction unit 206 corrects the feature map Mi in the same manner as in S306 of Embodiment 1 to generate a corrected feature map 702, and then extracts the feature vector 408(Fi) from the corrected feature map 702 in the same manner as in S307 of Embodiment 1. Then, the face feature matching unit 208 calculates the similarity with the feature vector F in the same manner as in S308 of Embodiment 1 and performs the matching. The learning method for the networks used by the feature map generation unit 202, the attention detection unit 205, and the face feature extraction unit 207 in this embodiment is the same as in Embodiment 1.

[0047] As described above, the image processing device 100 according to this embodiment makes it possible to correct the feature map during matching. This eliminates the need to recreate the contents of the face feature database 210 even if the network parameters used in the attention detection unit 205 change. Therefore, it becomes possible to add a configuration that allows the network parameters used in the attention detection unit 205 to be changed according to conditions such as the usage environment, or to be updated as appropriate through additional learning.

[0048] <Embodiment 4> In Embodiments 1 to 3, the degree of attention was determined by focusing on local areas of organs within the image. However, it is also possible to use a degree of attention that focuses on the relationships between regions containing organs. In this embodiment, a method of correction using a degree of attention map that represents the relationships between regions containing organs will be described. This embodiment will be described based on Embodiment 2, and the parts common to Embodiment 2 will be omitted from the explanation, with the focus being on the differences from Embodiment 2.

[0049] The functional configuration of this embodiment is the same as that of Embodiment 2 shown in Figure 7. Figure 12 is a diagram showing the data flow used in this embodiment. Furthermore, the method of matching face images performed by the image processing device 100 according to this embodiment is the same as that in Figure 4, but the processing performed by the attention detection unit 205 in S305 and the processing performed by the feature map correction unit 206 in S306 differ from that of Embodiment 2, as will be explained below. In the following description, a combined map is used, which combines the organ map group and the feature map, as in Embodiment 2. However, as in Embodiment 1, it is also possible to calculate the attention map using a combined map that combines only the organ map group, in the same manner as in Embodiment 4.

[0050] In this embodiment, in S305, the attention level map 405 calculated by the attention level detection unit 205 has a vertical and horizontal width of N = ((H / K) × (W / K) × D1), as shown in Figure 12. That is, the vertical and horizontal widths are the number of elements of the feature map 402. The number of channels is 1. Such an attention level map 405 can be calculated using a neural network with a structure like that shown in Non-Patent Literature 2, for example.

[0051] The structure of the neural network used by the attention detection unit 205 according to this embodiment will be explained with reference to Figure 13. As shown in Figure 13, the neural network used by the attention detection unit 205 according to this embodiment comprises a first convolution unit 1101, a second convolution unit 1102, a first vectorization unit 1103, a second vectorization unit 1104, and a matrix calculation unit 1105. The first convolutional section 1101 and the second convolutional section 1102 are networks that take a map with channel count D2 obtained by the map merging section 204 as input, perform a 1x1 convolution, apply ReLU to each element, and output a map with channel count D1. Different parameters are applied to the networks of the first convolutional section 1101 and the second convolutional section 1102.

[0052] The first vectorization unit 1103 and the second vectorization unit 1104 vectorize an input map with vertical width H / K, horizontal width W / K, and number of channels D1, converting it into an output vector of dimension N = ((H / K) × (W / K) × D1). The elements (y, x, d) of the input map become the (y × (W / K × D1) + x × D1 + d)th element of the output vector. The output vector of the first vectorization unit 1103 is a horizontal vector, and the output vector of the second vectorization unit 1104 is a vertical vector. The matrix calculation unit 1105 takes an N-dimensional row vector and a column vector as input, calculates an N×N matrix by taking the matrix product of the row vector multiplied by the column vector from the right, and then outputs the result after smoothing it using, for example, the softmax function.

[0053] In this embodiment, in S305, the attention detection unit 205 first inputs the combined map 404 to the first convolution unit 1101 to calculate the first intermediate map. Furthermore, the attention detection unit 205 also inputs the combined map 404 to the second convolution unit 1102 to calculate the second intermediate map. Next, the first intermediate map is input to the first vectorization unit 1103 to calculate the first intermediate vector. Furthermore, the second intermediate map is input to the second vectorization unit 1104 to calculate the second intermediate vector. Then, the attention detection unit 205 inputs the first intermediate vector and the second intermediate vector to the matrix calculation unit 1105 to obtain an output. This is the attention map 405 according to this embodiment. The attention map 405 in this embodiment is a map with width N and height N.

[0054] In this embodiment, the attention map 405 has an attention value that represents the degree of association between any two points in the feature map. That is, the attention value representing the relationship between (y2, x2, d2) and elements (y1, x1, d1) in the feature map 402 corresponds to the values ​​of elements (y1×(W / K×D1)+x1×D1+d1, y2×(W / K×D1)+x2×D1+d2) in the attention map 405. The attention map 405 is not necessarily a symmetric matrix, and the attention value of (y1, x1, d1) for elements (y2, x2, d2) is not necessarily equal to the attention value of (y2, x2, d2) for elements (y1, x1, d1). Note that an additional step, such as adding the transpose matrix, may be taken to make them equal.

[0055] The attention map 405 calculated in this way suggests that, for example, if there is a high correlation between the right and left eyes, then the attention given to the left eye region relative to the right eye region, and vice versa, is expected to be high. In other words, it is thought that the attention given to areas with a high correlation between the regions of organs is high.

[0056] Next, the correction method performed by the feature map correction unit 206 in S305 of this embodiment will be explained. First, the feature map correction unit 206 vectorizes the feature map 402 in the same way as the first vectorization unit 1103. That is, the feature map correction unit 206 generates an N-dimensional vector V in which the elements (y, x, d) of the feature map 402 are (y × (W / K × D1) + x × D1 + d)th elements. Next, the feature map correction unit 206 takes the matrix product of the attention map 405 and V. The result of the matrix product, V', is a vector with the same dimensions as V. Finally, the feature map correction unit 206 generates a map with vertical width H / K, horizontal width W / K, and number of channels D1 in which the (y × (W / K × D1) + x × D1 + d)th element of V' is the element (y, x, d). This is the corrected feature map 406. The learning method for the networks used by the feature map generation unit 202, the attention level detection unit 205, and the face feature extraction unit 207 in this embodiment is the same as in Embodiment 1, except for the network structure used by the attention level detection unit 205 as described above.

[0057] As described above, the image processing device 100 according to this embodiment makes it possible to detect the relationship between organ regions as a degree of attention. By correcting the feature map using this degree of attention, it is possible to accurately match images that include target objects such as human faces.

[0058] <Other Embodiments> Although each embodiment has been described in detail above, the present invention can take the form of, for example, a system, apparatus, method, program, or recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, interface devices, imaging devices, web applications, etc.), or to an apparatus consisting of a single device.

[0059] Furthermore, the object of the present invention can also be achieved as follows: a recording medium (or storage medium) containing software program code (computer program) that realizes the functions of each of the embodiments described above is supplied to a system or device. The storage medium is a computer-readable storage medium. The computer (or CPU or MPU) of the system or device then reads and executes the program code stored on the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the embodiments described above, and the recording medium containing that program code constitutes the present invention. [Explanation of Symbols]

[0060] 100: Image processing device

Claims

1. A generation means for generating a feature map that shows the characteristics of a target object from an image, A detection means for detecting the degree of relationship between regions of a plurality of components of the target object, based on the positions of the components of the target object, as a degree of attention corresponding to the position on the image, from the image. A correction means for correcting the feature map based on the aforementioned attention level, A matching means that performs matching processing of multiple images including the target object using the corrected feature map, An image processing apparatus characterized by having

2. The system further includes a calculation means for calculating a first map that represents the likelihood of the element being present at the location on the image, The image processing apparatus according to claim 1, characterized in that the detection means detects the degree of attention based on the first map.

3. The calculation means calculates the first map for each of the plurality of components, The image processing apparatus according to claim 2, characterized in that the detection means detects the degree of attention based on a combined map obtained by combining the first group of maps calculated by the calculation means.

4. The image processing apparatus according to claim 3, characterized in that the detection means detects the degree of attention based on a combined map obtained by further combining the feature map with the first map group.

5. The image processing apparatus according to claim 3 or 4, wherein the detection means calculates the attention map by a first neural network that takes the combined map as input and outputs an attention map representing the degree of attention at each position on the image.

6. The image processing apparatus according to claim 5, wherein the generation means generates the feature map by a second neural network that inputs an image including the target object and outputs the feature map.

7. The image processing apparatus according to claim 6, wherein the matching means extracts the feature quantities by a third neural network that inputs the corrected feature map for each of a plurality of images including the target object and outputs feature quantities that represent features contributing to the identification of the target object, and then compares the extracted feature quantities to perform the matching process.

8. The image processing apparatus according to claim 7, further comprising a learning means for learning at least one of the parameters used in the first neural network, the second neural network, and the third neural network, such that the corrected feature map obtained by correcting the output from the second neural network with the output from the first neural network is input to the third neural network, and the feature quantity output captures features that contribute to the identification of the target object.

9. The system further includes a storage means for storing the feature map generated by the generation means, The image processing apparatus according to any one of claims 1 to 8, characterized in that the correction means reads the feature map from the storage means and corrects the read feature map when performing the matching process.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that the correction means corrects the feature map by assigning a weight to the feature map according to the degree of attention.

11. The image processing apparatus according to claim 10, characterized in that the correction means corrects the feature map by taking the Hadamard product of each layer of the feature map and the level of attention.

12. The image processing apparatus according to any one of claims 1 to 11, characterized in that the target object is a human face and the constituent elements are facial organs.

13. A generation process that generates a feature map showing the characteristics of the target object from an image, A detection step of detecting the degree of relationship between regions of a plurality of components based on the positions of the components of the target object, as a degree of attention corresponding to the position on the image, from the image; A correction step is performed to correct the feature map based on the aforementioned attention level, A matching step which performs matching processing on multiple images including the target object using the corrected feature map, An image processing method characterized by including

14. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images