Image processing device, image processing method and computer program

Abstract

To provide an image processing device capable of quickly searching for a predetermined subject in a case where a number of small subjects are present in an image to be searched, or in other cases.SOLUTION: An image processing device includes: image acquisition means that acquires a query image as an image including a search object, and an image to be searched; feature vector extraction means that extracts a first feature vector indicating a feature of the search object included in the query image; feature vector map extraction means that extracts a feature vector map indicating a feature vector of the subject at each position of the image to be searched; and heat map calculation means that makes a calculation between the first feature vector extracted by the feature vector extraction means and the feature vector map extracted by the feature vector extraction means to calculate a heat map indicating a probability that the search object is present.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a computer program for image retrieval. [Background technology]

[0002] In recent years, many feature extraction techniques have been proposed to extract useful information from images of objects captured in captured images through advanced processing. Among these, feature extraction techniques that use multi-layered neural networks called deep nets (also called deep neural networks or deep learning) to extract feature vectors of objects in images have been actively researched.

[0003] The feature extractor is trained by a method called distance learning. In distance learning, the distance between feature vectors is first defined. Examples of distance definitions include Euclidean distance and cosine similarity.

[0004] In distance learning, the feature extractor is trained in advance to output feature vectors with short distances for images of the same object or objects belonging to the same class, and to output feature vectors with long distances for images of objects that are not the same or objects that do not belong to the same class.

[0005] One application of distance learning is face recognition. A feature extractor is trained to output a close feature vector for images of the same person, and a distant feature vector for images of other people. A first face image and a second face image are input to the feature extractor, respectively, to obtain a first feature vector and a second feature vector.

[0006] By calculating the distance between the first feature vector and the second feature vector and comparing the distance with a predetermined threshold, it is possible to determine whether the person in the first face image and the person in the second face image are the same person or different people. A technology for achieving face recognition using distance learning is disclosed in Non-Patent Document 1.

[0007] Another application of distance learning is determining the identity of cars. A feature extractor is trained to output feature vectors that are close to each other for images of the same car model, and feature vectors that are far apart for images of different car models. An image of a first car and an image of a second car are input to the feature extractor, respectively, to obtain a first feature vector and a second feature vector.

[0008] By calculating the distance between the first feature vector and the second feature vector and comparing the distance with a predetermined threshold, it is possible to determine whether the car shown in the first car image and the car shown in the second car image are the same model or different models. In a method of determining the model of a car by a classifier that classifies the model of the car, the classifier needs to be retrained when a new model is released, but by using a method of training a feature extractor, it is possible to handle new car models without retraining.

[0009] One use case of a feature extractor is a face search system. For example, if it were possible to quickly search for the face of a specific target, such as a lost child or a wanted criminal, among the many faces captured on surveillance camera footage, it would be possible to prevent accidents and crimes. An example of a face search system is disclosed in Patent Document 1.

[0010] In this system, facial features of the person to be searched are registered in advance in an authentication database. Then, facial regions of the person are extracted from the image to be searched, which is taken by a security camera, and features are calculated for each of the extracted facial regions. If any of the obtained features is sufficiently close to the features registered in the face recognition database, the search target is deemed to have been found. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] JP 2020-52822 A [Non-patent literature]

[0012] [Non-Patent Document 1] Deng, Jiankang, et al. “Arcface: Additive angular margin loss for deep face recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. Summary of the Invention [Problem to be solved by the invention]

[0013] The object search system as described above must first perform object detection on the image to be searched, and then perform feature extraction and feature vector comparison for each of the detected objects. This means that there is a problem that the detection time increases when many small objects are included in the image to be searched. For example, the feature extractor disclosed in Non-Patent Document 1 is designed to extract a feature vector from a face image in which the vertical and horizontal widths of the face position normalized are both 112 pixels, based on the positions of five face organ points. In addition, the feature extractor disclosed in Non-Patent Document 1 employs a deep net with a large number of parameters, and has high accuracy in identifying human faces, but requires a long processing time. In other words, it can be said that the system is designed to identify faces with high accuracy from clear face images. Therefore, when many small faces are included in the image to be searched, the processing time increases, making it difficult to quickly search for a specific face.

[0014] Therefore, an object of the present invention is to provide an image processing device capable of searching for a predetermined subject at high speed when a large number of small subjects exist in an image to be searched. [Means for solving the problem]

[0015] One aspect of the present invention is an image processing device, comprising: A query image is an image including a subject to be searched, and an image acquisition means acquires a searched image; a feature vector extraction means for extracting a first feature vector representing a feature of the search target included in the query image; a feature vector map extracting means for extracting a feature vector map representing a feature vector of a subject at each position of the searched image; a heat map calculation means for calculating a heat map representing the likelihood that the search target exists by performing an operation between the first feature vector extracted by the feature vector extraction means and the feature vector map extracted by the feature vector map extraction means; The present invention is characterized by having the following. Effect of the Invention

[0016] According to the present invention, it is possible to realize an image processing device capable of searching for a predetermined subject at high speed when a large number of small subjects exist in an image to be searched. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 2] 2 is a functional block diagram showing an example of a functional configuration of the image processing device according to the first embodiment. FIG. [Diagram 3] 5 is a flowchart showing the flow of a learning process by the image processing device according to the first embodiment. [Figure 4] FIG. 13 is a diagram illustrating an example of normalization of a face image. [Diagram 5] FIG. 13 is a diagram showing an example of creating a correct feature vector map. [Figure 6] 5 is a flowchart showing the flow of search processing by the image processing device according to the first embodiment. [Figure 7] FIG. 1 shows an overall system for searching for a specific face. [Figure 8] FIG. 11 is a functional block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Figure 9] 11 is a flowchart showing the flow of search processing by the image processing device according to the second embodiment. [Figure 10] FIG. 11 is a functional block diagram showing an example of a functional configuration of an image processing device according to a third embodiment. [Figure 11] 13 is a flowchart showing the flow of search processing by an image processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0019] In the following, an example will be described in which a specific face is searched for as a subject. The face to be searched for is called a query face, an image that is known to contain the query face is called a query image, and an image that may contain the query face is called a searched image. The objective is to find the query face from the searched image.

[0020] However, the search subject of the present invention is not limited to a face, and may be any subject whose identity can be determined by a feature vector. That is, in the embodiment, an example in which a face is used as a query subject will be described, but the query subject is not limited to a face. For example, the query subject may be a specific animal or a specific object.

[0021] <Embodiment 1> 1 is a diagram showing an example of the hardware configuration of an image processing device according to embodiment 1. The image processing device 100 has, as the hardware configuration, a control device 11, a storage device 12, an input device 13, an output device 14, an I / F device 15, and the like.

[0022] The control device 11 has a built-in CPU as a computer, and controls the entire image processing device 100. The storage device 12 holds computer programs and data necessary for the operation of the control device 11. The input device 13 is a human interface device or the like, and inputs user operations to the image processing device 100.

[0023] The output device 14 generates display image signals and the like to be displayed on a display device (not shown) such as a display, and presents to a user the processing results and the like of the image processing device 100. The I / F device 15 is a wired interface such as a universal serial bus, Ethernet (registered trademark), or optical cable, or a wireless interface such as Wi-Fi or Bluetooth (registered trademark).

[0024] The I / F device 15 has functions such as connecting a camera or the like to input captured images to the image processing device 100, transmitting processing results obtained by the image processing device 100 to the outside, and inputting programs, data, etc. necessary for the operation of the image processing device 100 to the image processing device 100.

[0025] Fig. 2 is a functional block diagram showing an example of the functional configuration of the image processing device in embodiment 1. Note that some of the functional blocks shown in Fig. 2 are realized by causing a CPU serving as a computer included in the control device to execute a computer program stored in a memory serving as a storage medium.

[0026] However, some or all of these functions may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0027] In addition, the functional blocks shown in Fig. 2 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 2 also applies to Figs. 5, 7, 8, and 10.

[0028] The image processing device 100 has, as functional components, an image acquisition unit 201, a feature vector extraction unit 202, a feature vector map extraction unit 203, a heat map calculation unit 204, and a face position estimation unit 205. The image processing device 100 also has a feature extractor learning unit 206 and a feature vector map extractor learning unit 207.

[0029] The image acquisition unit 201, which serves as an image acquisition means, acquires a query image, which is an image including a subject to be searched, and a searched image which may include the search subject. The query image and the searched image acquired by the image acquisition unit 201 may be images captured by a security camera or the like, images recorded in a storage device such as a hard disk, or images received via a network such as the Internet.

[0030] The query image acquired by the image acquisition unit 201 is sent to the feature vector extraction unit 202 , and the image to be searched acquired by the image acquisition unit 201 is sent to the feature vector map extraction unit 203 .

[0031] The feature vector extraction unit 202 detects a query face from the query image acquired by the image acquisition unit 201, and extracts a feature vector of the query face. The feature vector extracted by the feature vector extraction unit 202 is sent to the heat map calculation unit 204. Here, the feature vector extraction unit 202 functions as a feature vector extraction means that extracts a first feature vector representing a feature of the search target included in the query image.

[0032] The feature vector map extraction unit 203 extracts a feature vector map from the to-be-searched image acquired by the image acquisition unit 201. The feature vector map extracted by the feature vector map extraction unit 203 is sent to the heat map calculation unit 204. Here, the feature vector map extraction unit 203 functions as a feature vector map extraction means that extracts a feature vector map that represents the feature vector of the subject at each position of the to-be-searched image.

[0033] The heat map calculation unit 204 calculates a heat map from the feature vectors extracted by the feature vector extraction unit 202 and the feature vector map extracted by the feature vector map extraction unit 203.

[0034] Here, the heat map calculation unit 204 functions as a heat map calculation means that performs an operation between the first feature vector and the feature vector map to calculate a heat map that indicates the likelihood that the search target exists. The heat map estimated by the heat map calculation unit 204 is sent to the face position estimation unit 205.

[0035] The face position estimation unit 205 estimates a candidate position where the query face is likely to appear in the searched image from the heat map estimated by the heat map calculation unit 204. Here, the face position estimation unit 205 functions as a candidate position estimation means that estimates a candidate position, which is a position where the search target is likely to appear in the searched image from the heat map.

[0036] The feature extractor learning unit 206 learns the feature extractor used by the feature vector extraction unit 202 when extracting a feature vector from an image, and stores the parameters thereof as feature extractor parameters 208 in the storage device 12 .

[0037] The feature vector map extractor learning unit 207 learns the feature vector map extractor used when the feature vector map extraction unit 203 extracts a feature vector map from an image. Then, the parameters are stored in the storage device 12 as feature vector map extractor parameters 209.

[0038] The image processing device 100 according to the present embodiment trains a feature extractor and a feature vector map extractor in a learning phase, and searches for a specific subject using the trained feature extractor and feature vector map extractor in a search phase. The flow of the learning process and the flow of the search process will be described below in order.

[0039] An example of the flow of the learning process of the image processing device 100 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of the learning process by the image processing device according to embodiment 1. Note that the operation of each step in the flowchart in Fig. 3 is performed by a CPU serving as a computer in the control device executing a computer program stored in a memory.

[0040] In step S301, the feature extractor learning unit 206 learns a feature extractor used by the feature vector extraction unit 202 when extracting a feature vector from an image, and stores the parameters as feature extractor parameters 208 in the storage device 12. The feature extractor is trained by a known method called distance learning, or the like.

[0041] We will now explain how to create training data used to train the feature extractor. First, collect a large number of face images with known person IDs. The number of different person IDs in the training data is C, and the person IDs are assigned in order from 1 to C.

[0042] Next, a normalized face image is obtained by performing normalization processing on the face image. A normalized face image is an image extracted so that both the vertical and horizontal widths are a predetermined number of pixels, the face is not tilted to the left or right, the size of the face is almost constant, and the relative position of the face to the normalized face image is almost constant. The purpose of normalization processing is to reduce the variation in the in-plane rotation angle and size of the face captured in the face image.

[0043] 4 is a diagram showing an example of normalization of a face image, in which a face image 400 is an image including one face 410 whose ID (identifier) ​​as identification information is known. For the face image 400, the positions of organ points such as the eyes, nose, and mouth of the face 410 are estimated using a known organ point position estimation method.

[0044] If the estimation of the organ point positions fails, the face image is not included in the learning data. In Fig. 4, as an example, it is assumed that the positions of the right eye center 401, the left eye center 402, the right mouth edge 403, and the left mouth edge 404 are estimated.

[0045] Next, based on the positions of the organ points, a normalized face image 405 is extracted from the face image 400. This extraction is performed, for example, by applying a geometric transformation such as an affine transformation to the face image 400 so that the right eye center 401, the left eye center 402, the right mouth edge 403, and the left mouth edge 404 approach predetermined positions 406, 407, 408, and 409, respectively, of the normalized face image 405. By the above method, learning data can be created.

[0046] Next, the feature extractor is trained using the created training data. The feature extractor is a deep net that takes a normalized face image as input and outputs a fixed-dimensional feature vector. Hereinafter, the number of dimensions of the feature vector is defined as D. D can take values ​​such as 128, 256, or 512.

[0047] The feature extractor can be constructed by connecting a fully connected layer to a network that is a derivative of a CNN (Convolutional Neural Network), such as ResNet (Residual Neural Network) or VGG (Visual Geometry Group). In this case, the output of the fully connected layer is designed to be D.

[0048] When training the feature extractor, a classification layer is connected after the feature extractor, which takes the feature vector as input and outputs a C-dimensional probability vector. The i-th component of the probability vector (i is an integer between 1 and C) represents the probability that the person ID of the face captured in the face image is i. The feature extractor and classification layer are trained using methods such as backpropagation so that the probability vector output by the classification layer approaches a one-hot vector.

[0049] Here, a One-Hot vector is a vector in which, when the ID of the correct person in the face image is j, only the j-th component is 1 and the other components are 0. When training the feature extractor, a constraint is set so that the length of the feature vector output by the feature extractor (e.g., L2 norm) approaches a predetermined value.

[0050] The feature extractor is pre-trained according to a distance learning framework, i.e., the feature extractor is pre-trained to output feature vectors with close distances for images of the same object or objects belonging to the same class, and to output feature vectors with far distances for images of non-identical objects or objects not belonging to the same class.

[0051] As an example, the feature extractor may be trained by a method in which the angle between feature vectors output by the feature extractor is defined as the distance, as in Non-Patent Document 1. That is, for images including the same subject or subjects belonging to the same class, the feature extractor is trained so that the angle between the feature vectors output by the feature extractor becomes smaller than a predetermined angle. Also, for images including non-identical objects or objects belonging to non-identical classes, the feature extractor is trained so that the angle between the feature vectors output by the feature extractor becomes larger.

[0052] It is assumed that images including the same subject or subjects belonging to the same class are input to the feature vector extraction unit 202 and the feature vector map extraction unit 203. In this case, learning may be performed in advance so that the difference in distance between the feature vector output by the feature vector extraction unit 202 and the feature vector at the position of the subject in the feature vector map is equal to or smaller than a predetermined value.

[0053] After training, the classification layer is discarded to obtain a feature extractor that extracts feature vectors from images. The feature extractor training unit 206 stores the parameters of this feature extractor in the storage device 12 as feature extractor parameters 208.

[0054] In step S302, the feature vector map extractor learning unit 207 learns a feature vector map extractor used when the feature vector map extraction unit 203 extracts a feature vector map from an image, and stores the parameters as feature vector map extractor parameters 209 in the storage device 12.

[0055] The feature vector map extractor is a deep net that takes an image with height H and width W as input and outputs a feature vector map with height H / Kh, width W / Kw, and depth D.

[0056] Here, Kh and Kw are constants that determine the ratio between the size of the input image and the size of the feature vector map. The values ​​of Kh and Kw may be set arbitrarily. The larger Kh and Kw are, the smaller the size of the feature vector map will be, so the amount of calculation of the feature vector map extractor can be reduced. Kh and Kw may be the same value.

[0057] The feature vector at position (x / Kw, y / Kh) of the feature vector map represents the features of a face near position (x, y) of the input image. This feature vector is designed to have a high similarity to the feature vector obtained when a face image showing a face near position (x, y) of the input image is input to the feature extractor trained in step S301. An example of a training method for the feature vector map extractor will be described below.

[0058] First, a method for generating training data used for training the feature vector map extractor will be described. The training data is composed of pairs of images containing faces and correct feature vector maps corresponding to the images.

[0059] An example of creating a correct feature vector map will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an example of creating a correct feature vector map. First, an image 501 containing one or more faces is prepared. The height of the image 501 is H and the width is W. Next, a vector map with a height of H / Kh, a width of W / Kw, and a depth of D is generated, and all values ​​are initialized to zero.

[0060] Next, the positions of faces 502 to 505 are estimated from the image 501. The positions of the faces are obtained from annotation information separately added to the image 501, or by applying a known face detection method to the image 501.

[0061] Next, for each of the faces 502-505 in the image 501, a normalized face image is generated by the same method as described in step S301, and a feature vector is estimated by applying the feature extractor trained in step S301. A normalized face image 506 is an example of a normalized face image corresponding to the face 502. A feature vector 507 is an example of a feature vector corresponding to the face 502.

[0062] Next, the feature vector corresponding to the face at position (x, y) in the image 501 is added to position (x / Kw, y / Kh) in the vector map. At this time, the feature vector may also be added to positions within a certain range from position (x / Kw, y / Kh) in the vector map.

[0063] Furthermore, the feature vector may be added to the vector map after a process of shortening the length of the feature vector as the distance from the vector map position (x / Kw, y / Kh) increases.

[0064] For example, the center position of a face 502 in an image 501 is (x1, y1), and the feature vector 507 corresponding to the face 502 is v. The vector that is superimposed on the position (x1 / Kw+Δx, y1 / Kh+Δy) of the correct feature vector map is v·f(Δx, Δy)·g(Δx, Δy).

[0065] That is, a feature vector obtained when an image including a subject at position (x, y) in the image is input to feature vector extraction unit 202 is multiplied by a coefficient calculated based on Δx and Δy. In this way, a feature vector at position (x / Kw+Δx,y / Kh+Δy) in the correct feature vector map may be calculated.

[0066] Here, the function f(x, y) is a function that takes smaller values ​​as x and y become larger (for example, a two-dimensional normal distribution with a mean vector of (0, 0)T). Also, the function g(x, y) is a function that takes 1 when Δx Δx + Δy Δy is equal to or smaller than the square of R, and takes 0 otherwise. Here, R is a predetermined value that represents a real number other than 0.

[0067] By multiplying by the function g(x, y), the feature vector map extractor is trained to output a zero vector corresponding to a non-face region in the image 501. The vector map that has undergone the above processing becomes the ground truth vector map 508 corresponding to the image 501.

[0068] Image 501 may be artificially created. For example, a background image is prepared, and a human face image is superimposed at an appropriate position on the background image to artificially create image 501. If information on the position on the background image where the face image is superimposed is stored as metadata, a correct answer feature vector map corresponding to image 501 can be generated without estimating the face position. Learning data can be created by the above method.

[0069] Next, the created training data is used to train a feature vector map extractor. The feature vector map extractor is designed using a deep network called a Fully Convolutional Network, for example, U-Net.

[0070] The feature vector map extractor performs learning using an error backpropagation method or the like so that the output when an image of the training data is given to the feature vector map extractor approaches the correct feature vector map of the training data. The feature vector map extractor learning unit 207 stores the parameters of this feature vector map extractor in the storage device 12 as feature vector map extractor parameters 209.

[0071] Next, an example of the flow of search processing by the image processing device 100 according to this embodiment will be described with reference to Figs. 6 and 7. Fig. 6 is a flowchart showing the flow of search processing by the image processing device in embodiment 1. Fig. 7 is a diagram showing the entire system for searching for a specific face. Note that the operation of each step in the flowchart in Fig. 6 is performed by a CPU serving as a computer in a control device executing a computer program stored in a memory.

[0072] In step S601 (image acquisition step), the image acquisition unit 201 acquires a query image 301, which is an image including a subject to be searched, and a to-be-searched image 304, which may include the subject to be searched. The query image 301 contains only one query face, and no faces other than the query face are included. The vertical width of the to-be-searched image 304 is H and the horizontal width is W.

[0073] In step S602, the feature vector extraction unit 202 detects a query face 302 from the query image 301 acquired by the image acquisition unit 201, and extracts a feature vector 303 of the query face 302. Here, step S602 functions as a feature vector extraction step for extracting a first feature vector representing a feature of a search target included in the query image.

[0074] The extraction of the feature vector will now be described in detail. First, the feature vector extraction unit 202 generates a normalized query image by the same method as described in step S301. Next, the feature vector extraction unit 202 extracts a feature vector 303 of the query face 302 from the normalized query image by using the feature extractor parameters 208 previously learned in step S301. The feature vector 303 is a vector with a dimensionality of D.

[0075] In step S603, the feature vector map extraction unit 203 extracts a feature vector map 310 from the searched image 304 acquired by the image acquisition unit 201. The searched image 304 is an image that may include the same person as the query face 302. Here, it is assumed that the face 305 is the face of the same person as the query face 302. Here, step S603 functions as a feature vector map extraction step that extracts a feature vector map that represents the feature vector of the subject at each position in the searched image.

[0076] The feature vector map extraction unit 203 uses the feature vector map extractor parameters 209 previously learned in step S302 to extract a feature vector map 310 from the to-be-searched image 304. The feature vector map 310 is a map with a vertical width of H / Kh, a horizontal width of W / Kw, and a depth of D, and a feature vector at a position (x / Kw, y / Kh) of this map represents a feature amount of a face in the vicinity of the position (x, y) of the to-be-searched image 304.

[0077] In step S604, the heat map calculation unit 204 calculates a heat map 311 based on the feature vector 303 extracted by the feature vector extraction unit 202 and the feature vector map 310 extracted by the feature vector map extraction unit 203. Here, step S604 functions as a heat map calculation step that performs an operation between the first feature vector and the feature vector map to calculate a heat map that indicates the likelihood that the search target exists.

[0078] The heat map 311 is a map with a vertical width of H / Kh, a horizontal width of W / Kw, and a depth of 1. The value of a position (x, y) of the heat map 311 is calculated based on the similarity (e.g., inner product) between the feature vector at the position (x, y) of the feature vector map 310 and the feature vector 303.

[0079] The heat map 311 can be interpreted as a map representing the likelihood of the presence of the query face 302. That is, if the center of the query face 302 is at position (x1, y1) in the to-be-searched image 304, the value at position (x1 / Kw, y1 / Kh) in the heat map 311 is expected to be a local maximum.

[0080] Also, the value of the heat map 311 is expected to decrease the farther away from the position (x1 / Kw, y1 / Kh). Also, if there is no face near the position (x2, y2) of the searched image 304, or if there is a face of a person different from the query face 302, the value of the position (x2 / Kw, y2 / Kh) of the heat map 311 is expected to be close to zero.

[0081] In step S605, the face position estimation unit 205 estimates candidate positions in the to-be-searched image 304 where the query face 302 may appear, from the heat map 311 estimated by the heat map calculation unit 204.

[0082] The candidate positions can be estimated by selecting positions in the heat map 311 where the values ​​are maximum and equal to or greater than a threshold value. That is, the candidate positions in the to-be-searched image are determined based on positions in the heat map calculated by the heat map calculation unit 204 where the values ​​are maximum and equal to or greater than a threshold value.

[0083] When the candidate positions are clustered locally, the number of candidate positions may be reduced by performing a process of merging candidate positions that are within a predetermined distance. The candidate position (x3, y3) found on the heat map 311 corresponds to the candidate position (x3·Kw, y3·Kh) in the searched image 304.

[0084] 7 are examples of candidate positions in the heat map 311. The candidate positions 312 and 313 in the heat map 311 correspond to the faces 305 and 306, respectively, in the to-be-searched image 304, and indicate the possibility that the faces 305 and 306 are the same person as the query face 302.

[0085] When the number of candidate positions estimated by the face position estimation unit 205 is zero, it is determined that the query face 302 does not appear in the searched image 304. When the number of candidate positions estimated by the face position estimation unit 205 is one or more, it is determined that the query face 302 appears in the searched image 304. The face position estimation unit 205 may generate a searched image in which a circle, a rectangle, or the like is superimposed around the estimated candidate positions, and display the generated image on the display device via the output device 14.

[0086] As described above, according to the image processing device 100 of embodiment 1, when a number of small subjects are captured in a searched image, it is possible to quickly estimate whether a certain subject is captured and to quickly estimate candidate positions where a particular subject may be captured.

[0087] <Embodiment 2> In the second embodiment, a case will be described in which high-precision face authentication is performed in addition to the face search described in the first embodiment. Descriptions of the same parts as in the first embodiment will be omitted, and only the differences will be described.

[0088] 8 is a functional block diagram showing an example of the functional configuration of an image processing device according to embodiment 2. The image processing device 100 has a face position estimation unit 801 and a high-precision face authentication unit 802. The face position estimation unit 801 estimates a candidate position where a query face may appear in a to-be-searched image from a heat map calculated by the heat map calculation unit 204. The candidate position estimated by the face position estimation unit 801 is sent to the high-precision face authentication unit 802.

[0089] The high-precision face authentication unit 802 functions as a determination means that uses high-precision face authentication for each face in a candidate position estimated by the face position estimation unit 801 to determine with high accuracy whether a query face is captured.

[0090] An example of the process flow of the image processing device 100 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of search process by the image processing device according to the second embodiment. Note that the operation of each step in the flowchart in Fig. 9 is performed by a CPU serving as a computer in the control device executing a computer program stored in a memory. Steps S301 to S305 in Fig. 9 are the same as steps S301 to S305 in Fig. 3, and therefore description thereof will be omitted.

[0091] In step S901, the high-precision face authentication unit 802 uses high-precision face authentication to determine with high accuracy whether or not a query face is captured in each face at the candidate position estimated by the face position estimation unit 801. Hereinafter, an example will be described in which high-precision face authentication is performed on a face at the candidate position 312 in the heat map 311 in Fig. 7.

[0092] A face image to be subjected to high-precision face recognition is obtained from the searched image 304. When the position of the candidate position 312 in the heat map 311 is (x4, y4), the candidate position in the searched image 304 corresponding to this position is (x4·Kw, y4·Kh). A face is detected by applying a known face detection method to the vicinity (surroundings) of the position (x4·Kw, y4·Kh) in the searched image 304.

[0093] Here, the face detection process is performed on each of the images obtained by resizing the searched image 304 to slightly different sizes, and the face detection results are integrated to detect faces of different sizes. After the faces are detected, a normalized face image is generated and a feature vector is extracted using the feature extractor parameters 208 in the same manner as described in step S602.

[0094] This feature vector is expected to be the feature vector of the face 305. The similarity between the extracted feature vector and the feature vector 303 of the query face 302 is calculated, and depending on whether the similarity is greater than a preset threshold, it is determined with high accuracy whether the face 305 and the query face 302 are the same person.

[0095] That is, in the second embodiment, for each of the candidate positions in the searched image, a subject image is acquired from the searched image, and a feature vector extraction means is applied to the subject image to extract a second feature vector. Then, an operation is performed between the first feature vector and the second feature vector to determine whether a search target exists at the candidate position in the searched image.

[0096] Conventional face search systems first perform face detection on the image to be searched, and then perform highly accurate face recognition processing on each detected face, whereas the face search system of this embodiment performs highly accurate face recognition on only a small number of candidates narrowed down by the heat map. Therefore, the time required for face search can be significantly reduced when the image to be searched contains many small subjects.

[0097] It is also possible to change the settings to prioritize either the accuracy or speed of face search. If accuracy is prioritized, the threshold value for selecting candidate positions in step S605 can be reduced to increase the number of candidate positions, or the number of parameters for the feature vector map extractor and feature extractor can be increased, but this will increase the processing time.

[0098] If priority is given to speed, the threshold for selecting candidate positions in step S605 may be increased to reduce the number of candidate positions, or the number of parameters for the feature vector map extractor or feature extractor may be reduced, but this will result in a decrease in accuracy. A UI having, for example, a slide bar may be used to allow the user to select whether priority is given to accuracy or speed.

[0099] As described above, according to the image processing device 100 of the second embodiment, when a large number of small objects are included in a searched image, it is possible to quickly narrow down candidate positions where a specific object (query object) may be included using a heat map. Then, by performing highly accurate object determination only on the candidate positions, it is possible to quickly and accurately search for a specific object.

[0100] <Embodiment 3> In the third embodiment, as in the second embodiment, a case will be described in which high-precision face authentication is performed in addition to the face search described in the first embodiment. Descriptions of the same parts as in the first and second embodiments will be omitted, and only the differences will be described.

[0101] 10 is a functional block diagram showing an example of the functional configuration of an image processing device according to embodiment 3. The image processing device 100 includes a face position estimation unit 1001, a high-resolution image acquisition unit 1002, and a high-precision face authentication unit 1003.

[0102] The face position estimation unit 1001 estimates candidate positions where the query face may appear in the to-be-searched image from the heat map calculated by the heat map calculation unit 204. The candidate positions estimated by the face position estimation unit 1001 are sent to the high-resolution image acquisition unit 1002.

[0103] A high-resolution image acquisition unit 1002 acquires a high-resolution image of a face corresponding to a candidate position on the heat map estimated by the face position estimation unit 1001 by controlling a camera such as a PTZ (pan-tilt-zoom) camera.

[0104] Here, the high-resolution image acquisition unit 1002 functions as a high-resolution image acquisition means for acquiring a subject image with a higher resolution at each candidate position in the searched image. The high-resolution image acquired by the high-resolution image acquisition unit 1002 is sent to the high-precision face recognition unit 1003.

[0105] The high-precision face authentication unit 1003 functions as a determination means for determining with high accuracy whether a query face is included in the high-resolution image acquired by the high-resolution image acquisition unit 1002, using high-precision face authentication.

[0106] An example of the process flow of the image processing device 100 according to this embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the flow of search process by the image processing device according to the third embodiment. Note that the operation of each step in the flowchart in Fig. 11 is performed by a CPU serving as a computer in a control device executing a computer program stored in a memory. Steps S301 to S305 in Fig. 11 are the same as steps S301 to S305 in Fig. 3, and therefore description thereof will be omitted.

[0107] In step S1101, the high-resolution image acquisition unit 1002 acquires a high-resolution image of a face corresponding to a candidate position on the heat map estimated by the face position estimation unit 1001 by controlling a camera such as a PTZ camera. Hereinafter, a case where highly accurate face recognition is performed on a face at the candidate position 312 on the heat map 311 in Fig. 7 will be described as an example.

[0108] The facial images to be subjected to high-precision facial recognition are obtained by controlling a camera such as a PTZ camera. The PTZ camera is calibrated in advance so that an image of a zoomed-in image of any position of the image to be searched can be obtained.

[0109] When the position of the candidate position 312 in the heat map 311 is (x4, y4), the position of the searched image 304 corresponding to this position is (x4·Kw, y4·Kh). A high-resolution image corresponding to the position (x4·Kw, y4·Kh) of the searched image 304 is acquired by driving the PTZ camera, for example, to point the PTZ camera at that position and zooming in. Therefore, the face 305 appears more clearly in this high-resolution image than before the PTZ camera was driven.

[0110] In step S1102, the high-precision face authentication unit 1003 determines with high accuracy whether a query face is included in the high-resolution image acquired by the high-resolution image acquisition unit 1002, using high-precision face authentication. An example of this will be described below.

[0111] In step S1101, the high-precision face authentication unit 1003 receives the high-resolution image acquired by the high-resolution image acquisition unit 1002. Next, a known face detection method is applied to the high-resolution image to detect a face, and then a normalized face image is generated and a feature vector is extracted using the feature extractor parameters 208 in the same manner as described in step S602.

[0112] This feature vector is expected to be the feature vector of the face 305. The similarity between the extracted feature vector and the feature vector 303 of the query face 302 is calculated, and whether or not the face 305 and the query face 302 are the same person is determined based on whether or not the similarity is greater than a preset threshold value.

[0113] As described above, in the third embodiment, for each of the candidate positions in the searched image, a high-resolution image is acquired using the high-resolution image acquisition unit 1002. Then, the feature vector extraction unit 202 is applied to the high-resolution image to extract a second feature vector, and an operation is performed between the first feature vector and the second feature vector to determine whether a search target exists at the candidate position in the searched image.

[0114] As in the second embodiment, the third embodiment can significantly reduce the time required for face search when many small subjects are included in the image to be searched. Furthermore, since highly accurate face recognition is performed based on a high-resolution image, it is possible to determine whether a specific face is included with higher accuracy than in the second embodiment.

[0115] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The above embodiment includes the following combinations.

[0116] (Configuration 1) An image processing device comprising: a query image which is an image including a subject to be searched; an image acquisition means for acquiring a searched image; a feature vector extraction means for extracting a first feature vector representing a feature of the search target included in the query image; a feature vector map extraction means for extracting a feature vector map representing the feature vector of the subject at each position of the searched image; and a heat map calculation means for performing an operation between the first feature vector extracted by the feature vector extraction means and the feature vector map extracted by the feature vector map extraction means to calculate a heat map representing the likelihood that the search target exists.

[0117] (Configuration 2) The image processing device according to configuration 1, further comprising a candidate position estimation means for estimating a candidate position in the searched image which is a position where the search target is likely to appear from the heat map calculated by the heat map calculation means.

[0118] (Configuration 3) The image processing device according to configuration 1 or 2, wherein the subject is a face.

[0119] (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the feature vector extraction means is pre-trained to output feature vectors with close distances for images of the same subject or subjects belonging to the same class, and to output feature vectors with far distances for images of subjects that are not the same or subjects that do not belong to the same class.

[0120] (Configuration 5) The image processing device described in any one of configurations 1 to 4, characterized in that the feature vector map extraction means is trained in advance so that when images including the same subject or subjects belonging to the same class are input to the feature vector extraction means and the feature vector map extraction means, the difference in distance between the feature vector output by the feature vector extraction means and the feature vector at the position of the subject in the feature vector map output by the feature vector map extraction means is less than a predetermined value.

[0121] (Configuration 6) The image processing device described in any one of configurations 1 to 5, characterized in that the feature vector extraction means is trained based on a pair of an image and a correct feature vector map, and when Kw and Kh are constants, a feature vector at a position (x / Kw+Δx,y / Kh+Δy) of the correct feature vector map is calculated by multiplying a feature vector obtained when an image including a subject at a position (x,y) of the image is input to the feature vector extraction means by a coefficient calculated based on Δx and Δy.

[0122] (Configuration 7) The candidate position estimation means The image processing device according to any one of configurations 2 to 6, characterized in that the candidate positions in the searched image are determined based on positions in the heat map calculated by the heat map calculation means where the value is maximum and above a threshold value.

[0123] (Configuration 8) An image processing device described in any one of configurations 2 to 7, further comprising a determination means for obtaining a subject image from the searched image at each of the candidate positions in the searched image, applying the feature vector extraction means to the subject image to extract a second feature vector, performing an operation between the first feature vector and the second feature vector, and determining whether the search target is present at the candidate position in the searched image.

[0124] (Configuration 9) A high-resolution image acquisition means for acquiring a subject image having a higher resolution at each of the candidate positions in the searched image is further provided, The image processing device described in any one of configurations 2 to 8, further comprising a determination means for determining whether or not the search target is present at the candidate position in the searched image, acquiring a high-resolution image using the high-resolution image acquisition means, applying the feature vector extraction means to the high-resolution image to extract a second feature vector, performing an operation between the first feature vector and the second feature vector, and determining whether or not the search target is present at the candidate position in the searched image.

[0125] (Method) An image processing method comprising: an image acquisition step for acquiring a query image, which is an image including a subject that is to be searched; an image to be searched; a feature vector extraction step for extracting a first feature vector representing a feature of the search target contained in the query image; a feature vector map extraction step for extracting a feature vector map representing the feature vector of the subject at each position of the image to be searched; and a heat map calculation step for performing an operation between the first feature vector extracted by the feature vector extraction step and the feature vector map extracted by the feature vector map extraction step to calculate a heat map representing the likelihood that the search target exists.

[0126] (Program) A computer program for controlling each means of the image processing device according to any one of configurations 1 to 9 by a computer.

[0127] In order to realize a part or all of the control in the above-described embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0128] 201: Image acquisition unit 202: Feature vector extraction unit 203: Feature vector map extraction unit 204: Heat map calculation unit 205: Face position estimation unit

Claims

1. an image acquisition means for acquiring a query image which is an image including a subject to be searched, and a searched image including a plurality of subjects, one of which may be the subject to be searched; a feature vector extraction means for extracting a first feature vector representing a feature of the search target included in the query image; a feature vector map extracting means for extracting a feature vector map representing a feature vector of a subject at each position in the searched image; a heat map calculation means for calculating a heat map representing the likelihood that the search target exists by performing an operation between the first feature vector extracted by the feature vector extraction means and the feature vector map extracted by the feature vector map extraction means; and 1. An image processing device comprising:

2. 2. The image processing device according to claim 1, further comprising a candidate position estimation means for estimating a candidate position in the searched image that is highly likely to contain the search target, from the heat map calculated by the heat map calculation means.

3. 2. The image processing device according to claim 1, wherein the subject is a face.

4. The feature vector extraction means For images of the same subject or subjects belonging to the same class, feature vectors with close distances are output.

2. The image processing device according to claim 1, wherein the image processing device is pre-trained so that feature vectors with a large distance are output for images of different subjects or of subjects not belonging to the same class.

5. 2. The image processing device according to claim 1, wherein the feature vector map extraction means is trained in advance so that when images including the same subject or subjects belonging to the same class are input to the feature vector extraction means and the feature vector map extraction means, the difference in distance between the feature vector output by the feature vector extraction means and the feature vector at the position of the subject in the feature vector map output by the feature vector map extraction means is equal to or less than a predetermined value.

6. The feature vector extraction means is trained based on pairs of images and correct feature vector maps, and when Kw and Kh are constants, 2. The image processing device according to claim 1, wherein the feature vector at position (x / Kw+Δx, y / Kh+Δy) of the correct feature vector map is calculated by multiplying a feature vector obtained when an image including a subject at position (x, y) of the image is input to the feature vector extraction means by a coefficient calculated based on Δx and Δy.

7. The candidate position estimation means 3. The image processing device according to claim 2, wherein the candidate positions in the image to be searched are determined based on positions in the heat map calculated by the heat map calculation means where the values ​​are maximum and equal to or greater than a threshold value.

8. acquiring a subject image from the searched image for each of the candidate positions in the searched image, applying the feature vector extraction means to the subject image to extract a second feature vector, and performing an operation between the first feature vector and the second feature vector; 3. The image processing apparatus according to claim 2, further comprising a determination unit for determining whether the search target exists at the candidate position in the image to be searched.

9. a high-resolution image acquisition unit for acquiring a subject image with a higher resolution at each of the candidate positions in the searched image; The image processing device described in claim 2 further comprises a determination means for determining whether the search target exists at the candidate position in the search target image by performing an operation between the first feature vector and the second feature vector to obtain a high-resolution image at each of the candidate positions in the search target image using the high-resolution image acquisition means, applying the feature vector extraction means to the high-resolution image to extract a second feature vector, and determining whether the search target exists at the candidate position in the search target image.

10. an image acquisition step of acquiring a query image, which is an image including a subject to be searched, and a searched image including a plurality of subjects, one of which may be the search subject; a feature vector extraction step of extracting a first feature vector representing a feature of the search target included in the query image; a feature vector map extraction step of extracting a feature vector map representing a feature vector of a subject at each position in the searched image; a heat map calculation step of calculating a heat map representing the likelihood that the search target exists by performing an operation between the first feature vector extracted in the feature vector extraction step and the feature vector map extracted in the feature vector map extraction step; An image processing method comprising:

11. A computer program for causing a computer to function as each means of the image processing device described in any one of claims 1 to 9.

Images